Collaborative Robotic Masonry and Early Stage Fatigue Prediction

The nature of craft has often been dictated by the type and nature of the tool. The authors intend to establish a new relationship between a mechanically articulated tool and a human through the development a symbiotic relationship between them. This study attempts to develop and deploy a framework for collaborative robotic masonry involving one mason and one industrial robotic arm. This study aims to study the harmful posture and muscular stress developed during the construction work and involve a robotic arm to aid the mason to reduce the cumulative damage to one's body. Through utilization of RGBD sensors and surface electromyography procedure the study develops a framework that distributes the task between the mason and robot. The kinematics and electromyography detects the fatigue and harmful postures and activates the robot to collaborate with the mason in the process.


INTRODUCTION
This study focuses on the human robot collaboration in a non-deterministic method involving, real time posture sensing and muscle activity monitoring to maintain the well-being and the safety of the mason.With the integration of real time sensing of the mason, this study has attempted to establish an understandable, natural and humane collaboration between a human mason with a robot counterpart.The craft of masonry comes with an inherent risk of work related musculoskeletal disorder due to cumulative effect of carrying load, biomechanically harmful posture and repetitive cycle of work over a prolong period of time.The involvement of the robot with realtime sensing and analysis, in the process is an at-tempt to avoid those harmful postures and repetitive movements to keep the physical strain level of the human mason to a minimum.This specific process is the combined result of the followings; (i) analyzing harmful posture in terms of biomechanical effect and muscle fatigue, (ii) classifying the postures into categories based on prevalent nature of the posture, (iii) predicting a posture and probable muscle fatigue before the occurrence of the same, (iv) creating a control system for the robot to aid the mason in order to avoid the harmful postures and reduce / balance the fatigue on the specific muscle groups of the mason.
Previous work in this specific study has been three fold in nature.One part deals with the design of robot control protocol for interactive control of the  (Gannon, 2015) or Manus (Gannon, 2018.) has showcased the intuitive interactivity between one or more industrial robot arms with people with behaviors like mirror, avoid or follow.Second part deals with the detection of changes in gait parameters due to fatigue in a task cycle.Similarly, Project CHARM (SIGCHI (Group: U.S.), SIGART, IEEE Robotics and Automation, Croft, & Alcazar, 2013)deals with interaction between a robot co-worker and assembly line worker based on the need, behavior and expectation through gestures and motion sensing.Third part outlines the nature of the robotic behavior to engage human with the robot in a more natural way rather than machine like interactive cues from either the robot or the mason.Simple robots or robotic appliance like AUR Robotic Lamp or Travis by Guy Hoffman engages with human with relatable human gestures to increase engage-ment between robot and a human (Hoffman, 2015.).
The study aims to create a seamless workflow bridging these methods such a way that (i) through motion sensing and body surface electromyography the system can estimate harmful posture and predict muscle fatigue for specific muscle group which will (ii) enable the robot to take part in collaborative construction process to help avoid the harmful postures, (iii) based on the stress on a particular muscle group the robot positions the delivery of the material in different planes to neutralize the fatigue buildup in that muscle group while activating the other muscle group for the time being.A graphical user interface has also been developed to inform the mason about the fatigue level, posture severity and left/right bias of the work.This interface also acts as a medium of communication to the mason about the data on which the robot the acting but in a natural way.
The paper presents the theoretical and compu-tational framework, experimental setup, evaluation and classification algorithms, interactive robot control and the communication methods of this cyber physical system.

METHODS AND SETUP
The experimental setup consists of 2 parts in order to interface between the human and the robot;(i) recording a timestamped posture data to analyze the bio mechanical severity of the pose, (ii) collecting timestamped muscle activity data from 3 major muscle groups.To interface between various tools, robots and sensor McNeel Rhinoceros 6 was used with Grasshopper and Python was used.Within Grasshopper its add-on Firefly was used to interface with the specific sensor, microcontroller boards while Python was used to prepare and analyze the data in real time as they are acquired from the sensor

Recording Posture Data
During a work cycle, the analysis of the posture is necessary to identify the biomechanical severity of the pose.For analyzing the posture, Microsoft Kinect for Xbox One was used.The skeletal tracking feature of Kinect was used through Firefly in grasshopper by tracking 25 points over human body over a Cartesian space.Two analysis methods have been proposed and tested by the authors to build up the analysis system.First one stores the pose data in term of x, y, z coordinates with concurrent time stamp and analyze it later to create clustering dataset based on posture analysis.Second one was a real time posture analysis system to calculate the numbers of critical pose with visualization to communicate back to the mason.In order to prevent musculoskeletal disorder, prevention from biomechanically hazardous pose is necessary in time.In industries with reported higher rate of work related injury RULA (rapid upper body assessment) is deployed, an observation based survey to score severity of a particular posture of a worker during the work cycle and advice accordingly (Lynn & Corlett, 1993).In RULA system, position of upper arm, lower arm, wrist, neck, trunk and muscle use and force is systematically observed.Based on the angular position the individual limb with respect to global frames, these are scored over a numerical matrix to obtain the final score.The final score can range between 1-7 and based on the score the severity of the posture is assessed while suggesting the alternatives or remedies (Middlesworth, 2012).In the experimental setup, it is required to evaluate the pose based on certain parameter in order to cluster them into similar groups.In case of RULA assessment 2 key situations were identified.(i) It is based on observation only which leaves a certain margin of error and (ii) at any given point of time RULA can be applied to either left or right side of the body.In the experiment the authors have been able to synthesize the full skeleton of the body comprised of 25 points from which upper arms, lower arms, neck, spine can be constructed in forms of center line of that body part.Thus, for a more holistic scoring system an evaluation system has been made that takes consideration of both sides of the body at the same time.
The scoring method is similar to RULA but the each score is independent unlike RULA.A multiplication factor has been introduced based on the contribution of the posture to the development of lower back problems.A custom python based component has been developed and tested by the authors that calculates the flexion, abduction and bend angles of several parts viz.lower and upper arm, shoulder, trunk and neck.These scoring system was initially tested on students at Tech College, Aalborg.

Recording Muscle Activity Data
For the tracking the muscle activity level MyoWare surface electromyography sensors were used as they provide noninvasive muscle activity sensing.Another advantage of using MyoWare is, it can easily interface with Grasshopper through Firefly and Arduino.Two sets of sensors are placed on Biceps Brachii, Trapezius and Latissimus Dorsi to get estimate the fatigue level in arms, shoulders and lower back respectively

Figure 5 Pose Classification Categories
From the sensors a 10 bit ADC data is obtained and eventually processed to EMG value using simple algebraic equations supplied from the dataset.These data is now analyzed and processed to estimate the amount of fatigue in the specific muscle.Further Root Mean Square (RMS) velocity and Mean Average value (MAV) is calculated whose increase or decrease with respect to iteration and time gives the idea of level fatigue (Cifrek, Medved, Tonković, & Ostojić, 2009).Furthermore, authors have developed a data analysis method which takes in consideration the pattern of data happening 2 seconds earlier than the present timeframe to analyze the increase, decrease, highest state or lowest state in the muscle activity.This data analysis is incorporated within Grasshopper using GHPython.For each muscle group, a specific trigger is associated which detects the change in stress pattern and activated the robot for delivering the brick to the mason.This delivery plane to the mason depends on which triggers have been activated viz arms, shoulder and lower back with the sub categorization of left and right side.

Robot Control
In

Interface
To communicate with the mason in a natural interactive way two kind of GUI has been designed.The first one exhibits with the data authors have synthesized from the Kinect and another displays the data analyzed and synthesized from the EMG sensors.In the second interface, a tentative robot position and robot simulation window also has been provide.These interface are designed over the native Rhinoceros viewports with the addition of graphs and visualization of 3rd party addon like Conduit and Human.

DESIGN EXPERIMENTATION
Following the mentioned computational setup the authors have conducted several experiments with the system at different stages and with different parameters.These can be categorized into four broad categories.(i) Experiments with Kinect, (ii) experiments with Kinect and the robot, (iii) experiment with MyoWare EMG sensors and (iv) experiments with Kinect, MyoWare EMG sensor and the Robot.
The robot is mounted on a table at height of 840mm from ground with a horizontal workspace of 1220 X 2100 mm where the z = 0 lies 3mm above the horizontal surface.The Kinect is mounted on a tripod and kept beside the robot at a height of 1200mm to obtain the maximum field of view.The MyoWare sensors are attached on the body of one of the authors while the experiments were being conducted.The center of Kinect was placed 572mm apart from the robot origin and eventually compensated in the code to keep the robot origin and Kinect origin virtually same to keep the skeletal tracking aligned with the robot movement.The initial studies were made on a computationally derived brick wall having a length of 1500mm with 30 bricks each sizing 228 X 108 X 54mm.The first 10 bricks were positioned normally by the one of the author working as a mason while the pose and muscle activity data were being analyzed by the system.From the 11th brick to 30th brick robot handed over the brick to the mason at different delivery planes while his stats were displayed on the screen to inform him about his posture and muscle stress to make him aware of the certain movements.
Figure 9 Interaction with Robot : idle robot while mason in working; robot active; brick handover After each brick delivery the robot was programmed to wait with an eager gesture to hand over the next brick if necessary.This gestural movement was introduced to the robot to give a sense of constant observation by the working counterpart rather than a mundane mechanical object handing over the brick.The aim of the study was also to incorporate certain gestures to bridge the communication gap with the robots.

RESULTS
From the study with the specific method and computational framework yielded certain results and offered insights to the betterment of the process.These results can be categorized in the following categories: sensing, computational methods, robot control methods and some metaphysical arguments about a human robot collaboration.

Sensing
Between the two sensing methods employed in the system there were no conflict due to protocols.Though it was observed that the EMG sensors data gave more insight in the stress formation than the kinematics sensing.The sensors on the muscles gave uninterrupted results being attached to the body using gel pads but the Kinect resulted in improper or assumed pose due to auto-occlusion.These data might have contributed to some false alarms in the system.

Computational Methods
Grasshopper and GHPython has been used the computational base in the study but it is observed that some of the iterative components developed by the authors turns the speed down while slowing down the framerate from 20 frame per second to 4-5 frame per second.

Robot Control
Due to the development of the state machine, some custom components and easy control and visualization through Grasshopper made it simple to upload the program to the robot and activate in time of fa-tigue.The interactive gestures of the robot was intuitive during the waiting and handover.During the progress of one task, the collaboration between human and robot was developed at each iteration of the brick handling.The time gap between the delivery, the handover of the brick and waiting for the instructional cue from the mason seemed to be progressively smoother in nature at iteration.

DISCUSSION
The interaction within the mason to mason while collaborating is usually multimodal in nature.The sensing comes through a huge set of experience ('learning' in terms of a machine) accumulated over the years.Visual cues, auditory cues and sensing through experience creates a bond between the two collaborative humans.The mechanomorphic appearance of the robot firstly creates a sense of alienation between the mason and the 'collaborator' (Daas & Wit, 2018).But some simple mimetic and communicative robotic movement augments to the gestural communication between the mason and the robotic collaborator.Due to the morphological property, despite of the movement of them, the industrial robots scores low on both the horizontal (human likeliness) and vertical (affinity) axes in the case of 'Uncanny Valley' (Mori, MacDorman, & Kageki, 2012).It has been argued that a sense of likeliness can be developed through the deliberate development of a nonhuman design (Mori et al., 2012).In this particular case the relatable gesture like a human counterpart might give some sense of familiarity to the mason.
Matter -DIGITAL PRODUCTION AND ROBOTICS 2 -Volume 3 -eCAADe 37 / SIGraDi 23 | 177 The present work gives a sense of human robot collaboration considering safety concern of the mason in the AEC industry which only considers the musculoskeletal data and the posture data obtained from the sensors to develop a collaborative framework.But in reality, the collaboration might need more than fatigue and stress data and could depend on the visual cues from the mason.Already, the authors have implemented a small amount of pose estimation using neural network to predict the next pose of the mason, when robots can intervene to prevent the harmful posture.This can be extended into the framework by implementing more robust visual analysis considering the gestures of the mason.In this procedure it is observed that it is possible to incorporate the robot in a collaborative environment with human which also in turn opens up novel possibilities of tectonics involving the human and robot both as a decision maker based on the local or global experience.
Figure 1 Conceptual Model of the Framework Figure 2 Graphical Representation of Raw Kinect data and Analyzed Kinect Data Figure 3 Posture Scoring Component the experiment one UR 10 robot has been used with FT sensor and OnRobot RG6 Gripper.The control of the robots has been designed through the Grasshopper add-on Robots with custom GHPython scripts.The center of the robot control lies the state machine developed by the authors which keeps track of the which state the mason and the robot in and responds to the state accordingly("Finite State Machines | Brilliant Math & Science Wiki, " 2018).There are preset target plane for the robot to deliver the brick to based on the fatigue level and the posture of the mason.These poses and fatigue level in specific muscle groups trigger a specific state based on which the delivery plane of the brick.
Figure 6 Pose Estimation Using Neural Network Figure 10 Robot Control based on Sensor Trigger