Human Movement Direction Classification using Virtual Reality and Eye Tracking

Collaborative robots are becoming increasingly more popular in industries, providing flexibility and increased productivity for complex tasks. However, the robots are not yet that interactive since they cannot yet interpret humans and adapt to their behaviour, mainly due to limited sensory input. Rapidly expanding research fields that could make collaborative robots smarter through an understanding of the operators intentions are; virtual reality, eye tracking, big data, and artificial intelligence. Prediction of human movement intentions could be one way to improve these robots. This can be broken down into the three stages,\ua0Stage One:\ua0Movement Direction Classification,\ua0Stage Two:\ua0Movement Phase Classification,\ua0and\ua0Stage Three:\ua0Movement Intention Prediction.\ua0This paper defines these stages and presents a solution to\ua0Stage One\ua0that shows that it is possible to collect gaze data and use that to classify a person’s movement direction. The next step is naturally to develop the remaining two stages

Intended Human Arm Movement Direction Prediction using Eye Tracking

Collaborative robots are becoming increasingly popular in industries, providing flexibility and increased productivity for complex tasks. However, the robots are still not interactive enough since they cannot yet interpret humans and adapt to their behaviour, mainly due to limited sensory input. Prediction of human movement intentions could be one way to improve these robots. This paper presents a system that uses a recurrent neural network to predict the intended human arm movement direction, solely based on eye gaze, utilizing the notion of uncertainty to determine whether to trust a prediction or not. The network was trained with eye tracking data gathered using a virtual reality environment. The presented deep learning solution makes predictions on continuously incoming data and reaches an accuracy of 70.7%, for predictions with high certainty, and correctly classifies 67.89% of the movements at least once. The movements are, in 99% of the cases, correctly predicted the first time, before the hand reaches the target and more than 24% ahead of time in 75% of the cases. This means that a robot could receive warnings regarding in which direction an operator is likely to move and adjust its behaviour accordingly

Comparison of LSTM, Transformers, and MLP-mixer neural networks for gaze based human intention prediction

Collaborative robots have gained popularity in industries, providing flexibility and increased productivity for complex tasks. However, their ability to interact with humans and adapt to their behavior is still limited. Prediction of human movement intentions is one way to improve the robots adaptation. This paper investigates the performance of using Transformers and MLP-Mixer based neural networks to predict the intended human arm movement direction, based on gaze data obtained in a virtual reality environment, and compares the results to using an LSTM network. The comparison will evaluate the networks based on accuracy on several metrics, time ahead of movement completion, and execution time. It is shown in the paper that there exists several network configurations and architectures that achieve comparable accuracy scores. The best performing Transformers encoder presented in this paper achieved an accuracy of 82.74%, for predictions with high certainty, on continuous data and correctly classifies 80.06% of the movements at least once. The movements are, in 99% of the cases, correctly predicted the first time, before the hand reaches the target and more than 19% ahead of movement completion in 75% of the cases. The results shows that there are multiple ways to utilize neural networks to perform gaze based arm movement intention prediction and it is a promising step toward enabling efficient human-robot collaboration

Assessing worker performance using dynamic cost functions in human robot collaborative tasks

The aim of this research is to develop a framework to allow efficient Human Robot, HR, collaboration on manufacturing assembly tasks based on cost functions that quantify capabilities and performance of each element in a system and enable their efficient evaluation. A proposed cost function format is developed along with initial development of two example cost function variables, completion time and fatigue, obtained as each worker is completing assembly tasks. The cost function format and example variables were tested with two example tasks utilizing an ABB YuMi Robot in addition to a simulated human worker under various levels of fatigue. The total costs produced clearly identified the best worker to complete each task with these costs also clearly indicating when a human worker is fatigued to a greater or lesser degree than expected

Classification and Quantification of Human Error in Manufacturing: A Case Study in Complex Manual Assembly

Manual assembly operations are sensitive to human errors that can diminish the quality of final products. The paper shows an application of human reliability analysis in a realistic manufacturing context to identify where and why manual assembly errors occur. The techniques SHERPA and HEART were used to perform the analysis of human reliability. Three critical tasks were selected for analysis based on quality records: (1) installation of three types of brackets using fasteners, (2) fixation of a data cable to the assembly structure using cushioned loop clamps and (3) installation of cap covers to protect inlets. The identified error modes with SHERPA were: 36 action errors, nine selection errors, eight information retrieval errors and six checking errors. According to HEART, the highest human error probabilities were associated with assembly parts sensitive to geometry-related errors (brackets and cushioned loop clamps). The study showed that perceptually engaging assembly instructions seem to offer the highest potential for error reduction and performance improvement. Other identified areas of action were the improvement of the inspection process and workers’ provision with better tracking and better feedback. Implementation of assembly guidance systems could potentially benefit worker’s performance and decrease assembly errors

Ruuviyksikön kokoonpantavuuden tarkastelu ja kokoonpanon automatisoitavuuden kehittäminen

Kokoonpano on usein valmistavassa teollisuudessa osa-alue, jossa manuaalisen työn osuus on suurin. Ihmisten ja teknologian tulee toimia yhteistyössä kokoonpanotyön yksinkertaistamiseksi, sekä tehokkuuden ja tuottavuuden parantamiseksi. Selittävänä tekijänä manuaalisen työn suurelle osuudelle pidetään usein ihmisten joustavuutta, adaptoituvuutta ja luovuutta. Ihminen suoriutuu automaatioita paremmin tuntumaa, sekä luovaa ajattelua vaativissa toimenpiteissä. Automaation etuihin sen sijaan kuuluu sen väsymätön ja tasalaatuinen suorittaminen raskaissa sekä toistuvissa toimenpiteissä. Ihmisen poistaminen vaativista kokoonpanoprosesseista on kompleksista ja sen taloudellinen perustelu on vaikeaa. Vaativissa kokoonpanoprosesseissa tulisi ihmisen poistamisen sijaan keskittyä luomaan ihmisen sekä automaation välille paras mahdollinen työnjako. Tämä diplomityö keskittyy tarkastelemaan kohdeyritys Gardner Denver Oy:n tiloissa tapahtuvaa Enduro- mallin ruuviyksikön kokoonpanotyötä. Työ tarkastelee kokoonpantavan ruuviyksikön rakennetta, sekä nykyisessä kokoonpanoprosessissa sovellettavia työkaluja, liikkeitä ja kokoonpanometodeja. Tästä kokonaisuudesta arvioidaan sen nykytilanteen kehitettäviä osa-alueita. Kehitettäviä osa-alueita ovat esimerkiksi suunnittelun heikkoudet ja ihmistä tarpeettomasti kuormittavat kokoonpano-operaatiot. Ongelmakohtien arviointi perustuu käyttäjiltä kerättyyn tietoon sekä vertaisarvioiduissa teoksissa esiteltyihin yleisesti tunnistettuihin tapauksiin. Työn tuloksena on saatu useita eri kategorian tuloksia sekä kehitysehdotuksia. Käyttäjille suoritettu kyselytutkimus antaa käyttäjien näkökulmasta suuntaa tämän hetken haastavimmista tilanteista sekä automaation tarpeesta kokoonpanotyössä. Käyttäjien ilmoittamiin ongelmiin on esiteltynä kirjallisuuden avulla perusteltuja ratkaisuehdotuksia, sekä Autodesk Inventor sovelluksella tehtyjä malleja suunnittelumuutoksista ja prototyyppityökalusta. Lyöntioperaatiot eivät ole suositeltavia automaation eivätkä ihmisten kannalta. Tästä syystä laakereiden sisäkoolien prässäämiseen roottoreille tarvittava voima on laskettu ja tulokset esitetty. Vastaavat tulokset ovat nähtävissä myös laakereiden prässäämisestä laakeripesiin.Assembly is often the area in the manufacturing industry where the share of manual work is the largest. People and technology must work together to simplify assembly work and improve efficiency and productivity. People's flexibility, adaptability and creativity are often considered to be an explanatory factor for the large share of manual work. Humans perform better than automation in procedures that require feeling and creative thinking. The advantages of automation, on the other hand, include its tireless and consistent performance in heavy and repetitive procedures. Removing humans from demanding assembly processes is complex and its economic justification is difficult. In demanding assembly processes, instead of eliminating humans, the focus should be on creating the best possible division of labor between humans and automation. This thesis focuses on examining the assembly work of the screw unit of the Enduro model that takes place in the premises of the target company Gardner Denver Oy. The work examines the structure of the screw unit to be assembled, as well as the tools, movements and assembly methods applied in the current assembly process. From this whole, the aspects of its current situation that need to be developed are evaluated. Areas that need to be developed are, for example, design weaknesses and assembly operations that burden people unnecessarily. The assessment of problem areas is based on information collected from users and commonly recognized cases presented in peer-reviewed works. As a result of the work, several results of different categories and development proposals have been obtained. The survey conducted for users gives direction from the users' point of view about the most challenging situations at the moment and the need for automation in assembly work. Solutions to the problems reported by the users are presented, substantiated with the help of literature, as well as models of design changes and the prototype tool made with the Autodesk Inventor application. Strike operations are not recommended for automation or humans. For this reason, the force required to press the inner cores of the bearings onto the rotors has been calculated and the results presented. Corresponding results can also be seen from pressing the bearings into the bearing housings

Development of a methodology for the human-robot interaction based on vision systems for collaborative robotics

L'abstract è presente nell'allegato / the abstract is in the attachmen

Optimised task allocation using dynamic production data in human-robot teams

The demand of both industrial and consumer customers for increasingly higher degrees of customisation in products will see greater amounts of high mix production in the future of manufacturing. Despite this, automation must be implemented to improve the efficiency and output of manufacturing processes. However, traditional automation methods are often unsuitable due to long lead times for setup and little flexibility to adapt them to new tasks. Human-Robot (HR) teams provide a potential way to implement easily reconfigurable automation into future factories by utilising the best characteristics of human workers such as adaptability and intelligence with those of robot workers such as strength and repeatability. Robust task planning is required to implement such HR teams. However, current approaches allow adaptation to change in performance or composition of HR teams or optimisation of tasks as a whole but not necessarily both. In this research, a novel generalised task planning framework is proposed that uses a semi-online task planning approach, utilising online production data to determine worker capabilities then planning a manufacturing task for the HR team offline between task iterations. A system architecture is defined for such a framework but the focus of this research is the development and testing of the core technologies required for the framework to function to assess its utility. These include dynamic cost functions utilising online production data to accurately quantify the capabilities of human and robot workers across a work shift. These use continuous variables to quantify gradual changes in worker performance across a work shift; and discrete variables to detect instantaneous changes in capabilities that occur during a single task iteration. Additionally, a dynamic task planner is developed that implements dual layers of the Discrete Gravitational Search Algorithm to search for an optimum set of task assignments and task plan for a HR team given worker costs. Finally, mechanisms are proposed to intelligently implement task replanning across a work shift to optimise a HR team’s performance whilst ensuring it does not occur too frequently or unnecessarily. These core technologies were tested individually in example cases then combined together to test the ability of the task planning framework to optimise the performance of a HR team in two example manufacturing tasks across simulated work shifts. This showed that the dynamic cost functions represent an effective way to quantify and detect any changes in a worker’s capabilities across a work shift. Additionally, task replanning was shown to improve the performance of the HR team in some scenarios, such as the human worker being over fatigued, by reassigning subtasks to the robot worker as their performance declines. Importantly, the proposed task planning framework represents a generalised methodology that can easily be redeployed to different manufacturing tasks or compositions of HR teams