Trabalho apresentado em 14th edition of IEEE International Symposium on Medical Measurement and Applications (MeMeA'19), junho 2019, Istanbul, TurquiaN/

Groza, V.

Pereira, José Miguel Costa Dias

Postolache, Octavian

Roza, V. C.

English

Repositório Comum

Emotions Assessment on Simulated Flights   V. C. Roza, O. Postolache V. Groza J. M. Dias Pereira, Instit. de Telecomunicações ISCTE-IUL, Lisbon Portugal valber_cesar_roza@iscte.pt Instit. de Telecomunicações, ISCTE-IUL, Lisbon Portugal opostolache@lx.it.pt, University of Ottawa, Canada vgroza@uottawa.ca Instit. de Telecomunicações, EST-IPS Setúbal, Portugal dias.pereira@estsetubal.ips.pt     Abstract— The emotions on pilots play important role on their performance during the service. Thus, an emotion prediction methodology based on physiological parameters such as galvanic skin response and heart rate as so as the facial recognition was considered in the present work. Several tests with eight volunteers were carried out that were used flight simulator. A small camera and the Face Reader software were used to record the users’ face during the fly task and to perform the video off-line processing to extract facial emotions during performed flights. The considered emotions were: happy, sad, angry, surprised, scared and disgust. To predict these emotions, the Artificial Neural Network (ANN) was applied. The experiment shows that is possible to predict emotions using these data and the best predict model was reached with 2 hidden layers, having a minimum squared error of 0.219.  Keywords— Emotion monitoring, heart rate, GSR sensing, emotion analysis, flight simulator, aviation safety.  I. INTRODUCTION Emotion analysis and prediction are an important research field that are useful to try to understand and predict accurately several emotions inside a certain contexts and tasks [1]. Several algorithms were developed to reach the best emotional predictions which can be applied on different scenarios (e.g. entertainments, cities’ routines, stressful administrative jobs, critical aviation’s, procedures). The acquisition of these emotions is majority taken from physiological sensing, speech, face and body gestures. Emotions play a critical role in human bio-regulation and survival. These emotions’ characteristics are leaded by the brain as results of several chemical processes that joins several biological and external factors, which produce an output reflected as an emotion [2]. The influence of emotions can also compromise the decision- making and cognitive functioning in the aviation affecting the heart functioning [3], for instance. Its effects may result in severe aviation accidents caused by human failure as consequence of systematic errors that sometimes produces some hazards as for instance: failures in the analysis of flight problems and failures in the choice of a correct action that a certain situation requires [4]. These are some reasons that motivate the need of emotional researches inside of the aviation contexts as for instance: drugs abuse, familiar problems, suicides, alcohol consumptions, workload, stress long flights, among others.  Looking for the same problem, Antonio et al. (2018), developed a study based on flight simulator, emotion and heart rate. They confirmed the sensitivity of the HR to cognitive demand and training effects, with increased HR when the task was more difficult and decreased HR with training (time-on-task) which can be critical in many flight situations [5]. The Boeing Aerospace Company presented a statistical summary [6], about the commercial jet airplane accidents confirmed to worldwide operations for 1959 through 2016, considering airplanes that are heavier than 60,000 pounds maximum gross weight.  There was shown that each year, the amount of air accidents is getting lower (decreasing the events of accident) including fatalities with hull losses or not. Even with these data, there are other problems that need to find good solutions, as for instance, the emotional factors that can be dangerous on the flight operations when it is treated with irresponsibility. II. METHODOLOGY  Preliminary training was carried out with the volunteers including simulation tasks such as: flight maneuvers, airplane controls in the air, takeoff, climb, navigation (cruise route), descend, approach and landing. The training focused on how each flight control works, controls that are later used to execute airplane navigation tasks during the proposed experiment. In the present work eight volunteers (beginners on flight simulator (Fig.1)) were performed seven different tasks based on basic concepts of visual flight rules (VFR) through the air traffic rules and procedures applicable to air traffic in Lisbon FIR and Santa Maria Oceanic FIR. The rules and procedure conform with Annex 2 and 11 from the Convention on International Civil Aviation [7][8]. Fig. 1. Simulator GUI for Aircraft Cessna–172. During the flight tasks, physiological parameters (heart rate, galvanic skin response) and user’s face information were acquired and stored in a remote server where the emotion classifier was implemented. The implemented architecture is presented in Fig.2.                         Fig. 2. System architecture applied to aviation context. The flight route and the main tasks to be performed by the users were already described in [8] and are associated with a real flight plan that can assure effective emotional reactions on volunteers’ level. A. Flight Route The proposed route has almost 8.4 nmi (Nautical Miles) of distance from Lisbon International Airport (ICAO LPPT) to Alverca (ICAO LPAR), intercepting the waypoints WP1, WP2 and WP3 with estimated time of arrival (ETA) of almost 7 minutes.  The takeoff runway from Lisbon  was the  03  (HDG 026) and landing runway from Alverca was the 04 (HDG 039), as shown in Fig. 3. Each step of the simulated flight was presented, showing the ideal velocities, directions and altitudes. Fig. 3. Flight route (red line) of the experiment (Lisbon to Alverca).  These references of altitudes and velocities were based on the real checklist of the Cessna–172 aircraft, and the real air charts parameters from the Lisbon air space. Because the simulations were executed with beginner users of flight simulator, the air charts, fuel mixtures, VOR/ILS navigation, real checklists, ATIS, or technical communications were not applied to them.  B. Tasks and Users’ Checklist This work considers seven tasks related with the main moments of a real flight, such as: takeoff (Task1), climb (Task 2), navigation/cruise route (Task 3), descent (Task 4), initial approach (Task 5), final approach (Task 6) and landing (Task 7). Were also considered two different users: the first pilot and the second pilot (co-pilot). Each using a set of well-defined tasks along the simulated flight. The pilot and co–pilot are guided by checklists. These checklists show to both what to do during each moment of the flight from the main experiment as presented in Table I. TABLE I CHECKLISTS USED DURING THE FLIGHT SIMULATION. Task Pilot Checklist Co–Pilot Checklist T1 Throttle Wait For                      Full               80KIAS -                    – Alert       80KIAS ”Call-out” T2 Climb Roll             Climb Intercept               550ft  Right for 10s             1,800ft                 WP1 Alert                          Climb Alert                    Roll Time Alert                         Climb Alert                      Intercept T3 Throttle Check Intercept Throttle Wait                  70%               Route         WP2     10%                          5s Alert          Check Alert      Alert Alert                  Throttle                        Route                  Intercept                   Throttle                           Time T4 Descent Intercept                900ft                 WP3 Alert Alert                 Descent               Intercept T5 Descent Throttle Pitch Descent                  550ft 40-65KIAS                  15◦                   250ft Alert            Flaps – Alert                    Descent                 Set Full                              –                      Descent T6 Descent                     15ft Alert                        Descent T7 Touch        Throttle       Full Stop –                  0KIAS                        – – Flaps –                              –                  Set 0◦                              –    The tasks of the co–pilot was executed by the experiment manager and the checklists structure, such as the route, were based on a real flight procedure. C. Experiment set-up The simulated flight experiment was carried out in laboratory where were imposed indoor environment conditions such as illumination, noise level, sound intensity, for all the volunteer of the experiment. Two computers were used for testing purposes: one is a Nvidia GTX1080TI, 32GB RAM, processor Intel i7 7th generation and 69-inch LED Smart TV; and the other is aNVidia GTX960, 16GB RAM, processor Intel i7 7th generation and 15-inch LED monitor.  The first computer was used by the pilot (running the Microsoft Flight Simulator) to execute the flight experiment, and the second computer was used by the manager of the experiment, to execute tasks of the co–pilot and to run all parallel software to acquire the signals from all sensors applied on this experiment. D. Face Recordings During the experiment, the user face was recorded using the Software Face Reader v7.1 (2017) as shown in Fig. 4. These emotional data from each face were stored and processed after the simulation to be used as the emotion classes to be used by the emotion classifier. Were considered six emotions: happy, sad, angry, surprised, disgust and scared.  Fig. 4. Face Reader v7.1 software used on this experiment. This experiment considers two physiological parameters: including heart rate (HR) that was obtained through photoplethysmography signal (PPG) and galvanic skin response (GSR) (Fig. 5). The PPG and GSR measurement channels are part of Shimmer 3 GSR wearable device characterized by Bluetooth communication. The face recording was also performed during the experiment. Face Reader software [9] was off-line applied to perform the facial emotions recognition.   Fig. 5. HR and GSR measurement: PPG sensor placed on the left ear GSR electrodes placed on the indicator and middle fingers. III. RESULTS AND DISCUSSIONS This work is supported by many data acquisitions, producing a dataset that was acquired during the experiment based on emotion using face recordings, HR and GSR data. The acquired data were processed to predict the emotions felt by the users during the experiments’ tasks. Fig. 6 presents some emotional expressions from users during the experiments.  . The correspondent emotions were processed by Face Reader software v7.1, which output 6 different waves representing the amplitude of each six emotions during the experiments   Fig. 6. Set of emotions expressed by the users’ face during the flight simulations.  As an example, Fig. 7 represents the processing results of the emotions data from the users’ face during a test flight. These emotions detected from users’ faces are referent to one single flight without relation between emotions versus flight tasks.   Fig. 7. Emotions detected from users’ faces for one single flight.  The Heart Rate (HR) and Galvanic Skin Response (GSR) data were used as the input of the prediction method. As the GSR is the most sensitive method to measure the emotional arousal applied on this work, special care was taken to avoid motion artifact in data; thus, the experiment’s manager gave instructions to the user to not move the fingers with electrodes and to not talk during experiment. Fig. 8 shows the HR, GSR and the correspondent emotions classes (0 to 5) along the time.         Fig. 8. Heart rate (top), GSR acquisition (middle) and emotions selected as classes to be predicted (bottom).  Bottom plot represents the emotions classes for one flight, i.e., the reference of emotions used in the ANN training and test; these emotion classes was achieved taking the majorities emotions along the time for each flight experiment. As previously referred six emotion classes were considered: 0 (happy), 1 (sad), 2 (angry), 3 (surprised), 4 (scared) and 5 (disgust). It is also possible to note the HR and GSR changes in the approach and landing phases. These results confirm the Luque- Casado et al. (2016) expectations. It is also verified that HRV and some cognitive tasks results suggested that HRV is highly sensitive to overall demands of sustained attention over and above the influence of other cognitive processes [3][10]. It is also possible to verify the heart rate variability during the execution of each flight tasks, namely during the take-off to approach and landing. Fig. 9 shows the prediction output on three different ANN architectures used for data processing purposes: 2 inputs (HR and GSR raw data), 2 hidden layers, and 100, 500 and 2,000 training iterations. In these predictions, no feature extraction were used; it justify the low level of prediction. Even without features in ANN inputs, the ANN output reacted in order to follow the classes, which motivate the research to apply features further. Also, no cross validation was applied in these initial predictions.    Fig. 9. Heart rate (top), GSR acquisition (middle) and emotions selected as classes to be predicted (bottom).  IV.   CONCLUSION AND FUTURE WORKS This work presented an emotional based experiment using flight simulation. Is also presented a prediction method based on ANN considering the inputs from HR and GSR using the output emotions processed by the Face Reader Software which analyze emotions from human face. Initial results shown that is possible to predict emotions from HR and GSR raw data even with not features extraction phase. The emotions were captured during the simulated flight tasks such as: takeoff, climb, navigation, approaches and landing. This study also shows the higher emotional variability along the takeoff and landing tasks, which can be useful to understand the human behavior under the takeoff and landing as shown in the Boeing reports that presented the approach and landing as the most critical flight phases. In advance, these initial results and the developed dataset will be used to execute a more deep signal processing, feature extraction, filtering and cross validation, to optimize the time of training of the presented ANN, that on this work, was a little high; also, we intend to use this emotion prediction method to improve the aviation safety along the real flights to give support to the prevention of human errors. ACKNOWLEDGMENTS This work is funded by FCT/MEC through national funds and when applicable co-funded by FEDER – PT2020 partnership agreement under the project UID/EEA/50008/2019.       REFERENCES [1] V. C. C. Roza and O. A. Postolache, “Design of a multimodal interface based on psychophysiological sensing to identify emotion,” in 22nd IMEKO TC4 Symposium and 20th International Workshop on ADC Modelling and Testing, vol. 22, no. 1, pp. 1–6, 2017. [2] A. R. Damasio, “Emotion, cognition and the human brain,” in Unity of knowledge: The convergence of natural and human science, vol. 935, no. 1, 2001. [3] M. C. et al., “The effects of emotion on pilot decision-making: A neuroergonomic approach to aviation safety,” in Transportation Research Part C: Emerging Technologies, vol. 33, 2013, pp. 272–281. [4] G. M. Breakwell, “The psychology of risk,” in Cambridge University Press, vol. 1, no. 2, 2014, pp. 272–281. [5] Antonio R. Hidalgo-Muñoz et al., “Cardiovascular correlates of emotional state, cognitive workload and time-on-task effect during a realistic flight simulation,” International Journal of Psychophysiology, vol. 128, pp. 62-69, 2018. [6] B. A. Company, “Statistical summary of commercial jet airplane acci- dents,” in Worldwide Operations — 1959–2016, 2017, pp. 1–26. [7] “Rules of the air – Annex 2 to the convention on international civil aviation,” in International Civil Aviation Organization (ICAO), vol. 1, 2005, pp. 1–74. [8] V.C.C. Roza, O. Postolache, “Emotion Analysis Architecture Based on Face and Physiological Sensing Applied with Flight Simulator”, Proceedings of 2018 International Conference and Exposition on Electrical And Power Engineering (EPE), vol.1 pp.1036-1040, 2018. [9] A. Hopfensitz and C. Mantilla, “Emotional expressions by sports teams: An analysis of world cup soccer  player  portraits,”  Journal of Economic Psychology, 2018.  [10] A. Luque-Casado, J. C. Perales, D. Crdenas, and D. Sanabria, “Heart rate variability and cognitive processing: The au tonomic response to task demands,” Biological Psychology, vol. 113, pp.83-90, 2016.    

Emotions Assessment on Simulated Flights

https://comum.rcaap.pt/bitstream/10400.26/30680/1/COMP_CONF_Flights.pdf

Emotions Assessment on Simulated Flights

Abstract

Similar works

Full text

Available Versions

Repositório Comum