Detecting users’ cognitive load by galvanic skin response with affective interference

Experiencing high cognitive load during complex and demanding tasks results in performance reduction, stress, and errors. However, these could be prevented by a system capable of constantly monitoring users’ cognitive load fluctuations and adjusting its interactions accordingly. Physiological data and behaviors have been found to be suitable measures of cognitive load and are now available in many consumer devices. An advantage of these measures over subjective and performance-based methods is that they are captured in real time and implicitly while the user interacts with the system, which makes them suitable for real-world applications. On the other hand, emotion interference can change physiological responses and make accurate cognitive load measurement more challenging. In this work, we have studied six galvanic skin response (GSR) features in detection of four cognitive load levels with the interference of emotions. The data was derived from two arithmetic experiments and emotions were induced by displaying pleasant and unpleasant pictures in the background. Two types of classifiers were applied to detect cognitive load levels. Results from both studies indicate that the features explored can detect four and two cognitive load levels with high accuracy even under emotional changes. More specifically, rise duration and accumulative GSR are the common best features in all situations, having the highest accuracy especially in the presence of emotions

Exploring Emerging Technologies for Requirements Elicitation Interview Training: Empirical Assessment of Robotic and Virtual Tutors

Requirements elicitation interviews are a widely adopted technique, where the interview success heavily depends on the interviewer's preparedness and communication skills. Students can enhance these skills through practice interviews. However, organizing practice interviews for many students presents scalability challenges, given the time and effort required to involve stakeholders in each session. To address this, we propose REIT, an extensible architecture for Requirements Elicitation Interview Training system based on emerging educational technologies. REIT has components to support both the interview phase, wherein students act as interviewers while the system assumes the role of an interviewee, and the feedback phase, during which the system assesses students' performance and offers contextual and behavioral feedback to enhance their interviewing skills. We demonstrate the applicability of REIT through two implementations: RoREIT with a physical robotic agent and VoREIT with a virtual voice-only agent. We empirically evaluated both instances with a group of graduate students. The participants appreciated both systems. They demonstrated higher learning gain when trained with RoREIT, but they found VoREIT more engaging and easier to use. These findings indicate that each system has distinct benefits and drawbacks, suggesting that REIT can be realized for various educational settings based on preferences and available resources.Comment: Author submitted manuscrip

Self-adaptive unobtrusive interactions of mobile computing systems

[EN] In Pervasive Computing environments, people are surrounded by a lot of embedded services. Since pervasive devices, such as mobile devices, have become a key part of our everyday life, they enable users to always be connected to the environment, making demands on one of the most valuable resources of users: human attention. A challenge of the mobile computing systems is regulating the request for users¿ attention. In other words, service interactions should behave in a considerate manner by taking into account the degree to which each service intrudes on the user¿s mind (i.e., the degree of obtrusiveness). The main goal of this paper is to introduce self-adaptive capabilities in mobile computing systems in order to provide non-disturbing interactions. We achieve this by means of an software infrastructure that automatically adapts the service interaction obtrusiveness according to the user¿s context. This infrastructure works from a set of high-level models that define the unobtrusive adaptation behavior and its implication with the interaction resources in a technology-independent way. Our infrastructure has been validated through several experiments to assess its correctness, performance, and the achieved user experience through a user study.This work has been developed with the support of MINECO under the project SMART-ADAPT TIN2013-42981-P, and co-financed by the Generalitat Valenciana under the postdoctoral fellowship APOSTD/2016/042.Gil Pascual, M.; Pelechano Ferragud, V. (2017). Self-adaptive unobtrusive interactions of mobile computing systems. Journal of Ambient Intelligence and Smart Environments. 9(6):659-688. https://doi.org/10.3233/AIS-170463S65968896Aleksy, M., Butter, T., & Schader, M. (2008). Context-Aware Loading for Mobile Applications. Lecture Notes in Computer Science, 12-20. doi:10.1007/978-3-540-85693-1_3Y. Bachvarova, B. van Dijk and A. Nijholt, Towards a unified knowledge-based approach to modality choice, in: Proc. Workshop on Multimodal Output Generation (MOG), 2007, pp. 5–15.Barkhuus, L., & Dey, A. (2003). Is Context-Aware Computing Taking Control away from the User? Three Levels of Interactivity Examined. Lecture Notes in Computer Science, 149-156. doi:10.1007/978-3-540-39653-6_12Bellotti, V., & Edwards, K. (2001). Intelligibility and Accountability: Human Considerations in Context-Aware Systems. Human–Computer Interaction, 16(2-4), 193-212. doi:10.1207/s15327051hci16234_05D. Benavides, P. Trinidad and A. Ruiz-Cortés, Automated reasoning on feature models, in: Proceedings of the 17th International Conference on Advanced Information Systems Engineering, CAiSE’05, Springer-Verlag, Berlin, 2005, pp. 491–503.Bernsen, N. O. (1994). Foundations of multimodal representations: a taxonomy of representational modalities. Interacting with Computers, 6(4), 347-371. doi:10.1016/0953-5438(94)90008-6Bettini, C., Brdiczka, O., Henricksen, K., Indulska, J., Nicklas, D., Ranganathan, A., & Riboni, D. (2010). A survey of context modelling and reasoning techniques. Pervasive and Mobile Computing, 6(2), 161-180. doi:10.1016/j.pmcj.2009.06.002Blumendorf, M., Lehmann, G., & Albayrak, S. (2010). Bridging models and systems at runtime to build adaptive user interfaces. Proceedings of the 2nd ACM SIGCHI symposium on Engineering interactive computing systems - EICS ’10. doi:10.1145/1822018.1822022D.M. Brown, Communicating Design: Developing Web Site Documentation for Design and Planning, 2nd edn, New Riders Press, 2010.J. Bruin, Statistical Analyses Using SPSS, 2011, http://www.ats.ucla.edu/stat/spss/whatstat/whatstat.htm#1sampt.J. Cámara, G. Moreno and D. Garlan, Reasoning about human participation in self-adaptive systems, in: SEAMS 2015, 2015, pp. 146–156.Campbell, A., & Choudhury, T. (2012). From Smart to Cognitive Phones. IEEE Pervasive Computing, 11(3), 7-11. doi:10.1109/mprv.2012.41Y. Cao, M. Theune and A. Nijholt, Modality effects on cognitive load and performance in high-load information presentation, in: Proceedings of the 14th International Conference on Intelligent User Interfaces, IUI’09, ACM, New York, 2009, pp. 335–344.Chang, F., & Ren, J. (2007). Validating system properties exhibited in execution traces. Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering - ASE ’07. doi:10.1145/1321631.1321723H. Chen and J.P. Black, A quantitative approach to non-intrusive computing, in: Mobiquitous’08: Proceedings of the 5th Annual International Conference on Mobile and Ubiquitous Systems, ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), ICST, Brussels, 2008, pp. 1–10.Chittaro, L. (2010). Distinctive aspects of mobile interaction and their implications for the design of multimodal interfaces. Journal on Multimodal User Interfaces, 3(3), 157-165. doi:10.1007/s12193-010-0036-2Clerckx, T., Vandervelpen, C., & Coninx, K. (2008). Task-Based Design and Runtime Support for Multimodal User Interface Distribution. Lecture Notes in Computer Science, 89-105. doi:10.1007/978-3-540-92698-6_6Cook, D. J., & Das, S. K. (2012). Pervasive computing at scale: Transforming the state of the art. Pervasive and Mobile Computing, 8(1), 22-35. doi:10.1016/j.pmcj.2011.10.004Cornelissen, B., Zaidman, A., van Deursen, A., Moonen, L., & Koschke, R. (2009). A Systematic Survey of Program Comprehension through Dynamic Analysis. IEEE Transactions on Software Engineering, 35(5), 684-702. doi:10.1109/tse.2009.28Czarnecki, K. (2004). Generative Software Development. Lecture Notes in Computer Science, 321-321. doi:10.1007/978-3-540-28630-1_33M. de Sá, C. Duarte, L. Carriço and T. Reis, Designing mobile multimodal applications, in: Information Science Reference, 2010, pp. 106–136, Chapter 5.C. Duarte and L. Carriço, A conceptual framework for developing adaptive multimodal applications, in: Proceedings of the 11th International Conference on Intelligent User Interfaces, IUI’06, ACM, New York, 2006, pp. 132–139.Evers, C., Kniewel, R., Geihs, K., & Schmidt, L. (2014). The user in the loop: Enabling user participation for self-adaptive applications. Future Generation Computer Systems, 34, 110-123. doi:10.1016/j.future.2013.12.010Fagin, R., Halpern, J. Y., & Megiddo, N. (1990). A logic for reasoning about probabilities. Information and Computation, 87(1-2), 78-128. doi:10.1016/0890-5401(90)90060-uFerscha, A. (2012). 20 Years Past Weiser: What’s Next? IEEE Pervasive Computing, 11(1), 52-61. doi:10.1109/mprv.2011.78Floch, J., Frà, C., Fricke, R., Geihs, K., Wagner, M., Lorenzo, J., … Scholz, U. (2012). Playing MUSIC - building context-aware and self-adaptive mobile applications. Software: Practice and Experience, 43(3), 359-388. doi:10.1002/spe.2116Gibbs, W. W. (2005). Considerate Computing. Scientific American, 292(1), 54-61. doi:10.1038/scientificamerican0105-54Gil, M., Giner, P., & Pelechano, V. (2011). Personalization for unobtrusive service interaction. Personal and Ubiquitous Computing, 16(5), 543-561. doi:10.1007/s00779-011-0414-0Gil Pascual, M. (s. f.). Adapting Interaction Obtrusiveness: Making Ubiquitous Interactions Less Obnoxious. A Model Driven Engineering approach. doi:10.4995/thesis/10251/31660Haapalainen, E., Kim, S., Forlizzi, J. F., & Dey, A. K. (2010). Psycho-physiological measures for assessing cognitive load. Proceedings of the 12th ACM international conference on Ubiquitous computing - Ubicomp ’10. doi:10.1145/1864349.1864395Hallsteinsen, S., Geihs, K., Paspallis, N., Eliassen, F., Horn, G., Lorenzo, J., … Papadopoulos, G. A. (2012). A development framework and methodology for self-adapting applications in ubiquitous computing environments. Journal of Systems and Software, 85(12), 2840-2859. doi:10.1016/j.jss.2012.07.052Hassenzahl, M. (2004). The Interplay of Beauty, Goodness, and Usability in Interactive Products. Human-Computer Interaction, 19(4), 319-349. doi:10.1207/s15327051hci1904_2Hassenzahl, M., & Tractinsky, N. (2006). User experience - a research agenda. Behaviour & Information Technology, 25(2), 91-97. doi:10.1080/01449290500330331Ho, J., & Intille, S. S. (2005). Using context-aware computing to reduce the perceived burden of interruptions from mobile devices. Proceedings of the SIGCHI conference on Human factors in computing systems - CHI ’05. doi:10.1145/1054972.1055100Horvitz, E., Kadie, C., Paek, T., & Hovel, D. (2003). Models of attention in computing and communication. Communications of the ACM, 46(3), 52. doi:10.1145/636772.636798Horvitz, E., Koch, P., Sarin, R., Apacible, J., & Subramani, M. (2005). Bayesphone: Precomputation of Context-Sensitive Policies for Inquiry and Action in Mobile Devices. Lecture Notes in Computer Science, 251-260. doi:10.1007/11527886_33Kephart, J. O., & Chess, D. M. (2003). The vision of autonomic computing. Computer, 36(1), 41-50. doi:10.1109/mc.2003.1160055Korpipaa, P., Malm, E.-J., Rantakokko, T., Kyllonen, V., Kela, J., Mantyjarvi, J., … Kansala, I. (2006). Customizing User Interaction in Smart Phones. IEEE Pervasive Computing, 5(3), 82-90. doi:10.1109/mprv.2006.49S. Lemmelä, A. Vetek, K. Mäkelä and D. Trendafilov, Designing and evaluating multimodal interaction for mobile contexts, in: Proceedings of the 10th International Conference on Multimodal Interfaces, ICMI’08, ACM, New York, 2008, pp. 265–272.Lim, B. Y. (2010). Improving trust in context-aware applications with intelligibility. Proceedings of the 12th ACM international conference adjunct papers on Ubiquitous computing - Ubicomp ’10. doi:10.1145/1864431.1864491J.-Y. Mao, K. Vredenburg, P.W. Smith and T. Carey, User-centered design methods in practice: A survey of the state of the art, in: Proceedings of the 2001 Conference of the Centre for Advanced Studies on Collaborative Research, CASCON’01, IBM Press, 2001, p. 12.Maoz, S. (2009). Using Model-Based Traces as Runtime Models. Computer, 42(10), 28-36. doi:10.1109/mc.2009.336Mayer, R. E., & Moreno, R. (2003). Nine Ways to Reduce Cognitive Load in Multimedia Learning. Educational Psychologist, 38(1), 43-52. doi:10.1207/s15326985ep3801_6Motti, V. G., & Vanderdonckt, J. (2013). A computational framework for context-aware adaptation of user interfaces. IEEE 7th International Conference on Research Challenges in Information Science (RCIS). doi:10.1109/rcis.2013.6577709R. Murch, Autonomic Computing, IBM Press, 2004.Obrenovic, Z., Abascal, J., & Starcevic, D. (2007). Universal accessibility as a multimodal design issue. Communications of the ACM, 50(5), 83-88. doi:10.1145/1230819.1241668Patterson, D. J., Baker, C., Ding, X., Kaufman, S. J., Liu, K., & Zaldivar, A. (2008). Online everywhere. Proceedings of the 10th international conference on Ubiquitous computing - UbiComp ’08. doi:10.1145/1409635.1409645Pielot, M., de Oliveira, R., Kwak, H., & Oliver, N. (2014). Didn’t you see my message? Proceedings of the 32nd annual ACM conference on Human factors in computing systems - CHI ’14. doi:10.1145/2556288.2556973Poppinga, B., Heuten, W., & Boll, S. (2014). Sensor-Based Identification of Opportune Moments for Triggering Notifications. IEEE Pervasive Computing, 13(1), 22-29. doi:10.1109/mprv.2014.15S. Ramchurn, B. Deitch, M. Thompson, D. De Roure, N. Jennings and M. Luck, Minimising intrusiveness in pervasive computing environments using multi-agent negotiation, in: Mobile and Ubiquitous Systems: Networking and Services, MOBIQUITOUS 2004. The First Annual International Conference on, 2004, pp. 364–371.C. Roda, Human Attention and Its Implications for Human-Computer Interaction, Cambridge University Press, 2011.S. Rosenthal, A.K. Dey and M. Veloso, Using decision-theoretic experience sampling to build personalized mobile phone interruption models, in: Proceedings of the 9th International Conference on Pervasive Computing, Pervasive 2011, Springer-Verlag, Berlin, 2011, pp. 170–187.E. Rukzio, K. Leichtenstern and V. Callaghan, An experimental comparison of physical mobile interaction techniques: Touching, pointing and scanning, in: 8th International Conference on Ubiquitous Computing, UbiComp 2006, Orange County, California, 2006.Serral, E., Valderas, P., & Pelechano, V. (2010). Towards the Model Driven Development of context-aware pervasive systems. Pervasive and Mobile Computing, 6(2), 254-280. doi:10.1016/j.pmcj.2009.07.006D. Siewiorek, A. Smailagic, J. Furukawa, A. Krause, N. Moraveji, K. Reiger, J. Shaffer and F.L. Wong, Sensay: A context-aware mobile phone, in: Proceedings of the 7th IEEE International Symposium on Wearable Computers, ISWC’03, IEEE Computer Society, Washington, 2003, p. 248.Tedre, M. (2006). What should be automated? Proceedings of the 1st ACM international workshop on Human-centered multimedia - HCM ’06. doi:10.1145/1178745.1178753M. Valtonen, A.-M. Vainio and J. Vanhala, Proactive and adaptive fuzzy profile control for mobile phones, in: IEEE International Conference on Pervasive Computing and Communications, 2009, PerCom, 2009, pp. 1–3.Vastenburg, M. H., Keyson, D. V., & de Ridder, H. (2007). Considerate home notification systems: a field study of acceptability of notifications in the home. Personal and Ubiquitous Computing, 12(8), 555-566. doi:10.1007/s00779-007-0176-xWarnock, D., McGee-Lennon, M., & Brewster, S. (2011). The Role of Modality in Notification Performance. Lecture Notes in Computer Science, 572-588. doi:10.1007/978-3-642-23771-3_43Weiser, M., & Brown, J. S. (1997). The Coming Age of Calm Technology. Beyond Calculation, 75-85. doi:10.1007/978-1-4612-0685-9_6Van Woensel, W., Gil, M., Casteleyn, S., Serral, E., & Pelechano, V. (2013). Adapting the Obtrusiveness of Service Interactions in Dynamically Discovered Environments. Mobile and Ubiquitous Systems: Computing, Networking, and Services, 250-262. doi:10.1007/978-3-642-40238-8_2

Modality Effects on Cognitive Load and Performance in High-Load Information Presentation

In this study, we argue that modality planning in multimodal presentation systems needs to consider the modality characteristics at not only the presentational level but also the cognitive level, especially in a situation where the information load is high and the user task is time-critical. As a first step towards automatic cognitive-aware modality planning, we integrated the effect of different modalities on cognitive load and performance, using a high-load information presentation scenario. Mainly based on modality-related psychology theories, we selected five modality conditions (text, image, text+image, text+speech, and text+sound) and made hypotheses about their effects on cognitive load. Modality effects were evaluated by two cognitive load measurements and two performance measurements. Results confirmed most of the predicted modality effects, and showed that these effects become significant when the information load and the task demand are high. The findings of this study suggest that it is highly necessary to encode modality-related principles of human cognition into the modality planning procedure for systems that support high-load human-computer interaction

Multimodal information presentation for high-load human computer interaction

This dissertation addresses the question: given an application and an interaction context, how can interfaces present information to users in a way that improves the quality of interaction (e.g. a better user performance, a lower cognitive demand and a greater user satisfaction)? Information presentation is critical to the quality of interaction because it guides, constrains and even determines cognitive behavior. A good presentation is particularly desired in high-load human computer interactions, such as when users are under time pressure, stress, or are multi-tasking. Under a high mental workload, users may not have the spared cognitive capacity to cope with the unnecessary workload induced by a bad presentation. In this dissertation work, the major presentation factor of interest is modality. We have conducted theoretical studies in the cognitive psychology domain, in order to understand the role of presentation modality in different stages of human information processing. Based on the theoretical guidance, we have conducted a series of user studies investigating the effect of information presentation (modality and other factors) in several high-load task settings. The two task domains are crisis management and driving. Using crisis scenario, we investigated how to presentation information to facilitate time-limited visual search and time-limited decision making. In the driving domain, we investigated how to present highly-urgent danger warnings and how to present informative cues that help drivers manage their attention between multiple tasks. The outcomes of this dissertation work have useful implications to the design of cognitively-compatible user interfaces, and are not limited to high-load applications

Multimodal Content Delivery for Geo-services

This thesis describes a body of work carried out over several research projects in the area of multimodal interaction for location-based services. Research in this area has progressed from using simulated mobile environments to demonstrate the visual modality, to the ubiquitous delivery of rich media using multimodal interfaces (geo- services). To effectively deliver these services, research focused on innovative solutions to real-world problems in a number of disciplines including geo-location, mobile spatial interaction, location-based services, rich media interfaces and auditory user interfaces. My original contributions to knowledge are made in the areas of multimodal interaction underpinned by advances in geo-location technology and supported by the proliferation of mobile device technology into modern life. Accurate positioning is a known problem for location-based services, contributions in the area of mobile positioning demonstrate a hybrid positioning technology for mobile devices that uses terrestrial beacons to trilaterate position. Information overload is an active concern for location-based applications that struggle to manage large amounts of data, contributions in the area of egocentric visibility that filter data based on field-of-view demonstrate novel forms of multimodal input. One of the more pertinent characteristics of these applications is the delivery or output modality employed (auditory, visual or tactile). Further contributions in the area of multimodal content delivery are made, where multiple modalities are used to deliver information using graphical user interfaces, tactile interfaces and more notably auditory user interfaces. It is demonstrated how a combination of these interfaces can be used to synergistically deliver context sensitive rich media to users - in a responsive way - based on usage scenarios that consider the affordance of the device, the geographical position and bearing of the device and also the location of the device

Analyse visuelle et cérébrale de l’état cognitif d’un apprenant

Un état cognitif peut se définir comme étant l’ensemble des processus cognitifs inférieurs (par exemple : perception et attention) et supérieurs (par exemple : prise de décision et raisonnement), nécessitant de la part de l’être humain toutes ses capacités mentales en vue d’utiliser des connaissances existantes pour résoudre un problème donné ou bien d’établir de nouvelles connaissances. Dans ce contexte, une attention particulière est portée par les environnements d’apprentissage informatisés sur le suivi et l’analyse des réactions émotionnelles de l’apprenant lors de l’activité d’apprentissage. En effet, les émotions conditionnent l’état mental de l’apprenant qui a un impact direct sur ses capacités cognitives tel que le raisonnement, la prise de décision, la mémorisation, etc. Dans ce contexte, l’objectif est d’améliorer les capacités cognitives de l’apprenant en identifiant et corrigeant les états mentaux défavorables à l’apprentissage en vue d’optimiser les performances des apprenants. Dans cette thèse, nous visons en particulier à examiner le raisonnement en tant que processus cognitif complexe de haut niveau. Notre objectif est double : en premier lieu, nous cherchons à évaluer le processus de raisonnement des étudiants novices en médecine à travers leur comportement visuel et en deuxième lieu, nous cherchons à analyser leur état mental quand ils raisonnent afin de détecter des indicateurs visuels et cérébraux permettant d’améliorer l’expérience d’apprentissage. Plus précisément, notre premier objectif a été d’utiliser les mouvements des yeux de l’apprenant pour évaluer son processus de raisonnement lors d’interactions avec des jeux sérieux éducatifs. Pour ce faire, nous avons analysé deux types de mesures oculaires à savoir : des mesures statiques et des mesures dynamiques. Dans un premier temps, nous avons étudié la possibilité d’identifier automatiquement deux classes d’apprenants à partir des différentes mesures statiques, à travers l’entrainement d’algorithmes d’apprentissage machine. Ensuite, en utilisant les mesures dynamiques avec un algorithme d’alignement de séquences issu de la bio-informatique, nous avons évalué la séquence logique visuelle suivie par l’apprenant en cours de raisonnement pour vérifier s’il est en train de suivre le bon processus de raisonnement ou non. Notre deuxième objectif a été de suivre l’évolution de l’état mental d’engagement d’un apprenant à partir de son activité cérébrale et aussi d’évaluer la relation entre l’engagement et les performances d’apprentissage. Pour cela, une étude a été réalisée où nous avons analysé la distribution de l’indice d’engagement de l’apprenant à travers tout d’abord les différentes phases de résolution du problème donné et deuxièmement, à travers les différentes régions qui composent l’interface de l’environnement. L’activité cérébrale de chaque participant a été mesurée tout au long de l’interaction avec l’environnement. Ensuite, à partir des signaux obtenus, un indice d’engagement a été calculé en se basant sur les trois bandes de fréquences α, β et θ. Enfin, notre troisième objectif a été de proposer une approche multimodale à base de deux senseurs physiologiques pour permettre une analyse conjointe du comportement visuel et cérébral de l’apprenant. Nous avons à cette fin enregistré les mouvements des yeux et l’activité cérébrale de l’apprenant afin d’évaluer son processus de raisonnement durant la résolution de différents exercices cognitifs. Plus précisément, nous visons à déterminer quels sont les indicateurs clés de performances à travers un raisonnement clinique en vue de les utiliser pour améliorer en particulier, les capacités cognitives des apprenants novices et en général, l’expérience d’apprentissage.A cognitive state can be defined as a set of inferior (e.g. perception and attention) and superior (e.g. perception and attention) cognitive processes, requiring the human being to have all of his mental abilities in an effort to use existing knowledge to solve a given problem or to establish new knowledge. In this context, a particular attention is paid by computer-based learning environments to monitor and assess learner’s emotional reactions during a learning activity. In fact, emotions govern the learner’s mental state that has in turn a direct impact on his cognitive abilities such as reasoning, decision-making, memory, etc. In this context, the objective is to improve the cognitive abilities of the learner by identifying and redressing the mental states that are unfavorable to learning in order to optimize the learners’ performances. In this thesis, we aim in particular to examine the reasoning as a high-level cognitive process. Our goal is two-fold: first, we seek to evaluate the reasoning process of novice medical students through their visual behavior and second, we seek to analyze learners’ mental states when reasoning to detect visual and cerebral indicators that can improve learning outcomes. More specifically, our first objective was to use the learner’s eye movements to assess his reasoning process while interacting with educational serious games. For this purpose, we have analyzed two types of ocular metrics namely, static metrics and dynamic metrics. First of all, we have studied the feasibility of using static metrics to automatically identify two groups of learners through the training of machine learning algorithms. Then, we have assessed the logical visual sequence followed by the learner when reasoning using dynamic metrics and a sequence alignment method from bio-informatics to see if he/she performed the correct reasoning process or not. Our second objective was to analyze the evolution of the learner’s engagement mental state from his brain activity and to assess the relationship between engagement and learning performance. An experimental study was conducted where we analyzed the distribution of the learner engagement index through first, the different phases of the problem-solving task and second, through the different regions of the environment interface. The cerebral activity of each participant was recorded during the whole game interaction. Then, from the obtained signals, an engagement index was computed based on the three frequency bands α, β et θ. Finally, our third objective was to propose a multimodal approach based on two physiological sensors to provide a joint analysis of the learner’s visual and cerebral behaviors. To this end, we recorded eye movements and brain activity of the learner to assess his reasoning process during the resolution of different cognitive tasks. More precisely, we aimed to identify key indicators of reasoning performance in order to use them to improve the cognitive abilities of novice learners in particular, and the learning experience in general

Development of context-sensitive user interfaces

Dobro dizajniran, intuitivan i privlačan za korišćenje korisnički interfejs predstavlja ključni faktor uspeha računarskih proizvoda i sistema. Radi unapređenja razvoja i upotrebljivosti korisničkih interfejsa potrebno je uzeti u obzir karakteristike korisnika. Ovo zahteva interdisciplinaran pristup i korišćenje znanja iz različitih oblasti kao što su računarske, saznajne i biološke nauke. Pored toga, potrebno je uzeti u obzir karakteristike medija i fizičkog okruženja u kojem se odvija interakcija čoveka i računara. Razvoj korisničkog interfejsa treba da uvaži i karakteristike hardverskih uređaja koji se koriste u komunikaciji sa korisnikom, dostupne softverske resurse, kao i karakteristike programskih sistema koji treba da koriste korisnički interfejs. U skladu sa tim, uvodi se pojam kontekstno-osetljivog interfejsa koji se definiše kao korisnički interfejs koji je prilagodljiv kontekstu interakcije sa konkretnim korisnikom. Kontekst interakcije čine tri klase entiteta: korisnik računarskog sistema (čovek); hardverska i softverska platforma pomoću kojih korisnici interaguju sa sistemom i fizičko okruženje u kojem se odigrava interakcija sa sistemom. Posmatrajući evoluciju razvoja softvera uočavamo povećanje nivoa apstrakcije na kojem se softver opisuje. Dostignuti nivo razvoja omogućava platformski nezavisnu specifikaciju softvera koja se postepeno ili automatizovano prevodi u izvršne aplikacije za različite softverske i hardverske platforme. Arhitektura upravljana modelima, koja se koristi za razvoj složenih programskih rešenja, hijerarhijski organizuje koncepte i modele u više nivoa apstrakcije. Ovo je posebno bitno imajući u vidu da je razvoj kontekstno-osetljivih korisničkih interfejsa složen proces koji uključuje modelovanje velikog broja elemenata na različitim nivoima apstrakcije. U ovoj tezi smo istraživali problem unapređenja razvoja kontekstno-osetljivih korisničkih interfejsa. Predloženo je rešenje koje omogućava automatizaciju razvoja korisničkog interfejsa prilagođenog kontekstu interakcije čoveka i računara. Rešenje se ogleda u proširenju jezika za modelovanje, standardnog procesa razvoja softverskih sistema (Unified proces) i razvojnih alata elementima specifičnim za interakciju čoveka i računara. U skladu sa prethodnim, razvijen je model kontekstno-osetljive interakcije čoveka i računara i predloženi su modeli korisničkih interfejsa na različitim nivoima apstrakcije. Zbog standardizacije, široke prihvaćenosti, i dostupnosti razvojnih alata, odlučili smo se za proširenje UML (Unified Modeling Language) jezika za modelovanje i ATL (Atlas Transformation Language) jezika za transformacije modela. Primena predloženog pristupa je demonstrirana na primerima dve studije slučaja iz različitih domena...Well-designed, intuitive and catchy-to-use user interface represents key issue of success of computer products and systems. In order to improve development and usability of user interfaces it is needed to take into account user’s charasteristics. This entails interdisciplinary approach and use of knowledge from different fields such as computing, cognitive and biological sciences. In addition, it is needed to consider features of the physical environment and the medium in which interaction between human and computer takes place. Development of user interface must include characteristics of hardware devices employed in interaction with the user, availabale software resources, as well as characteristics of software systems using the interface. According to stated, concept of context-sensitive user interface is introduced, defined as a user interface adaptable to context of interaction with concrete user. Context of interaction is decomposed into three classes of entitites: user of a computer system (human); hardware and software platforms by which users interact with the system and physical environment in which interaction with system happens. Looking at the evolution of software development, we can notice that the abstraction level on which software is described is increasing all the time. The latest trend is to specify software using platform-independent models, which are then gradually and (semi-) automatically transformed into executable applications for different platforms and target devices. Modeldriven architecture, used for development of complex software solutions, hierarchically organizes concepts and models into multiple abstraction levels. This is especially important regarding development of context-sensitive user interfaces which appears to be a complex process involved with modeling of a large number of elements on different abstraction levels. In this thesis, we have been exploring problem concerned with the improvement of development of context-sensitive user interfaces. Solution enabling automation of development of user interface adaptable to context of interaction between human and computer is proposed. Solution includes extensions of modelling language, standard software development process (Unified process) and development tools with the elements specific for human-computer interaction. Based on previous, model of context-sensitive human-computer interaction has been developed and user interface models on different abstraction levels have been proposed. For reasons of standardization, wide acceptance and availability of development tools, we have decided to extend UML (Unified Modeling Language) modeling language and ATL (Atlas Transformation Language) language for model transformations. Application of the proposed approach is demonstrated with examples of two case studies from different domains..