Psychophysiology-based QoE assessment : a survey

We present a survey of psychophysiology-based assessment for quality of experience (QoE) in advanced multimedia technologies. We provide a classification of methods relevant to QoE and describe related psychological processes, experimental design considerations, and signal analysis techniques. We summarize multimodal techniques and discuss several important aspects of psychophysiology-based QoE assessment, including the synergies with psychophysical assessment and the need for standardized experimental design. This survey is not considered to be exhaustive but serves as a guideline for those interested to further explore this emerging field of research

Biosignalų požymių regos diskomfortui vertinti išskyrimas ir tyrimas

Comfortable stereoscopic perception continues to be an essential area of research. The growing interest in virtual reality content and increasing market for head-mounted displays (HMDs) still cause issues of balancing depth perception and comfortable viewing. Stereoscopic views are stimulating binocular cues – one type of several available human visual depth cues which becomes conflicting cues when stereoscopic displays are used. Depth perception by binocular cues is based on matching of image features from one retina with corresponding features from the second retina. It is known that our eyes can tolerate small amounts of retinal defocus, which is also known as Depth of Focus. When magnitudes are larger, a problem of visual discomfort arises. The research object of the doctoral dissertation is a visual discomfort level. This work aimed at the objective evaluation of visual discomfort, based on physiological signals. Different levels of disparity and the number of details in stereoscopic views in some cases make it difficult to find the focus point for comfortable depth perception quickly. During this investigation, a tendency for differences in single sensor-based electroencephalographic EEG signal activity at specific frequencies was found. Additionally, changes in eye tracker collected gaze signals were also found. A dataset of EEG and gaze signal records from 28 control subjects was collected and used for further evaluation. The dissertation consists of an introduction, three chapters and general conclusions. The first chapter reveals the fundamental knowledge ways of measuring visual discomfort based on objective and subjective methods. In the second chapter theoretical research results are presented. This research was aimed to investigate methods which use physiological signals to detect changes on the level of sense of presence. Results of the experimental research are presented in the third chapter. This research aimed to find differences in collected physiological signals when a level of visual discomfort changes. An experiment with 28 control subjects was conducted to collect these signals. The results of the thesis were published in six scientific publications – three in peer-reviewed scientific papers, three in conference proceedings. Additionally, the results of the research were presented in 8 conferences.Dissertatio

Remote Visual Observation of Real Places Through Virtual Reality Headsets

Virtual Reality has always represented a fascinating yet powerful opportunity that has attracted studies and technology developments, especially since the latest release on the market of powerful high-resolution and wide field-of-view VR headsets. While the great potential of such VR systems is common and accepted knowledge, issues remain related to how to design systems and setups capable of fully exploiting the latest hardware advances. The aim of the proposed research is to study and understand how to increase the perceived level of realism and sense of presence when remotely observing real places through VR headset displays. Hence, to produce a set of guidelines that give directions to system designers about how to optimize the display-camera setup to enhance performance, focusing on remote visual observation of real places. The outcome of this investigation represents unique knowledge that is believed to be very beneficial for better VR headset designs towards improved remote observation systems. To achieve the proposed goal, this thesis presents a thorough investigation of existing literature and previous researches, which is carried out systematically to identify the most important factors ruling realism, depth perception, comfort, and sense of presence in VR headset observation. Once identified, these factors are further discussed and assessed through a series of experiments and usability studies, based on a predefined set of research questions. More specifically, the role of familiarity with the observed place, the role of the environment characteristics shown to the viewer, and the role of the display used for the remote observation of the virtual environment are further investigated. To gain more insights, two usability studies are proposed with the aim of defining guidelines and best practices. The main outcomes from the two studies demonstrate that test users can experience an enhanced realistic observation when natural features, higher resolution displays, natural illumination, and high image contrast are used in Mobile VR. In terms of comfort, simple scene layouts and relaxing environments are considered ideal to reduce visual fatigue and eye strain. Furthermore, sense of presence increases when observed environments induce strong emotions, and depth perception improves in VR when several monocular cues such as lights and shadows are combined with binocular depth cues. Based on these results, this investigation then presents a focused evaluation on the outcomes and introduces an innovative eye-adapted High Dynamic Range (HDR) approach, which the author believes to be of great improvement in the context of remote observation when combined with eye-tracked VR headsets. Within this purpose, a third user study is proposed to compare static HDR and eye-adapted HDR observation in VR, to assess that the latter can improve realism, depth perception, sense of presence, and in certain cases even comfort. Results from this last study confirmed the author expectations, proving that eye-adapted HDR and eye tracking should be used to achieve best visual performances for remote observation in modern VR systems

Modelling the use of 3D video on the quality of experience

Последњих година, очигледан је брз развој различитих медија у различитим сферама као што су потрошачка електроника, аутомобилска инфо-забава (енгл. Infotainment), софтверa у сврху здравства итд. Због тога намеће се потреба за иновативним методама процене квалитета доживљаја (енгл. Quality of Experience - QoE) које корисници доживљавају као замену за задовољство потрошача таквих системима и услугама. Емоционално стање корисника игра кључну улогу у области QoE; стога га је неопходно узети у обзир приликом процене корисничког искуства и процеса дизајнирања 3Д видео садржаја. У овој докторској дисертацији представљено је моделовање проценитеља квалитета доживљава заснованог на повратној вишеслојној вештачкој неуронској мрежи као одговарајућој техници машинског учења за процену човековог емоционалног стања током гледања различитих типова 3Д видео садржаја. Циљ је дизајнирање проценитеља емоционалног стања на основу директних психо-физиолошких мерења. Разматрани психо- физиолошки сигнали укључују срчану фреквенцију (HR) израчунату на основу ехо-кардиограма (ECG), електро-дермалну активност (EDA) и активност мозга (BA) у електро-енцефалографским (EEG) сигналима. Експериментални део истраживања постављен је тако да су учесници гледали серију 3Д видео садржаја који се разликују у погледу визуелног квалитета и типа садржаја, док су поменути психо-физиолошки сигнали забележени помоћу специјалних сонди постављених у моменту гледања садржаја, а субјективно проживљене емоције пријављене помоћу упитника за самопроцену (SAM). Добијени резултати показују да је могуће конструисати тако високо прецизан процењивач емоционалних стања.Poslednjih godina, očigledan je brz razvoj različitih medija u različitim sferama kao što su potrošačka elektronika, automobilska info-zabava (engl. Infotainment), softvera u svrhu zdravstva itd. Zbog toga nameće se potreba za inovativnim metodama procene kvaliteta doživljaja (engl. Quality of Experience - QoE) koje korisnici doživljavaju kao zamenu za zadovoljstvo potrošača takvih sistemima i uslugama. Emocionalno stanje korisnika igra ključnu ulogu u oblasti QoE; stoga ga je neophodno uzeti u obzir prilikom procene korisničkog iskustva i procesa dizajniranja 3D video sadržaja. U ovoj doktorskoj disertaciji predstavljeno je modelovanje procenitelja kvaliteta doživljava zasnovanog na povratnoj višeslojnoj veštačkoj neuronskoj mreži kao odgovarajućoj tehnici mašinskog učenja za procenu čovekovog emocionalnog stanja tokom gledanja različitih tipova 3D video sadržaja. Cilj je dizajniranje procenitelja emocionalnog stanja na osnovu direktnih psiho-fizioloških merenja. Razmatrani psiho- fiziološki signali uključuju srčanu frekvenciju (HR) izračunatu na osnovu eho-kardiograma (ECG), elektro-dermalnu aktivnost (EDA) i aktivnost mozga (BA) u elektro-encefalografskim (EEG) signalima. Eksperimentalni deo istraživanja postavljen je tako da su učesnici gledali seriju 3D video sadržaja koji se razlikuju u pogledu vizuelnog kvaliteta i tipa sadržaja, dok su pomenuti psiho-fiziološki signali zabeleženi pomoću specijalnih sondi postavljenih u momentu gledanja sadržaja, a subjektivno proživljene emocije prijavljene pomoću upitnika za samoprocenu (SAM). Dobijeni rezultati pokazuju da je moguće konstruisati tako visoko precizan procenjivač emocionalnih stanja.In recent years, the rapid development of diverse media has been evident in disparate fields such as consumer electronics, automotive infotainment and healthcare software. There is a need for innovative methods to assess user perceived Quality of Experience (QoE), as a proxy for consumer satisfaction with such systems and services. Users emotional state plays a key role in QoE; thus, it is necessary to consider it in user experience evaluation and the design process of stereoscopic 3D video content. In the PhD thesis the use of a specially designed model based on a feedforward Multilayer Perception Artificial Neural Network as an appropriate Machine Learning technique for the estimation of human emotional state while viewing various categories of stereoscopic 3D video content is introduced. The goal is to design an emotional state estimator based on direct psychophysiological measurements. The considered psychophysiological signals include heart rate (HR) calculated from an echocardiogram (ECG), electro-dermal activity (EDA), and brain activity (BA) in EEG signals. In the experimental part of study, participants watched a series of 3D video contents varying in terms of visual quality and type of content, while the mentioned psychophysiological signals were recorded via specific equipment, and self-reported subjectively experienced emotions using a Self-Assessment Manikin (SAM) questionnaire. The obtained results show that it is possible to construct such a highly precise estimator of emotional state

Presence, what is it good for? Exploring the benefits of virtual reality at evoking empathy towards the marginalized

2020 Fall.Includes bibliographical references.This study examines the relationship between physical presence and empathy across three technology modalities: 1) virtual reality head-mount displays, 2) desktop virtual reality, and 3) text narratives with photographs displayed on a computer screen. Additionally, it examines if public support for a novel public health intervention increases when participants engage in a perspective-taking exercise designed to evoke empathy. Last, the study explores whether the benefits of empathy arousal, specifically the reduction of stereotypes toward the marginalized, depend on the technology modality used in the perspective-taking exercise. Prior studies have consistently found a positive correlation between physical presence and fear and anxiety, especially studies that have used virtual reality head-mount displays to induce presence. However, few studies have examined the relationship between physical presence and empathy. Although some studies have found a positive correlation between physical presence and empathy, these studies are few, lack comprehensive and consistent measurement, and commonly do not test the superiority of virtual reality head-mount displays at evoking empathy against more traditional technology modalities. Last, studies using virtual reality head-mount displays have found inconsistent results in how empathy affects public support and stereotypes. A 1x4 lab experiment (N = 199) was carried out to fill in these research gaps. Results include the follow: 1) physical presence was higher in the virtual reality head-mount display condition compared to the desktop virtual-reality condition and the text narrative and photograph condition; 2) physical presence was positively correlated with all four dimensions of empathy—perspective taking, fantasy, personal distress, and empathic concern; however, the relationship between presence and empathic concern was moderated by participants' mental health; 3) the amount of empathy participants experienced did not differ by experimental condition; however, cognitive empathy was lower in the control condition compared to each experimental condition; 4) public support was positively correlated with three of the four dimensions of empathy including perspective taking, fantasy, and empathic concern; 5) perceptions of stereotypes of people who inject drugs were higher in the control condition compared to the desktop virtual-reality condition and text narrative condition, but not the virtual reality head-mount display condition. Overall, this study adds to a growing body of literature exploring the benefits of virtual-reality perspective-taking exercises in three important ways. First, this study strengthens the assertion that virtual-reality head-mount displays produce more physical presence compared to desktop virtual reality and text narratives with photographs. Second, aligned with prior research, this study provides evidence of a positive correlation between physical presence and empathy arousal. However, in this study, empathy arousal appears to be increasing presence, which is a different causal pathway then the study predicted.Last, this study found that the virtual-reality head-mount display condition was the only experimental condition that did not significantly reduce stereotypes. Together, these results suggest both potential advantages and disadvantages for using virtual reality in perspective-taking exercises

Curved Displays, Empirical Horopters, and Ergonomic Design Guidelines

Department of Human Factors EngineeringVisual display products should be comprehensively evaluated from the perspectives of productivity, safety, and well-being. Curved display products are known to provide advantages. Although previous studies found that curved displays increase visual task performance, reduce visual fatigue, and improve the watching experience, these studies did not comprehensively examine the effects of display curvature. Moreover, they used low-fidelity curved screens that may not effectively reflect actual curved displays. The purpose of this thesis was to develop ergonomic design guidelines for determining appropriate display curvatures, considering the productivity, safety, and well-being of visual display terminal (VDT) users. Two studies on monitors and one study on TVs were conducted for this goal. In Study 1, the effects of the display curvature, display zone, and task duration on visual task performance and visual fatigue during a visual search task on a 50-inch multi-monitor were investigated. In Study 2, the effects of the display curvature and task duration on visual task performance, visual fatigue, and user satisfaction during a proofreading task on a 27-inch monitor were investigated, and the associations between ergonomic evaluation elements were then examined. Prediction models of visual fatigue and user satisfaction were subsequently developed. In Study 3, the effects of the display curvature, viewing distance, and lateral viewing position on presence, visual comfort, and user satisfaction during a TV watching task on a 55-inch TV were examined, and the importance of six viewing experience elements affecting user satisfaction was revealed. Finally, ergonomic design guidelines for curved displays were suggested. Based on the results of studies 1 and 2, an appropriate rest-break time was ecommended, taking into account visual task performance and visual fatigue. Study 1 examined the effects of the display curvature (400 R, 600 R, 1200 R, and flat), display zone (five zones), and task duration (15 and 30 min) on legibility and visual fatigue. A total of 27 participants completed two sets of 15-minute visual search tasks with each curvature setting. The 600 R and 1200 R settings yielded better results compared to the flat setup regarding legibility and perceived visual fatigue. Relative to the corresponding center zone, the outermost zones of the 1200 R and flat settings showed a decrease of 8%???37% in legibility, whereas those of the flat environment showed an increase of 26%???45% in perceived visual fatigue. Across curvatures, legibility decreased by 2%???8%, whereas perceived visual fatigue increased by 22% during the second task set. The two task sets showed an increase of 102% in the eye complaint score and a decrease of 0.3 Hz in the critical fusion frequency, both of which indicated a rise in visual fatigue. To sum up, a curvature of around 600 R, central display zones, and frequent breaks were recommended to improve legibility and reduce visual fatigue. Study 2 examined the effects of the display curvature and task duration on proofreading performance, visual discomfort, visual fatigue, mental workload, and user satisfaction. Fifty individuals completed four 15-min proofreading tasks at a particular curvature setting. Five display curvatures (600 R, 1140 R, 2000 R, 4000 R and flat) and five task durations (0, 15, 30, 45, and 60 min) were incorporated. The mean proofreading speed at its highest when the display curvature radius was equal to the viewing distance (600 R). Across curvatures, speedaccuracy tradeoffs occurred with proofreading, as indicated by an increase of 15.5% in its mean speed and a decrease of 22.3% in its mean accuracy over one hour. Meanwhile, the mean perceived visual discomfort, subjective visual fatigue, and mental workload increased, by 54%, 74%, and 24% respectively, during the first 15-min of proofreading. A decrease of 0.4 Hz in the mean critical fusion frequency during the first 15 min and a reduction in the mean blink frequency also indicated increases in visual fatigue and mental workload. The mean user satisfaction decreased by 11% until 45 min. A segmented regression model, in which perceived visual discomfort was used as a predictor, attributed 51% of the variability to visual fatigue. To sum up, a curvature of 600 R was recommended for speedy proofreading. Moreover, the breakpoint was observed be flexible, depending on VDT task types. These findings can contribute to determining ergonomic display curvatures and scheduling interim breaks for speedy but less visually fatiguing proofreading. Study 3 examined the effects of the display curvature, viewing distance, and lateral viewing position on the TV watching experience. The watching experience was assessed regarding the spatial presence, engagement, ecological validity, negative effects, visual comfort, image quality, and display satisfaction. Four display curvatures (2.3 m, 4 m, 6 m, and flat), two viewing distances (2.3 m and 4 m), and five lateral viewing positions (0 cm, 35 cm, 70 cm, 105 cm, and 140 cm) were evaluated. Seven pairs of individuals per curvature watched ten 5 min videos together, each time at a different viewing distance and lateral viewing position. Spatial presence and engagement increased when the display curvature approached the given viewing distance. Regardless of display curvature and viewing distance and TV watching experience factors, except negative effects, were degraded at more lateral viewing positions. Engagement could effectively explain the display satisfaction. These findings can contribute to enhancing TV watching experiences by recommending specific levels of display curvatures, viewing distances, and lateral viewing positions, as well as providing information on the relative importance of each watching experience element. This work suggested ergonomic design guidelines for curved displays. In Study 1, a curvature of approximately 600 R, central display zone, and frequent breaks were proposed to improve legibility and reduce visual fatigue during visual search tasks at the viewing distance of 500 mm. In Study 2, a curvature radius of 600 R and a minimum 15-minute break interval were proposed for a speedy proofreading task, at the viewing distance of 600 mm. In Study 3, a display radius of curvature similar to the viewing distance was recommended to improve the viewing experience. These results support that a curved display is ergonomically more beneficial when the display curvature approaches the empirical horopter. A relatively short 15-minute rest-time interval was suggested, considering the decrease of task accuracy and the increase of visual fatigue in studies 1 and 2. Two regression models were selected in Study 2 regarding predictive accuracy. They accounted for 70.4% of subjective visual fatigue variability and 60.2% of user satisfaction variability. Although this work was performed using relatively higher-fidelity mock-ups than previous studies, it is necessary to verify the findings with actual curved display products in the future. Furthermore, various tasks (e.g., word processing, graphics design, and gaming) and personal characteristics (e.g., presbyopia, gender, visual acuity, and product experience) should be considered to generalize the results of this thesis. These results can contribute to determining the ergonomic display curvature in consideration of productivity, safety, and well-being, and prioritizing elements of the visual fatigue and user satisfaction resulting from VDT work.ope

Tele-proximity: The Experienced Educator Perspective on Human to Human Connection in Distance Education.

Distance education is an integral part of many universities worldwide, aiming to offer students opportunities for life-long learning and equitable access. Distant learners face many barriers and as a result, they drop out more frequently than on-campus students. Educators seem to be equally affected by the 'transactional distance' and the new digital skills needed for teaching online. The purpose of this study is to explore the educators' perspective on how synchronous video communication (SVC) could offer alternative educational forms for distance learning. Specifically the study aims to fill gaps in the existing research literature: to describe synchronous teaching approaches that are used worldwide to support e-learners, to investigate the role of the educator, to discover how learning and subject content are affected by SVC, to see if it enhances the sense of togetherness (immediacy and intimacy) to specify the contextual factors influencing teaching and learning synchronously, and to formulate a theory. The Informed Grounded Theory (Thornberg, 2012) and the Community of Inquiry model (Garrison, Anderson & Archer, 2000) provided a scaffolding framework for designing interview questions and analysing findings. The findings of the study illustrate the empirical evidence of the value of SVC, its potential forms (teaching approaches), possibilities (linking educational purposes to the medium) and contextual factors (University policy, time zones, technologies, learning objectives, educator and student identities). Specifically, audiovisual cues have an impact on educators, the learning process and immediacy. The theory of Tele-proximity formulated to explain how educators and students are connected in synchronous networked environment via tele-operations. The study aims at helping educators/instructional designers, and administrators to face the difficulties of transactional distance and make informed decisions about synchronous video enhanced communication. Researchers may use Tele-proximity as a heuristic tool to continue the scientific dialogue on the potential of synchronous video-enhanced technologies in distance education

Améliorer les interactions homme-machine et la présence sociale avec l’informatique physiologique

This thesis explores how physiological computing can contribute to human-computer interaction (HCI) and foster new communication channels among the general public. We investigated how physiological sensors, such as electroencephalography (EEG), could be employed to assess the mental state of the users and how they relate to other evaluation methods. We created the first brain-computer interface that could sense visual comfort during the viewing of stereoscopic images and shaped a framework that could help to assess the over all user experience by monitoring workload, attention and error recognition.To lower the barrier between end users and physiological sensors,we participated in the software integration of a low-cost and open hardware EEG device; used off-the shelf webcams to measure heart rate remotely, crafted we arables that can quickly equip users so that electrocardiography, electrodermal activity or EEG may be measured during public exhibitions. We envisioned new usages for our sensors, that would increase social presence. In a study about human-agent interaction, participants tended to prefer virtual avatars that were mirroring their own internal state. A follow-up study focused on interactions between users to describe how physiological monitoringcould alter our relationships. Advances in HCI enabled us to seam lesslyintegrate biofeedback to the physical world. We developped Teegi, apuppet that lets novices discover by themselves about their brain activity. Finally, with Tobe, a toolkit that encompasses more sensors and give more freedom about their visualizations, we explored how such proxy shifts our representations, about our selves as well as about the others.Cette thèse explore comment l’informatique physiologique peut contribuer aux interactions homme-machine (IHM) et encourager l’apparition de nouveaux canaux de communication parmi le grand public. Nous avons examiné comment des capteurs physiologiques,tels que l’électroencéphalographie (EEG), pourraient être utilisés afin d’estimer l’état mental des utilisateurs et comment ils se positionnent par rapport à d’autres méthodes d’évaluation. Nous avons créé la première interface cerveau-ordinateur capable de discriminer le confort visuel pendant le visionnage d’images stéréoscopiques et nous avons esquissé un système qui peux aider à estimer l’expérience utilisateur dans son ensemble, en mesurant charge mentale, attention et reconnaissance d’erreur. Pour abaisser la barrière entre utilisateurs finaux et capteurs physiologiques, nous avons participé à l’intégration logicielle d’un appareil EEG bon marché et libre, nous avons utilisé des webcams du commerce pour mesurer le rythme cardiaque à distance, nous avons confectionné des wearables dont les utilisateurs peuvent rapidement s’équiper afin qu’électrocardiographie, activité électrodermale et EEG puissent être mesurées lors de manifestations publiques. Nous avons imaginé de nouveaux usages pour nos capteurs, qui augmenteraient la présence sociale. Dans une étude autour de l’interaction humain agent,les participants avaient tendance à préférer les avatars virtuels répliquant leurs propres états internes. Une étude ultérieure s’est concentrée sur l’interaction entre utilisateurs, profitant d’un jeu de plateau pour décrire comment l’examen de la physiologie pourrait changer nos rapports. Des avancées en IHM ont permis d’intégrer de manière transparente du biofeedback au monde physique. Nous avons développé Teegi, une poupée qui permet aux novices d’en découvrir plus sur leur activité cérébrale, par eux-mêmes. Enfin avec Tobe, un toolkit qui comprend plus de capteurs et donne plus de liberté quant à leurs visualisations, nous avons exploré comment un tel proxy décalenos représentations, tant de nous-mêmes que des autres

How visual discomfort affects 3DTV viewers' emotional arousal

International audienceWe present the results of a research on the effects of image distor-tions known to induce visual discomfort (namely, the introduction of Gaussian blur on one visual field) on the emotional reactions of 3DTV viewers. Emotional reactions were explored by means of self-reported as well as psychophysiological methods. The re-sults showed that, despite visual discomfort not having an effect on self-reported emotions, the level of electrodermal activity (a physiological correlate of emotional arousal) was clearly affected, suggesting the existence of seemingly complex effects of visual discomfort over viewers' emotional arousal

Towards the use of visual masking within virtual environments to induce changes in affective cognition

This thesis concerns the use of virtual environments for psychotherapy. It makes use of virtual environment properties that go beyond real-world simulation. The core technique used is based on research found within perception science, an effect known as backwards visual masking. Here, a rapidly displayed target image is rendered explicitly imperceptible via the subsequent display of a masking image. The aim of this thesis was to investigate the potential of visual masking within virtual environments to induce changes in affective cognition. Of particular importance would be changes in a positive direction as this could form the foundation of a psychotherapeutic tool to treat affect disorders and other conditions with an affective component. The initial pair of experiments looked at whether visual masking was possible within virtual environments, whether any measurable behavioural influence could be found and whether there was any evidence that affective cognitions could be influenced. It was found that the technique worked and could influence both behaviour and affective cognition. Following this, two experiments looked further at parameter manipulation of visual masking within virtual environments with the aim of better specifying the parameter values. Results indicated that the form of visual masking used worked better in a virtual environment when the target and mask were both highly textured and that affective effects were modulated by the number of exposures of the target. The final pair of experiments attempted to induce an affect contagion effect and an affect cognition-modification effect. An affect cognition-modification effect was found whereas an affect contagion effect was not. Overall, the results show that using visual masking techniques within virtual environments to induce affect cognition changes has merit. The thesis lays the foundation for further work and supports the use of this technique as basis of an intervention tool