Psychophysiology-based QoE assessment : a survey

We present a survey of psychophysiology-based assessment for quality of experience (QoE) in advanced multimedia technologies. We provide a classification of methods relevant to QoE and describe related psychological processes, experimental design considerations, and signal analysis techniques. We summarize multimodal techniques and discuss several important aspects of psychophysiology-based QoE assessment, including the synergies with psychophysical assessment and the need for standardized experimental design. This survey is not considered to be exhaustive but serves as a guideline for those interested to further explore this emerging field of research

Biosignalų požymių regos diskomfortui vertinti išskyrimas ir tyrimas

Comfortable stereoscopic perception continues to be an essential area of research. The growing interest in virtual reality content and increasing market for head-mounted displays (HMDs) still cause issues of balancing depth perception and comfortable viewing. Stereoscopic views are stimulating binocular cues – one type of several available human visual depth cues which becomes conflicting cues when stereoscopic displays are used. Depth perception by binocular cues is based on matching of image features from one retina with corresponding features from the second retina. It is known that our eyes can tolerate small amounts of retinal defocus, which is also known as Depth of Focus. When magnitudes are larger, a problem of visual discomfort arises. The research object of the doctoral dissertation is a visual discomfort level. This work aimed at the objective evaluation of visual discomfort, based on physiological signals. Different levels of disparity and the number of details in stereoscopic views in some cases make it difficult to find the focus point for comfortable depth perception quickly. During this investigation, a tendency for differences in single sensor-based electroencephalographic EEG signal activity at specific frequencies was found. Additionally, changes in eye tracker collected gaze signals were also found. A dataset of EEG and gaze signal records from 28 control subjects was collected and used for further evaluation. The dissertation consists of an introduction, three chapters and general conclusions. The first chapter reveals the fundamental knowledge ways of measuring visual discomfort based on objective and subjective methods. In the second chapter theoretical research results are presented. This research was aimed to investigate methods which use physiological signals to detect changes on the level of sense of presence. Results of the experimental research are presented in the third chapter. This research aimed to find differences in collected physiological signals when a level of visual discomfort changes. An experiment with 28 control subjects was conducted to collect these signals. The results of the thesis were published in six scientific publications – three in peer-reviewed scientific papers, three in conference proceedings. Additionally, the results of the research were presented in 8 conferences.Dissertatio

Modelling the use of 3D video on the quality of experience

Последњих година, очигледан је брз развој различитих медија у различитим сферама као што су потрошачка електроника, аутомобилска инфо-забава (енгл. Infotainment), софтверa у сврху здравства итд. Због тога намеће се потреба за иновативним методама процене квалитета доживљаја (енгл. Quality of Experience - QoE) које корисници доживљавају као замену за задовољство потрошача таквих системима и услугама. Емоционално стање корисника игра кључну улогу у области QoE; стога га је неопходно узети у обзир приликом процене корисничког искуства и процеса дизајнирања 3Д видео садржаја. У овој докторској дисертацији представљено је моделовање проценитеља квалитета доживљава заснованог на повратној вишеслојној вештачкој неуронској мрежи као одговарајућој техници машинског учења за процену човековог емоционалног стања током гледања различитих типова 3Д видео садржаја. Циљ је дизајнирање проценитеља емоционалног стања на основу директних психо-физиолошких мерења. Разматрани психо- физиолошки сигнали укључују срчану фреквенцију (HR) израчунату на основу ехо-кардиограма (ECG), електро-дермалну активност (EDA) и активност мозга (BA) у електро-енцефалографским (EEG) сигналима. Експериментални део истраживања постављен је тако да су учесници гледали серију 3Д видео садржаја који се разликују у погледу визуелног квалитета и типа садржаја, док су поменути психо-физиолошки сигнали забележени помоћу специјалних сонди постављених у моменту гледања садржаја, а субјективно проживљене емоције пријављене помоћу упитника за самопроцену (SAM). Добијени резултати показују да је могуће конструисати тако високо прецизан процењивач емоционалних стања.Poslednjih godina, očigledan je brz razvoj različitih medija u različitim sferama kao što su potrošačka elektronika, automobilska info-zabava (engl. Infotainment), softvera u svrhu zdravstva itd. Zbog toga nameće se potreba za inovativnim metodama procene kvaliteta doživljaja (engl. Quality of Experience - QoE) koje korisnici doživljavaju kao zamenu za zadovoljstvo potrošača takvih sistemima i uslugama. Emocionalno stanje korisnika igra ključnu ulogu u oblasti QoE; stoga ga je neophodno uzeti u obzir prilikom procene korisničkog iskustva i procesa dizajniranja 3D video sadržaja. U ovoj doktorskoj disertaciji predstavljeno je modelovanje procenitelja kvaliteta doživljava zasnovanog na povratnoj višeslojnoj veštačkoj neuronskoj mreži kao odgovarajućoj tehnici mašinskog učenja za procenu čovekovog emocionalnog stanja tokom gledanja različitih tipova 3D video sadržaja. Cilj je dizajniranje procenitelja emocionalnog stanja na osnovu direktnih psiho-fizioloških merenja. Razmatrani psiho- fiziološki signali uključuju srčanu frekvenciju (HR) izračunatu na osnovu eho-kardiograma (ECG), elektro-dermalnu aktivnost (EDA) i aktivnost mozga (BA) u elektro-encefalografskim (EEG) signalima. Eksperimentalni deo istraživanja postavljen je tako da su učesnici gledali seriju 3D video sadržaja koji se razlikuju u pogledu vizuelnog kvaliteta i tipa sadržaja, dok su pomenuti psiho-fiziološki signali zabeleženi pomoću specijalnih sondi postavljenih u momentu gledanja sadržaja, a subjektivno proživljene emocije prijavljene pomoću upitnika za samoprocenu (SAM). Dobijeni rezultati pokazuju da je moguće konstruisati tako visoko precizan procenjivač emocionalnih stanja.In recent years, the rapid development of diverse media has been evident in disparate fields such as consumer electronics, automotive infotainment and healthcare software. There is a need for innovative methods to assess user perceived Quality of Experience (QoE), as a proxy for consumer satisfaction with such systems and services. Users emotional state plays a key role in QoE; thus, it is necessary to consider it in user experience evaluation and the design process of stereoscopic 3D video content. In the PhD thesis the use of a specially designed model based on a feedforward Multilayer Perception Artificial Neural Network as an appropriate Machine Learning technique for the estimation of human emotional state while viewing various categories of stereoscopic 3D video content is introduced. The goal is to design an emotional state estimator based on direct psychophysiological measurements. The considered psychophysiological signals include heart rate (HR) calculated from an echocardiogram (ECG), electro-dermal activity (EDA), and brain activity (BA) in EEG signals. In the experimental part of study, participants watched a series of 3D video contents varying in terms of visual quality and type of content, while the mentioned psychophysiological signals were recorded via specific equipment, and self-reported subjectively experienced emotions using a Self-Assessment Manikin (SAM) questionnaire. The obtained results show that it is possible to construct such a highly precise estimator of emotional state

Assessment of Quality of Experience of High Dynamic Range Images Using the EEG and Applications in Healthcare

File embargoed until 30.09.2021 at author's request.Recent years have witnessed the widespread application of High Dynamic Range (HDR) imaging, which like the Human Visual System (HVS), has the ability to capture a wide range of luminance values. Areas of application include home-entertainment, security, scientific imaging, video processing, computer graphics, multimedia communications, and healthcare. However, in practice, HDR content cannot be displayed in full on standard or low dynamic range (LDR) displays, and this diminishes the benefits of HDR technology for many users. To address this problem, Tone-Mapping Operators (TMO) are used to convert HDR images so that they can be displayed on low-dynamic-range displays and preserve as far as possible the perception of HDR. However, this may affect the visual Quality of Experience (QoE) of the end-user. QoE is a vital issue in image and video applications. It is important to understand how humans perceive quality in response to visual stimuli as this can potentially be exploited to develop and optimise image and video processing algorithms. Image consumption using mobile devices has become increasingly popular, given the availability of smartphones capable of producing and consuming HDR images along with advances in high-speed wireless communication networks. One of the most critical issues associated with mobile HDR image delivery services concerns how to maximise the QoE of the delivered content for users. An open research question therefore addresses how HDR images with different types of content perform on mobile phones. Traditionally, evaluation of the perceived quality of multimedia content is conducted using subjective opinion tests (i.e., explicitly), such as Mean Opinion Scores (MOS). However, it is difficult for the user to link the quality they are experiencing to the quality scale. Moreover, MOS does not give an insight into how the user feels at a physiological level in response to satisfaction or dissatisfaction with the perceived quality. To address this issue, measures that can be taken directly (implicitly) from the participant have now begun to attract interest. The electroencephalogram (EEG) is a promising approach that can be used to assess quality related processes implicitly. However, implicit QoE approaches are still at an early stage and further research is necessary to fully understand the nature of the recorded neural signals and their associations with user-perceived quality. Nevertheless, the EEG is expected to provide additional and complementary information that will aid understanding of the human perception of content. Furthermore, it has the potential to facilitate real-time monitoring of QoE without the need for explicit rating activities. The main aim of this project was therefore to assess the QoE of HDR images employing a physiological method and to investigate its potential application in the field of healthcare. This resulted in the following five main contributions to the research literature: 1. A detailed understanding of the relationship between the subjective and objective evaluation of the most popular TMOs used for colour and greyscale HDR images. Different mobile displays and resolutions were therefore presented under normal viewing conditions for the end-user with an LDR display as a reference. Preliminary results show that, compared to computer displays, small screen devices (SSDs) such as those used in smartphones impact the performance of TMOs in that a higher resolution gave more favourable MOS results. 2. The development of a novel Electrophysiology-based QoE assessment of HDR image quality that can be used to predict perceived image quality. This was achieved by investigating the relationships between changes in EEG features and subjective quality test scores (i.e. MOS) for HDR images viewed with SSD. 3. The development of a novel QoE prediction model, based on the above findings. The model can predict user acceptability and satisfaction for various mobile HDR image scenarios based on delta-beta coupling. Subjective quality tests were conducted to develop and evaluate the model, where the HDR image quality was predicted in terms of MOS. 4. The development of a new method of detecting a colour vision deficiency (CVD) using EEG and HDR images. The results suggest that this method may provide an accurate way to detect CVD with high sensitivity and specificity (close to 100%). Potentially, the method may facilitate the development of a low-cost tool suitable for CVD diagnosis in younger people. 5. The development of an approach that enhances the quality of dental x-ray images. This uses the concepts of QoE in HDR images without re-exposing patients to ionising radiation, thus improving patient care. Potentially, the method provides the basis for an intelligent model that accurately predicts the quality of dental images. Such a model can be embedded into a tool to automatically enhance poor quality dental images.Ministry of Higher Education and Scientific Research (MoHESR

Curved Displays, Empirical Horopters, and Ergonomic Design Guidelines

Department of Human Factors EngineeringVisual display products should be comprehensively evaluated from the perspectives of productivity, safety, and well-being. Curved display products are known to provide advantages. Although previous studies found that curved displays increase visual task performance, reduce visual fatigue, and improve the watching experience, these studies did not comprehensively examine the effects of display curvature. Moreover, they used low-fidelity curved screens that may not effectively reflect actual curved displays. The purpose of this thesis was to develop ergonomic design guidelines for determining appropriate display curvatures, considering the productivity, safety, and well-being of visual display terminal (VDT) users. Two studies on monitors and one study on TVs were conducted for this goal. In Study 1, the effects of the display curvature, display zone, and task duration on visual task performance and visual fatigue during a visual search task on a 50-inch multi-monitor were investigated. In Study 2, the effects of the display curvature and task duration on visual task performance, visual fatigue, and user satisfaction during a proofreading task on a 27-inch monitor were investigated, and the associations between ergonomic evaluation elements were then examined. Prediction models of visual fatigue and user satisfaction were subsequently developed. In Study 3, the effects of the display curvature, viewing distance, and lateral viewing position on presence, visual comfort, and user satisfaction during a TV watching task on a 55-inch TV were examined, and the importance of six viewing experience elements affecting user satisfaction was revealed. Finally, ergonomic design guidelines for curved displays were suggested. Based on the results of studies 1 and 2, an appropriate rest-break time was ecommended, taking into account visual task performance and visual fatigue. Study 1 examined the effects of the display curvature (400 R, 600 R, 1200 R, and flat), display zone (five zones), and task duration (15 and 30 min) on legibility and visual fatigue. A total of 27 participants completed two sets of 15-minute visual search tasks with each curvature setting. The 600 R and 1200 R settings yielded better results compared to the flat setup regarding legibility and perceived visual fatigue. Relative to the corresponding center zone, the outermost zones of the 1200 R and flat settings showed a decrease of 8%???37% in legibility, whereas those of the flat environment showed an increase of 26%???45% in perceived visual fatigue. Across curvatures, legibility decreased by 2%???8%, whereas perceived visual fatigue increased by 22% during the second task set. The two task sets showed an increase of 102% in the eye complaint score and a decrease of 0.3 Hz in the critical fusion frequency, both of which indicated a rise in visual fatigue. To sum up, a curvature of around 600 R, central display zones, and frequent breaks were recommended to improve legibility and reduce visual fatigue. Study 2 examined the effects of the display curvature and task duration on proofreading performance, visual discomfort, visual fatigue, mental workload, and user satisfaction. Fifty individuals completed four 15-min proofreading tasks at a particular curvature setting. Five display curvatures (600 R, 1140 R, 2000 R, 4000 R and flat) and five task durations (0, 15, 30, 45, and 60 min) were incorporated. The mean proofreading speed at its highest when the display curvature radius was equal to the viewing distance (600 R). Across curvatures, speedaccuracy tradeoffs occurred with proofreading, as indicated by an increase of 15.5% in its mean speed and a decrease of 22.3% in its mean accuracy over one hour. Meanwhile, the mean perceived visual discomfort, subjective visual fatigue, and mental workload increased, by 54%, 74%, and 24% respectively, during the first 15-min of proofreading. A decrease of 0.4 Hz in the mean critical fusion frequency during the first 15 min and a reduction in the mean blink frequency also indicated increases in visual fatigue and mental workload. The mean user satisfaction decreased by 11% until 45 min. A segmented regression model, in which perceived visual discomfort was used as a predictor, attributed 51% of the variability to visual fatigue. To sum up, a curvature of 600 R was recommended for speedy proofreading. Moreover, the breakpoint was observed be flexible, depending on VDT task types. These findings can contribute to determining ergonomic display curvatures and scheduling interim breaks for speedy but less visually fatiguing proofreading. Study 3 examined the effects of the display curvature, viewing distance, and lateral viewing position on the TV watching experience. The watching experience was assessed regarding the spatial presence, engagement, ecological validity, negative effects, visual comfort, image quality, and display satisfaction. Four display curvatures (2.3 m, 4 m, 6 m, and flat), two viewing distances (2.3 m and 4 m), and five lateral viewing positions (0 cm, 35 cm, 70 cm, 105 cm, and 140 cm) were evaluated. Seven pairs of individuals per curvature watched ten 5 min videos together, each time at a different viewing distance and lateral viewing position. Spatial presence and engagement increased when the display curvature approached the given viewing distance. Regardless of display curvature and viewing distance and TV watching experience factors, except negative effects, were degraded at more lateral viewing positions. Engagement could effectively explain the display satisfaction. These findings can contribute to enhancing TV watching experiences by recommending specific levels of display curvatures, viewing distances, and lateral viewing positions, as well as providing information on the relative importance of each watching experience element. This work suggested ergonomic design guidelines for curved displays. In Study 1, a curvature of approximately 600 R, central display zone, and frequent breaks were proposed to improve legibility and reduce visual fatigue during visual search tasks at the viewing distance of 500 mm. In Study 2, a curvature radius of 600 R and a minimum 15-minute break interval were proposed for a speedy proofreading task, at the viewing distance of 600 mm. In Study 3, a display radius of curvature similar to the viewing distance was recommended to improve the viewing experience. These results support that a curved display is ergonomically more beneficial when the display curvature approaches the empirical horopter. A relatively short 15-minute rest-time interval was suggested, considering the decrease of task accuracy and the increase of visual fatigue in studies 1 and 2. Two regression models were selected in Study 2 regarding predictive accuracy. They accounted for 70.4% of subjective visual fatigue variability and 60.2% of user satisfaction variability. Although this work was performed using relatively higher-fidelity mock-ups than previous studies, it is necessary to verify the findings with actual curved display products in the future. Furthermore, various tasks (e.g., word processing, graphics design, and gaming) and personal characteristics (e.g., presbyopia, gender, visual acuity, and product experience) should be considered to generalize the results of this thesis. These results can contribute to determining the ergonomic display curvature in consideration of productivity, safety, and well-being, and prioritizing elements of the visual fatigue and user satisfaction resulting from VDT work.ope

Contributions to the Development of Objective Techniques for Presence Measurement in Virtual Environments by means of Brain Activity Analysis

En esta tesis, se propone el uso de la técnica de Doppler transcraneal (DTC) para monitorizar la actividad cerebral durante la exposición a entornos virtuales (EV) y así poder analizar los correlatos cerebrales del sentido de presencia. Las hipótesis de partida son las siguientes: 1) DTC se podrá utilizar fácilmente en combinación con sistemas de realidad virtual. 2) Los datos de velocidad de flujo sanguíneo medidos por DTC se podrán utilizar para analizar cambios de actividad cerebral durante la exposición a EV. 3) Habrá diferencias en la velocidad del flujo sanguíneo asociadas a distintos niveles de presencia. 4) Habrá correlación entre el grado de presencia medido por cuestionarios y parámetros de la velocidad de flujo sanguíneo. 5) Cada componente de la experiencia virtual tendrá una influencia en las variaciones de velocidad observadas. Para analizar las hipótesis planteadas, se realizaron cuatro experimentos distintos, en los que se analizó la velocidad del flujo sanguíneo durante: 1) distintas condiciones de navegación, 2) distintas condiciones de inmersión, 3) una tarea de percepción visual y 4) tareas motoras para manejo de un joystick. Durante la tesis, se han propuesto distintas técnicas de procesado de señal basadas en análisis espectral y en la obtención parámetros no lineales de la señal, que no habían sido utilizadas previamente en experimentos psicofisiológicos con DTC. Se ha observado que existe un incremento en la velocidad del flujo sanguíneo durante la exposición a un EV, el cual puede deberse a distintos factores que intervienen en la experiencia: tareas de interacción visuoespacial, tareas de atención, la creación y ejecución de un plan motor, cambios emocionales Los análisis han mostrado que existen correlaciones significativas entre la velocidad media de flujo sanguíneo en las arterias cerebrales medias durante la exposición al EV y respuestas a los cuestionarios de presencia utilizados.Rey Solaz, B. (2010). Contributions to the Development of Objective Techniques for Presence Measurement in Virtual Environments by means of Brain Activity Analysis [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/8505Palanci

Emotional Responses to Immersive Media

The five experiments presented in this thesis aimed to investigate the relationship between subjective presence (the sense of 'being there' in a mediated environment) and subjective and physiological emotional responses. The investigation served as an initial step in the evaluation of emotion-based corroborative measures of presence. Two of the determinants of presence (Media Form and Media Content) were experimentally manipulated in order to create varying levels of presence and different types of subjective and physiological emotional responses. Varying levels of presence were created by manipulating Media Form characteristics in the context of a video display - specifically, the absence and presence of stereoscopic cues (Experiments I and 2) and small versus large eye-to-screen visual angles (Experiments 3 and 5). Media Content (video clips) with varying types of emotional impact was presented to participants via the video displays (Experiments I, 2, 3 and 5). The research showed that enhancements in Media Form increased both subjective presence and subjective arousal across all Content types. However, there was little evidence to show that Media Form manipulations affected physiological arousal, indicating limited utility for physiological measures of presence in this context. Media Content characteristics were investigated in more depth in Experiments 4 and 5. Differences in SUbjective presence between Media Contents shown to elicit different types of subjective emotion were investigated and correlations between subjective presence and subjective emotion were examined. The research indicated that the quality and intensity of emotional responses elicited by Media Content are potentially important determinants of subjective presence. The thesis concludes by suggesting that in further investigations of presence and emotion, which aim to identify corroborative measures of presence, it may be useful to analyse the relationship between different dimensions of presence (physical Space, Naturalness and Engagement) and different components of emotional responses (Subjective, Cognitive, Behavioural and Physiological)

Socio-Cognitive and Affective Computing

Social cognition focuses on how people process, store, and apply information about other people and social situations. It focuses on the role that cognitive processes play in social interactions. On the other hand, the term cognitive computing is generally used to refer to new hardware and/or software that mimics the functioning of the human brain and helps to improve human decision-making. In this sense, it is a type of computing with the goal of discovering more accurate models of how the human brain/mind senses, reasons, and responds to stimuli. Socio-Cognitive Computing should be understood as a set of theoretical interdisciplinary frameworks, methodologies, methods and hardware/software tools to model how the human brain mediates social interactions. In addition, Affective Computing is the study and development of systems and devices that can recognize, interpret, process, and simulate human affects, a fundamental aspect of socio-cognitive neuroscience. It is an interdisciplinary field spanning computer science, electrical engineering, psychology, and cognitive science. Physiological Computing is a category of technology in which electrophysiological data recorded directly from human activity are used to interface with a computing device. This technology becomes even more relevant when computing can be integrated pervasively in everyday life environments. Thus, Socio-Cognitive and Affective Computing systems should be able to adapt their behavior according to the Physiological Computing paradigm. This book integrates proposals from researchers who use signals from the brain and/or body to infer people's intentions and psychological state in smart computing systems. The design of this kind of systems combines knowledge and methods of ubiquitous and pervasive computing, as well as physiological data measurement and processing, with those of socio-cognitive and affective computing

EVALUATION OF VISUALLY INDUCED MOTION SICKNESS CAUSED BY VIEWING OF 3D STEREOSCOPY USING ELECTROENCEPHALOGRAPHY TECHNIQUE

The 3D movies are attracting the viewers as they see objects flying out of the screen. However, many viewers reportof problems that they face after watching 3D movies. Visual fatigue, eye strain, headaches, dizziness, blurred vision or in other words, Visually Induced Motion Sickness (VIMS) are reported by viewers of 3D movies. In this thesis, we aim to compare a 3D passive technology with a conventional 2D technology to find whether 3D is causing trouble in the viewers or not

UNDERSTANDING INTERACTIVE EXPERIENCES: PERCEIVED INTERACTIVITY AND PRESENCE WITH AND WITHOUT OTHER AVATARS IN THE ONLINE VIRTUAL WORLD SECOND LIFE

Interactivity research lacks consensus regarding the qualities and consequences of interactive experiences. Empirical proof is needed to substantiate the numerous interactivity theories and provide direction for new media technology developers. Specifically, there is a shortage of research on differences between user experiences of interactivity when technology enables communication versus when it does not. In addition, interactivity research is often confounded by the construct of presence. This study’s objectives included: 1) identifying qualities associated with interactive experiences; 2) disambiguating the constructs of interactivity and presence; and 3) developing a measure of perceived interactivity for VW research. The experimental design measured perceived interactivity and presence following completion of a simple task in the online Virtual World (VW) known as Second Life. It was hypothesized that both perceived interactivity and presence would be greater for subjects encountering avatars believed to be controlled by other people than for subjects encountering no other avatars in the VW. A total of 180 subjects from the University of Kentucky participated in a 2 by 4 factorial experiment. Perceived interactivity was measured by modifying McMillan and Hwang’s Measure of Perceived Interactivity for the VW context. Two essential qualities of interactive experiences were identified: Responsiveness and engagement. These qualities are characteristic of unmediated, FTF conversation, which was perceived as the most interactive communication context above technologies routinely described as interactive. Decreased responsiveness of technology at a second study venue caused significant decline in perceived interactivity, demonstrating the importance of a technology’s reaction speed and control provided to the user. Significant main effects for perceived interactivity due to encountering other avatars were confounded by interaction effects due to differences in technology responsiveness. Interactivity and presence appear to be separate psychological constructs which covary in the context of a new media experience. Implications and directions for future research are discussed