The Observer Lens : Characterizing Visuospatial Features in Multimodal Interactions

Understanding the intricate nature of human interactions relies heavily on the ability to discern and interpret inherent information. Central to this interpretation are the sensory features—visual, spatial, and auditory—collectively referred to as visuospatial in this thesis. The low-level (e.g., motion kinematics) and high-level (e.g., gestures and speech) visuospatial features significantly influence human perception, aiding in the deduction of intent, goals, emotions, and more. From a computational viewpoint, these features are crucial for interpreting events, from discerning body poses to evaluating action similarity, particularly for computational systems designed to interact closely with humans. This thesis examines the impact of visuospatial features on human event observation within an informatics context, concentrating on (1) Investigating the effect of visuospatial features on the observers’ perception; and (2) Aligning this investigation towards outcomes applicable to informatics. Taking a human-centric perspective, the thesis methodically probes the role of visuospatial features, drawing from prior cognitive research, underscoring the significance of features like action kinematics, gaze, turn-taking, and gestures in event comprehension. Balancing both reductionist and naturalistic perspectives, the research examines specific visuospatial features and their impact on the human's visual processing and attention mechanisms: - Visual Processing: It highlights the visual processing effects of action features, including kinematics, local-motion, and global form, as well as the role of factors like semantics and familiarity. These are demonstrated using human performance metrics in perceptual tasks and comparative analyses with selected computational models employing basic kinematic representations, revealing the adaptive nature of the visual system and enhancing Human Action Recognition models. - Visual Attention: Also highlights the attentional effects of interaction cues, such as speech, hand action, body pose, motion, and gaze, using the developed `Visuospatial Model'. This model presents a systematic approach for characterizing visuospatial features in everyday events, exemplified using a curated movie dataset and a newly developed comprehensive dataset of naturalistic-multimodal events. The findings emphasize the integration of behavioral and perceptual parameters with computationally aligned strategies, such as benchmarking perceptual tasks against human behavioral and psychophysical metrics, thereby providing a richer context for developing systematic tools and methodologies for multimodal event characterization. At its core, the thesis characterizes the role of visuospatial features in shaping human perception and its implications for the development of cognitive technologies equipped with autonomous perception and interaction capabilities -- essential for domains like social robotics, autonomous driving, media studies, traffic safety, and virtual characters.Förståelse för den intrikata naturen hos mänskliga interaktioner beror i stor utsträckning på förmågan att urskilja och tolka inneboende information. Centralt i denna tolkning är de sensoriska aspekterna – visuella, rumsliga och auditiva – som tillsammans kallas visuospatiala i denna avhandling. De visuella särdragen på låg nivå (t.ex. rörelsekinematik) och hög nivå (t.ex. gester och tal) påverkar människans uppfattningsförmåga avsevärt och är till hjälp för att dra slutsatser om avsikter, mål, känslor och mer. Ur en beräkningssynpunkt är dessa egenskaper avgörande för att tolka händelser, från att utläsa kroppsställningar till att bedöma liket mellan handlingar, särskilt för beräkningssystem utformade för att interagera nära med människor. Denna avhandling undersöker effekten av visuospatiala särdrag på observation av händelser som involverar människor inom en informatikkontext, med fokus på att (1) Undersöka effekten av visuospatiala särdrag på observatörernas perception; och (2) Anpassa denna undersökning till situationer som är tillämpliga inom informatik. Med ett människocentrerat perspektiv undersöker avhandlingen metodiskt rollen av visuospatiala egenskaper, med utgångspunkt från tidigare kognitiv forskning, och understryker betydelsen av funktioner som aktionskinematik, ögonrörelser, turtagning och gester i händelseförståelse. Genom att balansera både reduktionistiska och naturalistiska perspektiv undersöker forskningen specifika visuospatiala egenskaper och deras inverkan på människans visuella bearbetnings- och uppmärksamhetsmekanismer: - Visuell bearbetning: Den belyser de visuella bearbetningseffekterna av handlingsaspekter, inklusive kinematik, lokal rörelse och global form, såväl som rollen av faktorer som semantik och förtrogenhet. Dessa demonstreras med hjälp av mänskliga prestationsmått i perceptuella uppgifter och jämförande analyser med utvalda beräkningsmodeller som använder grundläggande kinematiska representationer, vilket avslöjar det visuella systemets adaptiva karaktär och förbättrar modeller för igenkänning av mänskliga handlingar. - Visuell uppmärksamhet: Belyser också uppmärksamhetseffekterna av interaktionssignaler, såsom tal, handrörelser, kroppshållning, rörelser och blick, med hjälp av den utvecklade ``Visuospatiala modellen". Denna modell presenterar ett systematiskt tillvägagångssätt för att karakterisera visuospatiala funktioner i vardagliga händelser, exemplifierat med hjälp av en datamäng bestående av kurerat filmmaterial och en nyutvecklad omfattande datamängd av naturalistiska multimodala händelser. Fynden betonar integrationen av beteendemässiga och perceptuella parametrar med beräkningsmässigt anpassade strategier, såsom benchmarking av perceptuella uppgifter mot mänskliga beteendemässiga och psykofysiska mått, vilket ger ett rikare sammanhang för att utveckla verktyg och metoder för systematisk multimodal händelsekategorisering. I sin kärna betonar avhandlingen visuospatiala egenskapers roll i att forma mänsklig perception och dess implikationer för utvecklingen av kognitiva teknologier utrustade med autonom perception och interaktionsförmåga - väsentligt för domäner som social robotik, autonom körning, mediestudier, trafiksäkerhet, och virtuella karaktärer.Tre av sex delarbeten (övriga se rubriken Delarbeten/List of papers):In addition to the papers listed under 'List of Papers,' the following journal manuscripts are included in the thesis. These manuscripts are not present in the online copy of the thesis. Upon acceptance of the papers, the online document, along with this entry, will be updated.Paper-I: Hemeren, P., Nair, V., &amp; Drejing, K. (Submitted Manuscript). "Biological motion and attention: Interactions influenced by walking orientations, styles, and motion features". Submitted Manuscript for Scientific Journal, pp. 1-18. Paper-V: Nair, V., Bhatt, M., Suchan, J., Billing, E., &amp; Hemeren, P. (Submitted Manuscript). "How do naturalistic visuo-auditory cues guide human attention?" Submitted Manuscript for Scientific Journal, pp. 1-48. Paper-VI: Nair, V., Bhatt, M., Suchan, J., Billing, E., &amp; Hemeren, P. (Manuscript). "Human interaction scenarios: A naturalistic multimodal event dataset for behavioral research". Manuscript for Scientific Journal, pp. 1-28.In reference to IEEE copyrighted material which is used with permission in this thesis, the IEEE does not endorse any of The University of Skövde's products or services. Internal or personal use of this material is permitted. If interested in reprinting/republishing IEEE copyrighted material for advertising or promotional purposes or for creating new collective works for resale or redistribution, please go to http://www.ieee.org/publications_standards/publications/rights/rights_link.html to learn how to obtain a License from RightsLink.</p

Patients’ experiences of life after a myocardial infarction in Sweden : A qualitative literature overview

Bakgrund: Många personer drabbas av hjärtinfarkt i Sverige varje år. En hjärtinfarkt uppstår när hjärtat får syrebrist genom en tilltäppning i något av de blodkärl som förser hjärtat med syre. En hjärtinfarkt är en dödlig sjukdom som kan upplevas som ett trauma av den som överlevt. Detta medföljande trauma kan vara svårhanterligt utan rätt kunskap eller resurser. Sjuksköterskan har som uppgift att genom sina kärnkompetenser nå ut till patienter för att främja hälsa. Kliniskt arbetande sjuksköterskor behöver en ökad förståelse av hjärtinfarkt patienters upplevelser för att bättre kunna vårda, uppmärksamma, motivera och behandla dessa patienter med mål att så goda resultat som möjligt skall uppnås. Syfte: Belysa patienters upplevelser av livet efter en hjärtinfarkt i Sverige. Metod: En kvalitativ litteraturstudie baserad på 13 kvalitativa artiklar om patienters upplevelse efter en hjärtinfarkt. Resultat: I studien kunde 6 teman identifieras. “Förändrad livsvärld”. ”Hantera en livsomställning”, “Samspel med vården”, “Stöd från närstående”, “Information och kunskap”, och “En andra chans”. Konklusion: Patienter upplevde svårigheter med att anpassa sig till livsstilsförändringar de blivit rekommenderade till av sjukvårdspersonal och den nya vardagen som följer efter sin hjärtinfarkt. Faktorer som påverkade patienter positivt efter en hjärtinfarkt var copingstrategier, ett bra samspel med vården och stöd och förståelse från närstående.Background: In Sweden, many people develop a myocardial infarction each year. A myocardial infarction occurs when the heart suddenly experience oxygen deprivation due to a blockage in one of the main oxygen supplying blood vessels. A myocardial infarction is a deadly disease that can be experienced as a trauma for the person surviving it. This coherent trauma can be difficult to handle without the right knowledge or resources. A nurse's task is to use their core competences to reach out to patients and promote health. Clinical nurses need an enhanced understanding of the experiences of patients with myocardial infarction to be able to better care for, pay attention to, motivate and treat these patients with the goal of achieving the best possible results. Purpose: Elucidate patients experiences of life after a myocardial infarction in Sweden. Method: A qualitative literature study based on 13 qualitative articles about patients’ experience after a myocardial infarction. Results: 6 themes were identified in this study. "Change of life-world". "Managing a life adjustment", "Interaction with professional healthcare", "Support from close relatives","Information and knowledge", "A second chance". Conclusion: Patients experienced difficulties adapting to the lifestyle changes, they had been recommended to pursue by healthcare professionals, and to the new everyday life after their myocardial infarction. Factors that affected patients positively after a myocardial infarction were coping strategies, good interaction with professional healthcare, and the support and understanding from their close relatives

Developing a multivariable prediction model of global health-related quality of life in patients treated for rectal cancer : a prospective study in five countries

Purpose Rectal cancer and its treatment have a negative impact on health-related quality of life (HRQoL). If risk factors for sustained low HRQoL could be identified early, ideally before the start of treatment, individualised interventions could be identified and implemented to maintain or improve HRQoL. The study aimed to develop a multivariable prediction model for global HRQoL 12 months after rectal cancer treatment. Methods Within COLOR II, a randomised, multicentre, international trial of laparoscopic and open surgery for rectal cancer, a sub-study on HRQoL included 385 patients in 12 hospitals and five countries. The HRQoL study was optional for hospitals in the COLOR II trial. EORTC QLQ-C30 and EORTC QLQ-CR38 were analysed preoperatively and at 1 and 12 months postoperatively. In exploratory analyses, correlations between age, sex, fatigue, pain, ASA classification, complications, and symptoms after surgery to HRQoL were studied. Bivariate initial analyses were followed by multivariate regression models. Results Patient characteristics and clinical factors explained 4–10% of the variation in global HRQoL. The patient-reported outcomes from EORTC QLQ-C30 explained 55–65% of the variation in global HRQoL. The predominant predictors were fatigue and pain, which significantly impacted global HRQoL at all time points measured. Conclusion We found that fatigue and pain were two significant factors associated with posttreatment global HRQoL in patients treated for rectal cancer T1-T3 Nx. Interventions to reduce fatigue and pain could enhance global HRQoL after rectal cancer treatment.CC BY 4.0 DEEDEva Haglind [email protected] trial is registered with ClinicalTrials.gov No. NCT00297791Open access funding provided by University of Gothenburg. The study received financial support from the Swedish state under the agreement between the Swedish government and the county councils, the ALF-agreement (ALFGBG-138751; ALFGBG-4307771; ALFGBG-718221; ALFGBG-965245), the Swedish Cancer Society (CAN 2010/593CAN 2013/497; CAN 2016/362; Pj19 0303), the Alice Swenson Foundation, Anna-Lisa and Bror Björnsson Foundation, Axel Linders Foundation, Assar Gabrielsson Foundation, and the Gothenburg Medical Society.</p

Neurofeedback training for children with ADHD : Evaluating the effect of personalized and standardized neurofeedback protocols on theta rhythms, beta rhythms and the iAPF

Attention deficit hyperactivity disorder (ADHD) is a common neurodevelopmental disorder in children, characterized by symptoms such as hyperactivity, inattention and impulsivity, which often impairs the quality of life and successful performance. The disorder is associated with neurological, genetic and environmental factors affecting brain structure and cognitive function. Treatments include first-line pharmacological treatment, psychological therapy and neurofeedback training. Neurofeedback training with electroencephalography aims to regulate brain activity by targeting the dysregulated theta and beta rhythms, and the theta/beta ratio through personalized or standardized neurofeedback protocols to reduce ADHD symptoms and to improve cognitive function. This study aims to compare differences in the change of theta, beta and the individual alpha peak frequency (iAPF) based on personalized and standardized neurofeedback protocols targeting theta, beta and the theta/beta ratio. Nine children (age 7 to 13) were randomly assigned 16 sessions of personalized or standardized neurofeedback training over a span of two weeks; the neurofeedback protocol was personalized with the help of the iAPF. The theta and beta rhythms and the iAPF were measured by a quantitative electroencephalography and compared before and after the intervention by non-parametric Mann-Whitney-U, Wilcoxon Signed-Rank and Kruskal Wallis tests. These tests showed no significant difference between theta, beta and iAPF before and after the neurofeedback training within and across the personalized and standardized test groups. Previous studies showed an improvement of brain activity and cognitive function through neurofeedback training, but the small sample size and participant artifacts limited this study; concluding that valuable insights into neurofeedback training were gained

Balancing and scheduling human-robot collaborated assembly lines with layout and objective consideration

The recent Industry 4.0 trend, followed by the technological advancement of collaborative robots, has urged many industries to shift towards new types of assembly lines with human-robot collaboration (HRC). This type of manufacturing line, in which human skill is supported by robot agility, demands an integrated balancing and scheduling of tasks and operators among the stations. This study attempts to deal with these joint problems in the straight and U-shaped assembly lines while considering different objectives, namely, the number of stations (Type-1), the cycle time (Type-2), and the cost of stations, operators, and robot energy consumption (Type-rw). The latter type often arises in the real world, where multiple types of humans and robots with different skills and energy levels can perform the assembly tasks collaboratively or in parallel at stations. Additionally, practical constraints, namely robot tool changes, zoning, and technological requirements, are considered in Type-rw. Accordingly, different mixed-integer linear programming (MILP) models for straight and U-shaped layouts are proposed with efficient lower and upper bounds for each objective. The computational results validate the efficiency of the proposed MILP model with bounded objectives while addressing an application case and different test problem sizes. In addition, the analysis of results shows that the U-shaped layout offers greater flexibility than the straight line, leading to more efficient solutions for JIT production, particularly in objective Type-2 followed by Type-rw and Type-1. Moreover, the U-shaped lines featuring a high HRC level can further enhance the achievement of desired objectives compared to the straight lines with no or limited HRC.CC BY 4.0 DEEDCorresponding author: Email: [email protected] study was funded by the Knowledge Foundation (KKS) and Sweden’s Innovation Agency through the VF-KDO, ACCURATE 4.0, and PREFER projects.</p

The ‘service turn’ in a new public management context : a street-level bureaucrat perspective

It is increasingly argued that public management should build on a service logic instead of the prevailing manufacturing logic of New Public Management (NPM). Drawing from three cases in Swedish public healthcare, key features of a service logic such as value creation, co-production, and collaboration are prominent in formal documents and everyday talk. However, the 67 interviews in this study reveal that the service logic ideal is practically unreachable in a context impregnated by NPM. Instead, we suggest that street-level bureaucrats often need to address service logic expectations (public values, relationship-building, etc.) using an NPM logic (measurements, control, etc.). CC BY 4.0© 2023 The Author(s). Published by Informa UK Limited, trading as Taylor &amp; Francis Group.Published online: 02 Aug 2023Correspondence Address: E. Eriksson; Department of Work Life and Social Welfare, University of Borås, Borås, Sweden; email: [email protected]</p

Unveiling the Potential of Mixed Reality : Enhancing Time Measurement and Operator Support in Manual Assembly Processes

This study investigates the potential of Mixed Reality (MR) in the manual assembly processes and conducts a case study at a pump manufacturing plant in Sweden. An MR solution is developed to assist operators through visual instructions and guiding aides. The solution also captures the operator's motions using advanced hand and eye tracking features for real-time guidance and accurate time measurement. The proposed MR solution uses the build feature of HoLolens and a workstation editor, which facilitates the use of the solution in diverse assembly environments. The results of the experiments show that the developed MR solution can improve operator support, reduce errors, and enhance the overall efficiency of manual assembly processes. Moreover, it is shown to be an efficient tool for time measurement of the manual assembly process that has promising potential to replace sophisticated and time-consuming traditional time study methods.CC BY-NC-ND 4.0© 2024 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0)Correspondence Address: M. Fathi; Division of Intelligent Production Systems, School of Engineering Science, University of Skövde, Skövde, 54128, Sweden; email: [email protected] authors would like to thank the Knowledge Foundation (KKS), Sweden, for their financial support through the ACCURATE 4.0 project, under grant agreement No. 20200181. We also wish to extend our appreciation to the industrial partner of the project, Xylem Water Solutions Sweden AB. Their collaboration, expertise, and invaluable insights have significantly contributed to this study.ACCURATE 4.

Comfort versus privacy : Systematic reviews of IoT-devices in home environments

IoT-enheter inom hemmet blir allt vanligare och oron för hur användares integritet påverkas ökar i takt med att dessa enheter implementeras. Dessa enheter kan styras i hemmet eller på distans via tillhörande applikationer och har möjlighet att bidra med automatiserade funktioner och underhållning i en hemmiljö. Allt eftersom mer teknik implementeras i hemmet som kopplas upp mot internet kan detta utgöra risker för privatpersoners integritet. Detta arbete utförs i två separata strukturerade litteraturstudier. Den ena undersöker vilken typ av personuppgifter som kan komma att hämtas in av IoT-enheter i ett hem och den andra fokuserar på privatpersoners medvetenhet och inställning till insamlingen av dessa enheter. Båda studier använder sig av kvalitativa metoder för att undersöka respektive ämne för att nå en kombinerad slutsats om vilka personuppgifter som samlas in av IoT-enheter och privatpersoners medvetenhet om insamlingen.Home IoT-devices are becoming more commonplace, and the concern about what this implies for users' privacy rises along with it. These devices can be managed locally, within the home, or remotely, with their companion apps, in order to enable automated functions and entertainment. As more technical devices are implemented with connection to the internet, more risks occur regarding the privacy of the user. This paper consists of two systematic reviews, where the first one focuses on what personal information is being collected by in-home IoT-devices. The second systematic review focuses on the individuals’ awareness and attitude towards the collection of personal information by these devices. Both systematic reviews use a shared method that qualitatively investigates the data collected in order to reach a combined conclusion regarding what personal data is collected, and how aware users are about this collection.

Ambulance records as an information source to document and prevent fall injuries in Sweden

In Sweden, there has been only limited documentation for injuries requiring ambulance responses. The main objective of this study is, through the use of historic data, to assess the suitability of ambulance records to describe circumstances with fall injuries. Methods: The injury events data around patients were sourced from the ambulance data register. Descriptive statistics were used to describe injured patients based on age group, type of injury, place of injury, injury mechanism and consequence of an injury event. Two-group comparison was performed with Pearson’s chi-squared. Predictors of transport to hospital were identified using logistic regression analyses. Result: Ambulance provides unique data on all injury events, with direct implications for translational research, public policy and clinical practice (safety promotion). In 2002 ambulance attended 3964 injured people which represents 14% of ambulance attended workload in Värmland county, Sweden. The most common trauma location was the traffic area followed by residential area and nursing home. These three injury sites account for 2320 cases (61.6%). The most common cause of injury was falls (63.9%) followed by contact with another person (26.5%). Contact with another person is the most common site of injury in the traffic area (79.5%), and men aged 25-66 years are overrepresented. Conclusion: Logistic regression found that, age-group and place code were significant predictor for being attended by ambulance. Traffic, home and nursing homes were over-represented injury environments (61.6%) that require special attention. Most injury cases occur in the home and nursing homes among people over 67 years of age. Surprisingly, most of the injury events in the traffic environment are about hitting another person. Paramedics can provide rich and valuable data on injury events. Registration of such data is entirely possible and desirable, and can be used in preventive work.CC BY 4.0Funding: Swedish Rescue Civil Contingencies Agency: MSB.</p

Advancing Deep Learning-based Driver Intention Recognition : Towards a safe integration framework of high-risk AI systems

Progress in artificial intelligence (AI), onboard computation capabilities, and the integration of advanced sensors in cars have facilitated the development of Advanced Driver Assistance Systems (ADAS). These systems aim to continuously minimize human driving errors. {An example application of an ADAS could be to support a human driver by informing if an intended driving maneuver is safe to pursue given the current state of the driving environment. One of the components enabling such an ADAS is recognizing the driver's intentions. Driver intention recognition (DIR) concerns the identification of what driving maneuver a driver aspires to perform in the near future, commonly spanning a few seconds. A challenging aspect of integrating such a system into a car is the ability of the ADAS to handle unseen scenarios. Deploying any AI-based system in an environment where mistakes can cause harm to human beings is considered a high-risk AI system. Upcoming AI regulations require a car manufacturer to motivate the design, performance-complexity trade-off, and the understanding of potential blind spots of a high-risk AI system.} Therefore, this licentiate thesis focuses on AI-based DIR systems and presents an overview of the current state of the DIR research field. Additionally, experimental results are included that demonstrate the process of empirically motivating and evaluating the design of deep neural networks for DIR. To avoid the reliance on sequential Monte Carlo sampling techniques to produce an uncertainty estimation, we evaluated a surrogate model to reproduce uncertainty estimations learned from probabilistic deep-learning models. Lastly, to contextualize the results within the broader scope of safely integrating future high-risk AI-based systems into a car, we propose a foundational conceptual framework