The cockpit for the 21st century

Interactive surfaces are a growing trend in many domains. As one possible manifestation of Mark Weiser’s vision of ubiquitous and disappearing computers in everywhere objects, we see touchsensitive screens in many kinds of devices, such as smartphones, tablet computers and interactive tabletops. More advanced concepts of these have been an active research topic for many years. This has also influenced automotive cockpit development: concept cars and recent market releases show integrated touchscreens, growing in size. To meet the increasing information and interaction needs, interactive surfaces offer context-dependent functionality in combination with a direct input paradigm. However, interfaces in the car need to be operable while driving. Distraction, especially visual distraction from the driving task, can lead to critical situations if the sum of attentional demand emerging from both primary and secondary task overextends the available resources. So far, a touchscreen requires a lot of visual attention since its flat surface does not provide any haptic feedback. There have been approaches to make direct touch interaction accessible while driving for simple tasks. Outside the automotive domain, for example in office environments, concepts for sophisticated handling of large displays have already been introduced. Moreover, technological advances lead to new characteristics for interactive surfaces by enabling arbitrary surface shapes. In cars, two main characteristics for upcoming interactive surfaces are largeness and shape. On the one hand, spatial extension is not only increasing through larger displays, but also by taking objects in the surrounding into account for interaction. On the other hand, the flatness inherent in current screens can be overcome by upcoming technologies, and interactive surfaces can therefore provide haptically distinguishable surfaces. This thesis describes the systematic exploration of large and shaped interactive surfaces and analyzes their potential for interaction while driving. Therefore, different prototypes for each characteristic have been developed and evaluated in test settings suitable for their maturity level. Those prototypes were used to obtain subjective user feedback and objective data, to investigate effects on driving and glance behavior as well as usability and user experience. As a contribution, this thesis provides an analysis of the development of interactive surfaces in the car. Two characteristics, largeness and shape, are identified that can improve the interaction compared to conventional touchscreens. The presented studies show that large interactive surfaces can provide new and improved ways of interaction both in driver-only and driver-passenger situations. Furthermore, studies indicate a positive effect on visual distraction when additional static haptic feedback is provided by shaped interactive surfaces. Overall, various, non-exclusively applicable, interaction concepts prove the potential of interactive surfaces for the use in automotive cockpits, which is expected to be beneficial also in further environments where visual attention needs to be focused on additional tasks.Der Einsatz von interaktiven Oberflächen weitet sich mehr und mehr auf die unterschiedlichsten Lebensbereiche aus. Damit sind sie eine mögliche Ausprägung von Mark Weisers Vision der allgegenwärtigen Computer, die aus unserer direkten Wahrnehmung verschwinden. Bei einer Vielzahl von technischen Geräten des täglichen Lebens, wie Smartphones, Tablets oder interaktiven Tischen, sind berührungsempfindliche Oberflächen bereits heute in Benutzung. Schon seit vielen Jahren arbeiten Forscher an einer Weiterentwicklung der Technik, um ihre Vorteile auch in anderen Bereichen, wie beispielsweise der Interaktion zwischen Mensch und Automobil, nutzbar zu machen. Und das mit Erfolg: Interaktive Benutzeroberflächen werden mittlerweile serienmäßig in vielen Fahrzeugen eingesetzt. Der Einbau von immer größeren, in das Cockpit integrierten Touchscreens in Konzeptfahrzeuge zeigt, dass sich diese Entwicklung weiter in vollem Gange befindet. Interaktive Oberflächen ermöglichen das flexible Anzeigen von kontextsensitiven Inhalten und machen eine direkte Interaktion mit den Bildschirminhalten möglich. Auf diese Weise erfüllen sie die sich wandelnden Informations- und Interaktionsbedürfnisse in besonderem Maße. Beim Einsatz von Bedienschnittstellen im Fahrzeug ist die gefahrlose Benutzbarkeit während der Fahrt von besonderer Bedeutung. Insbesondere visuelle Ablenkung von der Fahraufgabe kann zu kritischen Situationen führen, wenn Primär- und Sekundäraufgaben mehr als die insgesamt verfügbare Aufmerksamkeit des Fahrers beanspruchen. Herkömmliche Touchscreens stellen dem Fahrer bisher lediglich eine flache Oberfläche bereit, die keinerlei haptische Rückmeldung bietet, weshalb deren Bedienung besonders viel visuelle Aufmerksamkeit erfordert. Verschiedene Ansätze ermöglichen dem Fahrer, direkte Touchinteraktion für einfache Aufgaben während der Fahrt zu nutzen. Außerhalb der Automobilindustrie, zum Beispiel für Büroarbeitsplätze, wurden bereits verschiedene Konzepte für eine komplexere Bedienung großer Bildschirme vorgestellt. Darüber hinaus führt der technologische Fortschritt zu neuen möglichen Ausprägungen interaktiver Oberflächen und erlaubt, diese beliebig zu formen. Für die nächste Generation von interaktiven Oberflächen im Fahrzeug wird vor allem an der Modifikation der Kategorien Größe und Form gearbeitet. Die Bedienschnittstelle wird nicht nur durch größere Bildschirme erweitert, sondern auch dadurch, dass Objekte wie Dekorleisten in die Interaktion einbezogen werden können. Andererseits heben aktuelle Technologieentwicklungen die Restriktion auf flache Oberflächen auf, so dass Touchscreens künftig ertastbare Strukturen aufweisen können. Diese Dissertation beschreibt die systematische Untersuchung großer und nicht-flacher interaktiver Oberflächen und analysiert ihr Potential für die Interaktion während der Fahrt. Dazu wurden für jede Charakteristik verschiedene Prototypen entwickelt und in Testumgebungen entsprechend ihres Reifegrads evaluiert. Auf diese Weise konnten subjektives Nutzerfeedback und objektive Daten erhoben, und die Effekte auf Fahr- und Blickverhalten sowie Nutzbarkeit untersucht werden. Diese Dissertation leistet den Beitrag einer Analyse der Entwicklung von interaktiven Oberflächen im Automobilbereich. Weiterhin werden die Aspekte Größe und Form untersucht, um mit ihrer Hilfe die Interaktion im Vergleich zu herkömmlichen Touchscreens zu verbessern. Die durchgeführten Studien belegen, dass große Flächen neue und verbesserte Bedienmöglichkeiten bieten können. Außerdem zeigt sich ein positiver Effekt auf die visuelle Ablenkung, wenn zusätzliches statisches, haptisches Feedback durch nicht-flache Oberflächen bereitgestellt wird. Zusammenfassend zeigen verschiedene, untereinander kombinierbare Interaktionskonzepte das Potential interaktiver Oberflächen für den automotiven Einsatz. Zudem können die Ergebnisse auch in anderen Bereichen Anwendung finden, in denen visuelle Aufmerksamkeit für andere Aufgaben benötigt wird

The cockpit for the 21st century

Interactive surfaces are a growing trend in many domains. As one possible manifestation of Mark Weiser’s vision of ubiquitous and disappearing computers in everywhere objects, we see touchsensitive screens in many kinds of devices, such as smartphones, tablet computers and interactive tabletops. More advanced concepts of these have been an active research topic for many years. This has also influenced automotive cockpit development: concept cars and recent market releases show integrated touchscreens, growing in size. To meet the increasing information and interaction needs, interactive surfaces offer context-dependent functionality in combination with a direct input paradigm. However, interfaces in the car need to be operable while driving. Distraction, especially visual distraction from the driving task, can lead to critical situations if the sum of attentional demand emerging from both primary and secondary task overextends the available resources. So far, a touchscreen requires a lot of visual attention since its flat surface does not provide any haptic feedback. There have been approaches to make direct touch interaction accessible while driving for simple tasks. Outside the automotive domain, for example in office environments, concepts for sophisticated handling of large displays have already been introduced. Moreover, technological advances lead to new characteristics for interactive surfaces by enabling arbitrary surface shapes. In cars, two main characteristics for upcoming interactive surfaces are largeness and shape. On the one hand, spatial extension is not only increasing through larger displays, but also by taking objects in the surrounding into account for interaction. On the other hand, the flatness inherent in current screens can be overcome by upcoming technologies, and interactive surfaces can therefore provide haptically distinguishable surfaces. This thesis describes the systematic exploration of large and shaped interactive surfaces and analyzes their potential for interaction while driving. Therefore, different prototypes for each characteristic have been developed and evaluated in test settings suitable for their maturity level. Those prototypes were used to obtain subjective user feedback and objective data, to investigate effects on driving and glance behavior as well as usability and user experience. As a contribution, this thesis provides an analysis of the development of interactive surfaces in the car. Two characteristics, largeness and shape, are identified that can improve the interaction compared to conventional touchscreens. The presented studies show that large interactive surfaces can provide new and improved ways of interaction both in driver-only and driver-passenger situations. Furthermore, studies indicate a positive effect on visual distraction when additional static haptic feedback is provided by shaped interactive surfaces. Overall, various, non-exclusively applicable, interaction concepts prove the potential of interactive surfaces for the use in automotive cockpits, which is expected to be beneficial also in further environments where visual attention needs to be focused on additional tasks.Der Einsatz von interaktiven Oberflächen weitet sich mehr und mehr auf die unterschiedlichsten Lebensbereiche aus. Damit sind sie eine mögliche Ausprägung von Mark Weisers Vision der allgegenwärtigen Computer, die aus unserer direkten Wahrnehmung verschwinden. Bei einer Vielzahl von technischen Geräten des täglichen Lebens, wie Smartphones, Tablets oder interaktiven Tischen, sind berührungsempfindliche Oberflächen bereits heute in Benutzung. Schon seit vielen Jahren arbeiten Forscher an einer Weiterentwicklung der Technik, um ihre Vorteile auch in anderen Bereichen, wie beispielsweise der Interaktion zwischen Mensch und Automobil, nutzbar zu machen. Und das mit Erfolg: Interaktive Benutzeroberflächen werden mittlerweile serienmäßig in vielen Fahrzeugen eingesetzt. Der Einbau von immer größeren, in das Cockpit integrierten Touchscreens in Konzeptfahrzeuge zeigt, dass sich diese Entwicklung weiter in vollem Gange befindet. Interaktive Oberflächen ermöglichen das flexible Anzeigen von kontextsensitiven Inhalten und machen eine direkte Interaktion mit den Bildschirminhalten möglich. Auf diese Weise erfüllen sie die sich wandelnden Informations- und Interaktionsbedürfnisse in besonderem Maße. Beim Einsatz von Bedienschnittstellen im Fahrzeug ist die gefahrlose Benutzbarkeit während der Fahrt von besonderer Bedeutung. Insbesondere visuelle Ablenkung von der Fahraufgabe kann zu kritischen Situationen führen, wenn Primär- und Sekundäraufgaben mehr als die insgesamt verfügbare Aufmerksamkeit des Fahrers beanspruchen. Herkömmliche Touchscreens stellen dem Fahrer bisher lediglich eine flache Oberfläche bereit, die keinerlei haptische Rückmeldung bietet, weshalb deren Bedienung besonders viel visuelle Aufmerksamkeit erfordert. Verschiedene Ansätze ermöglichen dem Fahrer, direkte Touchinteraktion für einfache Aufgaben während der Fahrt zu nutzen. Außerhalb der Automobilindustrie, zum Beispiel für Büroarbeitsplätze, wurden bereits verschiedene Konzepte für eine komplexere Bedienung großer Bildschirme vorgestellt. Darüber hinaus führt der technologische Fortschritt zu neuen möglichen Ausprägungen interaktiver Oberflächen und erlaubt, diese beliebig zu formen. Für die nächste Generation von interaktiven Oberflächen im Fahrzeug wird vor allem an der Modifikation der Kategorien Größe und Form gearbeitet. Die Bedienschnittstelle wird nicht nur durch größere Bildschirme erweitert, sondern auch dadurch, dass Objekte wie Dekorleisten in die Interaktion einbezogen werden können. Andererseits heben aktuelle Technologieentwicklungen die Restriktion auf flache Oberflächen auf, so dass Touchscreens künftig ertastbare Strukturen aufweisen können. Diese Dissertation beschreibt die systematische Untersuchung großer und nicht-flacher interaktiver Oberflächen und analysiert ihr Potential für die Interaktion während der Fahrt. Dazu wurden für jede Charakteristik verschiedene Prototypen entwickelt und in Testumgebungen entsprechend ihres Reifegrads evaluiert. Auf diese Weise konnten subjektives Nutzerfeedback und objektive Daten erhoben, und die Effekte auf Fahr- und Blickverhalten sowie Nutzbarkeit untersucht werden. Diese Dissertation leistet den Beitrag einer Analyse der Entwicklung von interaktiven Oberflächen im Automobilbereich. Weiterhin werden die Aspekte Größe und Form untersucht, um mit ihrer Hilfe die Interaktion im Vergleich zu herkömmlichen Touchscreens zu verbessern. Die durchgeführten Studien belegen, dass große Flächen neue und verbesserte Bedienmöglichkeiten bieten können. Außerdem zeigt sich ein positiver Effekt auf die visuelle Ablenkung, wenn zusätzliches statisches, haptisches Feedback durch nicht-flache Oberflächen bereitgestellt wird. Zusammenfassend zeigen verschiedene, untereinander kombinierbare Interaktionskonzepte das Potential interaktiver Oberflächen für den automotiven Einsatz. Zudem können die Ergebnisse auch in anderen Bereichen Anwendung finden, in denen visuelle Aufmerksamkeit für andere Aufgaben benötigt wird

From Manual Driving to Automated Driving: A Review of 10 Years of AutoUI

This paper gives an overview of the ten-year devel- opment of the papers presented at the International ACM Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutoUI) from 2009 to 2018. We categorize the topics into two main groups, namely, manual driving-related research and automated driving-related re- search. Within manual driving, we mainly focus on studies on user interfaces (UIs), driver states, augmented reality and head-up displays, and methodology; Within automated driv- ing, we discuss topics, such as takeover, acceptance and trust, interacting with road users, UIs, and methodology. We also discuss the main challenges and future directions for AutoUI and offer a roadmap for the research in this area.https://deepblue.lib.umich.edu/bitstream/2027.42/153959/1/From Manual Driving to Automated Driving: A Review of 10 Years of AutoUI.pdfDescription of From Manual Driving to Automated Driving: A Review of 10 Years of AutoUI.pdf : Main articl

Studying Person-Specific Pointing and Gaze Behavior for Multimodal Referencing of Outside Objects from a Moving Vehicle

Hand pointing and eye gaze have been extensively investigated in automotive applications for object selection and referencing. Despite significant advances, existing outside-the-vehicle referencing methods consider these modalities separately. Moreover, existing multimodal referencing methods focus on a static situation, whereas the situation in a moving vehicle is highly dynamic and subject to safety-critical constraints. In this paper, we investigate the specific characteristics of each modality and the interaction between them when used in the task of referencing outside objects (e.g. buildings) from the vehicle. We furthermore explore person-specific differences in this interaction by analyzing individuals' performance for pointing and gaze patterns, along with their effect on the driving task. Our statistical analysis shows significant differences in individual behaviour based on object's location (i.e. driver's right side vs. left side), object's surroundings, driving mode (i.e. autonomous vs. normal driving) as well as pointing and gaze duration, laying the foundation for a user-adaptive approach

A Framework For Abstracting, Designing And Building Tangible Gesture Interactive Systems

This thesis discusses tangible gesture interaction, a novel paradigm for interacting with computer that blends concepts from the more popular fields of tangible interaction and gesture interaction. Taking advantage of the human innate abilities to manipulate physical objects and to communicate through gestures, tangible gesture interaction is particularly interesting for interacting in smart environments, bringing the interaction with computer beyond the screen, back to the real world. Since tangible gesture interaction is a relatively new field of research, this thesis presents a conceptual framework that aims at supporting future work in this field. The Tangible Gesture Interaction Framework provides support on three levels. First, it helps reflecting from a theoretical point of view on the different types of tangible gestures that can be designed, physically, through a taxonomy based on three components (move, hold and touch) and additional attributes, and semantically, through a taxonomy of the semantic constructs that can be used to associate meaning to tangible gestures. Second, it helps conceiving new tangible gesture interactive systems and designing new interactions based on gestures with objects, through dedicated guidelines for tangible gesture definition and common practices for different application domains. Third, it helps building new tangible gesture interactive systems supporting the choice between four different technological approaches (embedded and embodied, wearable, environmental or hybrid) and providing general guidance for the different approaches. As an application of this framework, this thesis presents also seven tangible gesture interactive systems for three different application domains, i.e., interacting with the In-Vehicle Infotainment System (IVIS) of the car, the emotional and interpersonal communication, and the interaction in a smart home. For the first application domain, four different systems that use gestures on the steering wheel as interaction means with the IVIS have been designed, developed and evaluated. For the second application domain, an anthropomorphic lamp able to recognize gestures that humans typically perform for interpersonal communication has been conceived and developed. A second system, based on smart t-shirts, recognizes when two people hug and reward the gesture with an exchange of digital information. Finally, a smart watch for recognizing gestures performed with objects held in the hand in the context of the smart home has been investigated. The analysis of existing systems found in literature and of the system developed during this thesis shows that the framework has a good descriptive and evaluative power. The applications developed during this thesis show that the proposed framework has also a good generative power.Questa tesi discute l’interazione gestuale tangibile, un nuovo paradigma per interagire con il computer che unisce i principi dei più comuni campi di studio dell’interazione tangibile e dell’interazione gestuale. Sfruttando le abilità innate dell’uomo di manipolare oggetti fisici e di comunicare con i gesti, l’interazione gestuale tangibile si rivela particolarmente interessante per interagire negli ambienti intelligenti, riportando l’attenzione sul nostro mondo reale, al di là dello schermo dei computer o degli smartphone. Poiché l’interazione gestuale tangibile è un campo di studio relativamente recente, questa tesi presenta un framework (quadro teorico) che ha lo scopo di assistere lavori futuri in questo campo. Il Framework per l’Interazione Gestuale Tangibile fornisce supporto su tre livelli. Per prima cosa, aiuta a riflettere da un punto di vista teorico sui diversi tipi di gesti tangibili che possono essere eseguiti fisicamente, grazie a una tassonomia basata su tre componenti (muovere, tenere, toccare) e attributi addizionali, e che possono essere concepiti semanticamente, grazie a una tassonomia di tutti i costrutti semantici che permettono di associare dei significati ai gesti tangibili. In secondo luogo, il framework proposto aiuta a concepire nuovi sistemi interattivi basati su gesti tangibili e a ideare nuove interazioni basate su gesti con gli oggetti, attraverso linee guida per la definizione di gesti tangibili e una selezione delle migliore pratiche per i differenti campi di applicazione. Infine, il framework aiuta a implementare nuovi sistemi interattivi basati su gesti tangibili, permettendo di scegliere tra quattro differenti approcci tecnologici (incarnato e integrato negli oggetti, indossabile, distribuito nell’ambiente, o ibrido) e fornendo una guida generale per la scelta tra questi differenti approcci. Come applicazione di questo framework, questa tesi presenta anche sette sistemi interattivi basati su gesti tangibili, realizzati per tre differenti campi di applicazione: l’interazione con i sistemi di infotainment degli autoveicoli, la comunicazione interpersonale delle emozioni, e l’interazione nella casa intelligente. Per il primo campo di applicazione, sono stati progettati, sviluppati e testati quattro differenti sistemi che usano gesti tangibili effettuati sul volante come modalità di interazione con il sistema di infotainment. Per il secondo campo di applicazione, è stata concepita e sviluppata una lampada antropomorfica in grado di riconoscere i gesti tipici dell’interazione interpersonale. Per lo stesso campo di applicazione, un secondo sistema, basato su una maglietta intelligente, riconosce quando due persone si abbracciano e ricompensa questo gesto con uno scambio di informazioni digitali. Infine, per l’interazione nella casa intelligente, è stata investigata la realizzazione di uno smart watch per il riconoscimento di gesti eseguiti con oggetti tenuti nella mano. L’analisi dei sistemi interattivi esistenti basati su gesti tangibili permette di dimostrare che il framework ha un buon potere descrittivo e valutativo. Le applicazioni sviluppate durante la tesi mostrano che il framework proposto ha anche un valido potere generativo

Realistic Interaction with Virtual Objects within Arm's Reach

The automotive industry requires realistic virtual reality applications more than other domains to increase the efficiency of product development. Currently, the visual quality of virtual invironments resembles reality, but interaction within these environments is usually far from what is known in everyday life. Several realistic research approaches exist, however they are still not all-encompassing enough to be usable in industrial processes. This thesis realizes lifelike direct multi-hand and multi-finger interaction with arbitrary objects, and proposes algorithmic and technical improvements that also approach lifelike usability. In addition, the thesis proposes methods to measure the effectiveness and usability of such interaction techniques as well as discusses different types of grasping feedback that support the user during interaction. Realistic and reliable interaction is reached through the combination of robust grasping heuristics and plausible pseudophysical object reactions. The easy-to-compute grasping rules use the objects’ surface normals, and mimic human grasping behavior. The novel concept of Normal Proxies increases grasping stability and diminishes challenges induced by adverse normals. The intricate act of picking-up thin and tiny objects remains challenging for some users. These cases are further supported by the consideration of finger pinches, which are measured with a specialized finger tracking device. With regard to typical object constraints, realistic object motion is geometrically calculated as a plausible reaction on user input. The resulting direct finger-based interaction technique enables realistic and intuitive manipulation of arbitrary objects. The thesis proposes two methods that prove and compare effectiveness and usability. An expert review indicates that experienced users quickly familiarize themselves with the technique. A quantitative and qualitative user study shows that direct finger-based interaction is preferred over indirect interaction in the context of functional car assessments. While controller-based interaction is more robust, the direct finger-based interaction provides greater realism, and becomes nearly as reliable when the pinch-sensitive mechanism is used. At present, the haptic channel is not used in industrial virtual reality applications. That is why it can be used for grasping feedback which improves the users’ understanding of the grasping situation. This thesis realizes a novel pressure-based tactile feedback at the fingertips. As an alternative, vibro-tactile feedback at the same location is realized as well as visual feedback by the coloring of grasp-involved finger segments. The feedback approaches are also compared within the user study, which reveals that grasping feedback is a requirement to judge grasp status and that tactile feedback improves interaction independent of the used display system. The considerably stronger vibrational tactile feedback can quickly become annoying during interaction. The interaction improvements and hardware enhancements make it possible to interact with virtual objects in a realistic and reliable manner. By addressing realism and reliability, this thesis paves the way for the virtual evaluation of human-object interaction, which is necessary for a broader application of virtual environments in the automotive industry and other domains.Stärker als andere Branchen benötigt die Automobilindustrie realistische Virtual Reality Anwendungen für eine effiziente Produktentwicklung. Während sich die visuelle Qualität virtueller Darstellungen bereits der Realität angenähert hat, ist die Interaktion mit virtuellen Umgebungen noch weit vom täglichen Erleben der Menschen entfernt. Einige Forschungsansätze haben sich mit realistischer Interaktion befasst, gehen aber nicht weit genug, um in industriellen Prozessen eingesetzt zu werden. Diese Arbeit realisiert eine lebensnahe mehrhändige und fingerbasierte Interaktion mit beliebigen Objekten. Dabei ermöglichen algorithmische und technische Verbesserungen eine realitätsnahe Usability. Außerdem werden Methoden für die Evaluation dieser Interaktionstechnik vorgestellt und benutzerunterstützende Greiffeedbackarten diskutiert. Die verlässliche und gleichzeitig realistische Interaktion wird durch die Kombination von robusten Greifheuristiken und pseudophysikalischen Objektreaktionen erreicht. Die das menschliche Greifverhalten nachbildenden Greifregeln basieren auf den Oberflächennormalen der Objekte. Die Reduktion negativer Einflüsse verfälschter Normalen und eine höhere Griffstabilität werden durch das neuartige Konzept der Normal Proxies erreicht. Dennoch bleibt für manche Nutzer das Aufnehmen von dünnen und kleinen Objekten problematisch. Diese Fälle werden zusätzlich durch die Einbeziehung von Fingerberührungen unterstützt, die mit einem speziellen Fingertracking Gerät erfasst werden. Plausible Objektreaktionen auf Benutzereingaben werden unter Berücksichtigung typischer Objekteinschränkungen geometrisch berechnet. Die Arbeit schlägt zwei Methoden zur Evaluierung der fingerbasierten Interaktion vor. Ein Expertenreview zeigt, dass sich erfahrene Benutzer sehr schnell in die Technik einfinden. In einer Benutzerstudie wird nachgewiesen, dass fingerbasierte Interaktion im hier untersuchten Kontext vor indirekter Interaktion mit einem Eingabegerät bevorzugt wird. Während letztere robuster zu handhaben ist, stellt die fingerbasierte Interaktion einen deutlich höheren Realismus bereit und erreicht mit den vorgeschlagenen Verbesserungen eine vergleichbare Verlässlichkeit. Um Greifsituationen transparent zu gestalten, realisiert diese Arbeit ein neuartiges druckbasiertes taktiles Feedback an den Fingerspitzen. Alternativ wird ein vibrotaktiles Feedback am gleichen Ort realisiert und visuelles Feedback durch die Einfärbung der griffbeteiligten Fingersegmente umgesetzt. Die verschiedenen Feedbackansätze werden in der Benutzerstudie verglichen. Dabei wird Greiffeedback als Voraussetzung identifiziert, um den Greifzustand zu beurteilen. Taktiles Feedback verbessert dabei die Interaktion unabhängig vom eingesetzten Display. Das merklich stärkere Vibrationsfeedback kann während der Interaktion störend wirken. Die vorgestellten Interaktionsverbesserungen und Hardwareerweiterungen ermöglichen es, mit virtuellen Objekten auf realistische und zuverlässige Art zu interagieren. Indem die Arbeit Realismus und Verlässlichkeit gleichzeitig adressiert, bereitet sie den Boden für die virtuelle Untersuchung von Mensch-Objekt Interaktionen und ermöglicht so einen breiteren Einsatz virtueller Techniken in der Automobilindustrie und in anderen Bereichen

Realistic Interaction with Virtual Objects within Arm's Reach

The automotive industry requires realistic virtual reality applications more than other domains to increase the efficiency of product development. Currently, the visual quality of virtual invironments resembles reality, but interaction within these environments is usually far from what is known in everyday life. Several realistic research approaches exist, however they are still not all-encompassing enough to be usable in industrial processes. This thesis realizes lifelike direct multi-hand and multi-finger interaction with arbitrary objects, and proposes algorithmic and technical improvements that also approach lifelike usability. In addition, the thesis proposes methods to measure the effectiveness and usability of such interaction techniques as well as discusses different types of grasping feedback that support the user during interaction. Realistic and reliable interaction is reached through the combination of robust grasping heuristics and plausible pseudophysical object reactions. The easy-to-compute grasping rules use the objects’ surface normals, and mimic human grasping behavior. The novel concept of Normal Proxies increases grasping stability and diminishes challenges induced by adverse normals. The intricate act of picking-up thin and tiny objects remains challenging for some users. These cases are further supported by the consideration of finger pinches, which are measured with a specialized finger tracking device. With regard to typical object constraints, realistic object motion is geometrically calculated as a plausible reaction on user input. The resulting direct finger-based interaction technique enables realistic and intuitive manipulation of arbitrary objects. The thesis proposes two methods that prove and compare effectiveness and usability. An expert review indicates that experienced users quickly familiarize themselves with the technique. A quantitative and qualitative user study shows that direct finger-based interaction is preferred over indirect interaction in the context of functional car assessments. While controller-based interaction is more robust, the direct finger-based interaction provides greater realism, and becomes nearly as reliable when the pinch-sensitive mechanism is used. At present, the haptic channel is not used in industrial virtual reality applications. That is why it can be used for grasping feedback which improves the users’ understanding of the grasping situation. This thesis realizes a novel pressure-based tactile feedback at the fingertips. As an alternative, vibro-tactile feedback at the same location is realized as well as visual feedback by the coloring of grasp-involved finger segments. The feedback approaches are also compared within the user study, which reveals that grasping feedback is a requirement to judge grasp status and that tactile feedback improves interaction independent of the used display system. The considerably stronger vibrational tactile feedback can quickly become annoying during interaction. The interaction improvements and hardware enhancements make it possible to interact with virtual objects in a realistic and reliable manner. By addressing realism and reliability, this thesis paves the way for the virtual evaluation of human-object interaction, which is necessary for a broader application of virtual environments in the automotive industry and other domains.Stärker als andere Branchen benötigt die Automobilindustrie realistische Virtual Reality Anwendungen für eine effiziente Produktentwicklung. Während sich die visuelle Qualität virtueller Darstellungen bereits der Realität angenähert hat, ist die Interaktion mit virtuellen Umgebungen noch weit vom täglichen Erleben der Menschen entfernt. Einige Forschungsansätze haben sich mit realistischer Interaktion befasst, gehen aber nicht weit genug, um in industriellen Prozessen eingesetzt zu werden. Diese Arbeit realisiert eine lebensnahe mehrhändige und fingerbasierte Interaktion mit beliebigen Objekten. Dabei ermöglichen algorithmische und technische Verbesserungen eine realitätsnahe Usability. Außerdem werden Methoden für die Evaluation dieser Interaktionstechnik vorgestellt und benutzerunterstützende Greiffeedbackarten diskutiert. Die verlässliche und gleichzeitig realistische Interaktion wird durch die Kombination von robusten Greifheuristiken und pseudophysikalischen Objektreaktionen erreicht. Die das menschliche Greifverhalten nachbildenden Greifregeln basieren auf den Oberflächennormalen der Objekte. Die Reduktion negativer Einflüsse verfälschter Normalen und eine höhere Griffstabilität werden durch das neuartige Konzept der Normal Proxies erreicht. Dennoch bleibt für manche Nutzer das Aufnehmen von dünnen und kleinen Objekten problematisch. Diese Fälle werden zusätzlich durch die Einbeziehung von Fingerberührungen unterstützt, die mit einem speziellen Fingertracking Gerät erfasst werden. Plausible Objektreaktionen auf Benutzereingaben werden unter Berücksichtigung typischer Objekteinschränkungen geometrisch berechnet. Die Arbeit schlägt zwei Methoden zur Evaluierung der fingerbasierten Interaktion vor. Ein Expertenreview zeigt, dass sich erfahrene Benutzer sehr schnell in die Technik einfinden. In einer Benutzerstudie wird nachgewiesen, dass fingerbasierte Interaktion im hier untersuchten Kontext vor indirekter Interaktion mit einem Eingabegerät bevorzugt wird. Während letztere robuster zu handhaben ist, stellt die fingerbasierte Interaktion einen deutlich höheren Realismus bereit und erreicht mit den vorgeschlagenen Verbesserungen eine vergleichbare Verlässlichkeit. Um Greifsituationen transparent zu gestalten, realisiert diese Arbeit ein neuartiges druckbasiertes taktiles Feedback an den Fingerspitzen. Alternativ wird ein vibrotaktiles Feedback am gleichen Ort realisiert und visuelles Feedback durch die Einfärbung der griffbeteiligten Fingersegmente umgesetzt. Die verschiedenen Feedbackansätze werden in der Benutzerstudie verglichen. Dabei wird Greiffeedback als Voraussetzung identifiziert, um den Greifzustand zu beurteilen. Taktiles Feedback verbessert dabei die Interaktion unabhängig vom eingesetzten Display. Das merklich stärkere Vibrationsfeedback kann während der Interaktion störend wirken. Die vorgestellten Interaktionsverbesserungen und Hardwareerweiterungen ermöglichen es, mit virtuellen Objekten auf realistische und zuverlässige Art zu interagieren. Indem die Arbeit Realismus und Verlässlichkeit gleichzeitig adressiert, bereitet sie den Boden für die virtuelle Untersuchung von Mensch-Objekt Interaktionen und ermöglicht so einen breiteren Einsatz virtueller Techniken in der Automobilindustrie und in anderen Bereichen

차량용 헤드업 디스플레이 설계에 관한 인간공학 연구

학위논문 (박사) -- 서울대학교 대학원 : 공과대학 산업공학과, 2020. 8. 박우진.Head-up display (HUD) systems were introduced into the automobile industry as a means for improving driving safety. They superimpose safety-critical information on top of the drivers forward field of view and thereby help drivers keep their eyes forward while driving. Since the first introduction about three decades ago, automotive HUDs have been available in various commercial vehicles. Despite the long history and potential benefits of automotive HUDs, however, the design of useful automotive HUDs remains a challenging problem. In an effort to contribute to the design of useful automotive HUDs, this doctoral dissertation research conducted four studies. In Study 1, the functional requirements of automotive HUDs were investigated by reviewing the major automakers' automotive HUD products, academic research studies that proposed various automotive HUD functions, and previous research studies that surveyed drivers HUD information needs. The review results indicated that: 1) the existing commercial HUDs perform largely the same functions as the conventional in-vehicle displays, 2) past research studies proposed various HUD functions for improving driver situation awareness and driving safety, 3) autonomous driving and other new technologies are giving rise to new HUD information, and 4) little research is currently available on HUD users perceived information needs. Based on the review results, this study provides insights into the functional requirements of automotive HUDs and also suggests some future research directions for automotive HUD design. In Study 2, the interface design of automotive HUDs for communicating safety-related information was examined by reviewing the existing commercial HUDs and display concepts proposed by academic research studies. Each display was analyzed in terms of its functions, behaviors and structure. Also, related human factors display design principles, and, empirical findings on the effects of interface design decisions were reviewed when information was available. The results indicated that: 1) information characteristics suitable for the contact-analog and unregistered display formats, respectively, are still largely unknown, 2) new types of displays could be developed by combining or mixing existing displays or display elements at both the information and interface element levels, and 3) the human factors display principles need to be used properly according to the situation and only to the extent that the resulting display respects the limitations of the human information processing, and achieving balance among the principles is important to an effective design. On the basis of the review results, this review suggests design possibilities and future research directions on the interface design of safety-related automotive HUD systems. In Study 3, automotive HUD-based take-over request (TOR) displays were developed and evaluated in terms of drivers take-over performance and visual scanning behavior in a highly automated driving situation. Four different types of TOR displays were comparatively evaluated through a driving simulator study - they were: Baseline (an auditory beeping alert), Mini-map, Arrow, and Mini-map-and-Arrow. Baseline simply alerts an imminent take-over, and was always included when the other three displays were provided. Mini-map provides situational information. Arrow presents the action direction information for the take-over. Mini-map-and-Arrow provides the action direction together with the relevant situational information. This study also investigated the relationship between drivers initial trust in the TOR displays and take-over and visual scanning behavior. The results indicated that providing a combination of machine-made decision and situational information, such as Mini-map-and-Arrow, yielded the best results overall in the take-over scenario. Also, drivers initial trust in the TOR displays was found to have significant associations with the take-over and visual behavior of drivers. The higher trust group primarily relied on the proposed TOR displays, while the lower trust group tended to more check the situational information through the traditional displays, such as side-view or rear-view mirrors. In Study 4, the effect of interactive HUD imagery location on driving and secondary task performance, driver distraction, preference, and workload associated with use of scrolling list while driving were investigated. A total of nine HUD imagery locations of full-windshield were examined through a driving simulator study. The results indicated the HUD imagery location affected all the dependent measures, that is, driving and task performance, drivers visual distraction, preference and workload. Considering both objective and subjective evaluations, interactive HUDs should be placed near the driver's line of sight, especially near the left-bottom on the windshield.자동차 헤드업 디스플레이는 차내 디스플레이 중 하나로 운전자에게 필요한 정보를 전방에 표시함으로써, 운전자가 운전을 하는 동안 전방으로 시선을 유지할 수 있게 도와준다. 이를 통해 운전자의 주의 분산을 줄이고, 안전을 향상시키는데 도움이 될 수 있다. 자동차 헤드업 디스플레이 시스템은 약 30년 전 운전자의 안전을 향상시키기 위한 수단으로 자동차 산업에 처음 도입된 이래로 현재까지 다양한 상용차에서 사용되고 있다. 안전과 편의 측면에서 자동차 헤드업 디스플레이의 사용은 점점 더 증가할 것으로 예상된다. 그러나 이러한 자동차 헤드업 디스플레이의 잠재적 이점과 발전 가능성에도 불구하고, 유용한 자동차 헤드업 디스플레이를 설계하는 것은 여전히 어려운 문제이다. 이에 본 연구는 이러한 문제를 해결하고, 궁극적으로 유용한 자동차 헤드업 디스플레이 설계에 기여하고자 총 4가지 연구를 수행하였다. 첫 번째 연구는 자동차 헤드업 디스플레이의 기능 요구 사항과 관련된 것으로서, 헤드업 디스플레이 시스템을 통해 어떤 정보를 제공할 것인가에 대한 답을 구하고자 하였다. 이에 주요 자동차 제조업체들의 헤드업 디스플레이 제품들과, 자동차 헤드업 디스플레이의 다양한 기능들을 제안한 학술 연구, 그리고 운전자의 정보 요구 사항들을 체계적 문헌 고찰 방법론을 통해 포괄적으로 조사하였다. 자동차 헤드업 디스플레이의 기능적 요구 사항에 대하여 개발자, 연구자, 사용자 측면을 모두 고려한 통합된 지식을 전달하고, 이를 통해 자동차 헤드업 디스플레이의 기능 요구 사항에 대한 향후 연구 방향을 제시하였다. 두 번째 연구는 안전 관련 정보를 제공하는 자동차 헤드업 디스플레이의 인터페이스 설계와 관련된 것으로, 헤드업 디스플레이 시스템을 통해 안전 관련 정보를 어떻게 제공할 것인가에 대한 답을 구하고자 하였다. 실제 자동차들의 헤드업 디스플레이 시스템에서는 어떤 디스플레이 컨셉들이 사용되었는지, 그리고 학계에서 제안된 디스플레이 컨셉들에는 어떤 것들이 있는지 체계적 문헌 고찰 방법론을 통해 검토하였다. 검토된 결과는 각 디스플레이의 기능과 구조, 그리고 작동 방식에 따라 정리되었고, 관련된 인간공학적 디스플레이 설계 원칙과 실험적 연구 결과들을 함께 검토하였다. 검토된 결과를 바탕으로 안전 관련 정보를 제공하는 자동차 헤드업 디스플레이의 인터페이스 설계에 대한 향후 연구 방향을 제시하였다. 세 번째 연구는 자동차 헤드업 디스플레이 기반의 제어권 전환 관련 인터페이스 설계와 평가에 관한 것이다. 제어권 전환이란, 자율주행 상태에서 운전자가 직접 운전을 하는 수동 운전 상태로 전환이 되는 것을 의미한다. 따라서 갑작스런 제어권 전환 요청이 발생하는 경우, 운전자가 안전하게 대처하기 위해서는 빠른 상황 파악과 의사 결정이 필요하게 되고, 이를 효과적으로 도와주기 위한 인터페이스 설계에 대해 연구할 필요성이 있다. 이에 본 연구에서는 자동차 헤드업 디스플레이 기반의 총 4개의 제어권 전환 관련 디스플레이(기준 디스플레이, 미니맵 디스플레이, 화살표 디스플레이, 미니맵과 화살표 디스플레이)를 제안하였고, 제안된 디스플레이 대안들은 주행 시뮬레이터 실험을 통해 제어권 전환 수행 능력과 안구의 움직임 패턴, 그리고 사용자의 주관적 평가 측면에서 평가되었다. 또한 제안된 디스플레이 대안들에 대해 운전자들의 초기 신뢰도 값을 측정하여 각 디스플레이에 따른 운전자들의 평균 신뢰도 점수에 따라 제어권 전환 수행 능력과 안구의 움직임 패턴, 그리고 주관적 평가가 어떻게 달라지는지 분석하였다. 실험 결과, 제어권 전환 상황에서 자동화된 시스템이 제안하는 정보와 그와 관련된 주변 상황 정보를 함께 제시해 주는 디스플레이가 가장 좋은 결과를 보여주었다. 또한 각 디스플레이에 대한 운전자의 초기 신뢰도 점수는 디스플레이의 실제 사용 행태와 밀접한 관련이 있음을 알 수 있었다. 신뢰도 점수에 따라 신뢰도가 높은 그룹과 낮은 그룹으로 분류되었고, 신뢰도가 높은 그룹은 제안된 디스플레이들이 보여주는 정보를 주로 믿고 따르는 경향이 있었던 반면, 신뢰도가 낮은 그룹은 룸 미러나 사이드 미러를 통해 주변 상황 정보를 더 확인 하는 경향을 보였다. 네 번째 연구는 전면 유리창에서의 인터랙티브 헤드업 디스플레이의 최적 위치를 결정하는 것으로서 주행 시뮬레이터 실험을 통해 디스플레이의 위치에 따라 운전자의 주행 수행 능력, 인터랙티브 디스플레이 조작 관련 과업 수행 능력, 시각적 주의 분산, 선호도, 그리고 작업 부하가 평가되었다. 헤드업 디스플레이의 위치는 전면 유리창에서 일정한 간격으로 총 9개의 위치가 고려되었다. 본 연구에서 활용된 인터랙티브 디스플레이는 음악 선택을 위한 스크롤 방식의 단일 디스플레이였고, 운전대에 장착된 버튼을 통해 디스플레이를 조작하였다. 실험 결과, 인터랙티브 헤드업 디스플레이의 위치가 모든 평가 척도, 즉 주행 수행 능력, 디스플레이 조작 과업 수행 능력, 시각적 주의 분산, 선호도, 그리고 작업 부하에 영향을 미침을 알 수 있었다. 모든 평가 지표를 고려했을 때, 인터랙티브 헤드업 디스플레이의 위치는 운전자가 똑바로 전방을 바라볼 때의 시야 구간, 즉 전면 유리창에서의 왼쪽 아래 부근이 가장 최적인 것으로 나타났다.Abstract i Contents v List of Tables ix List of Figures x Chapter 1 Introduction 1 1.1 Research Background 1 1.2 Research Objectives and Questions 8 1.3 Structure of the Thesis 11 Chapter 2 Functional Requirements of Automotive Head-Up Displays: A Systematic Review of Literature from 1994 to Present 13 2.1 Introduction 13 2.2 Method 15 2.3 Results 17 2.3.1 Information Types Displayed by Existing Commercial Automotive HUD Systems 17 2.3.2 Information Types Previously Suggested for Automotive HUDs by Research Studies 28 2.3.3 Information Types Required by Drivers (users) for Automotive HUDs and Their Relative Importance 35 2.4 Discussion 39 2.4.1 Information Types Displayed by Existing Commercial Automotive HUD Systems 39 2.4.2 Information Types Previously Suggested for Automotive HUDs by Research Studies 44 2.4.3 Information Types Required by Drivers (users) for Automotive HUDs and Their Relative Importance 48 Chapter 3 A Literature Review on Interface Design of Automotive Head-Up Displays for Communicating Safety-Related Information 50 3.1 Introduction 50 3.2 Method 52 3.3 Results 55 3.3.1 Commercial Automotive HUDs Presenting Safety-Related Information 55 3.3.2 Safety-Related HUDs Proposed by Academic Research 58 3.4 Discussion 74 Chapter 4 Development and Evaluation of Automotive Head-Up Displays for Take-Over Requests (TORs) in Highly Automated Vehicles 78 4.1 Introduction 78 4.2 Method 82 4.2.1 Participants 82 4.2.2 Apparatus 82 4.2.3 Automotive HUD-based TOR Displays 83 4.2.4 Driving Scenario 86 4.2.5 Experimental Design and Procedure 87 4.2.6 Experiment Variables 88 4.2.7 Statistical Analyses 91 4.3 Results 93 4.3.1 Comparison of the Proposed TOR Displays 93 4.3.2 Characteristics of Drivers Initial Trust in the four TOR Displays 102 4.3.3 Relationship between Drivers Initial Trust and Take-over and Visual Behavior 104 4.4 Discussion 113 4.4.1 Comparison of the Proposed TOR Displays 113 4.4.2 Characteristics of Drivers Initial Trust in the four TOR Displays 116 4.4.3 Relationship between Drivers Initial Trust and Take-over and Visual Behavior 117 4.5 Conclusion 119 Chapter 5 Human Factors Evaluation of Display Locations of an Interactive Scrolling List in a Full-windshield Automotive Head-Up Display System 121 5.1 Introduction 121 5.2 Method 122 5.2.1 Participants 122 5.2.2 Apparatus 123 5.2.3 Experimental Tasks and Driving Scenario 123 5.2.4 Experiment Variables 124 5.2.5 Experimental Design and Procedure 126 5.2.6 Statistical Analyses 126 5.3 Results 127 5.4 Discussion 133 5.5 Conclusion 135 Chapter 6 Conclusion 137 6.1 Summary and Implications 137 6.2 Future Research Directions 139 Bibliography 143 Apeendix A. Display Layouts of Some Commercial HUD Systems Appendix B. Safety-related Displays Provided by the Existing Commercial HUD Systems Appendix C. Safety-related HUD displays Proposed by Academic Research 국문초록 187Docto