A survey on hardware and software solutions for multimodal wearable assistive devices targeting the visually impaired

The market penetration of user-centric assistive devices has rapidly increased in the past decades. Growth in computational power, accessibility, and cognitive device capabilities have been accompanied by significant reductions in weight, size, and price, as a result of which mobile and wearable equipment are becoming part of our everyday life. In this context, a key focus of development has been on rehabilitation engineering and on developing assistive technologies targeting people with various disabilities, including hearing loss, visual impairments and others. Applications range from simple health monitoring such as sport activity trackers, through medical applications including sensory (e.g. hearing) aids and real-time monitoring of life functions, to task-oriented tools such as navigational devices for the blind. This paper provides an overview of recent trends in software and hardware-based signal processing relevant to the development of wearable assistive solutions

자율주행 맥락하에서 차량 내 청각 신호음에 대한 사용자 선호도 분석 연구

학위논문(석사) -- 서울대학교대학원 : 공과대학 산업공학과, 2022. 8. 윤명환.최근에는 차량 내에 자율 기술의 발달로 자율주행차의 기능이 고도화가 이루어졌고, 우리가 알고 있는 인간-차량 인터랙션은 점점 인간-로봇 인터랙션으로 패러다임이 변하고 있다. 차량 내에서 청각 유저 인터페이스는 운전자의 인지 부하를 줄이고, 운전자에게 정보를 제공하기 위해 차량 기술에 사용되고 있다. 그러나, 자율주행차는 새로운 기술 도메인으로 인해 사용자, 즉, 탑승자 사용맥락과 시나리오에 따라 청각 피드백 유형 설계가 필요하다. 본 논문에서는 세 가지 주요 연구 목표는 (1) 탑승자의 관점에 기반한 자율주행차량에 대한 직관적인 청각 피드백 설계 제안, (2) 자율주행차량에 적용된 청각 피드백에 대한 선호도, 그리고, (3) 자율주행차에서 필요한 청각 사용자 경험 시나리오 도출하는 것이다. 본 논문의 연구 목적을 달성하기 위해 제작된 청각적 피드백의 인지용이성, 직관성 일관성 또는 적절성을 측정하여 청각 피드백의 유형 및 정보 유형의 패턴이 탑승객의 선호에 영향을 미치는지 조사하는 방향을 잡았다. 본 논문은 파일럿 테스트와 대규모 온라인 사운드 평가로 실험을 진행하였다. 파일럿 테스트에는 총 13명 참가자 연령 27.23세(±7.53) 대상으로 실시하였고, 제작된 사운드 샘플의 의도된 정보(조작 확인음, 조작 오류음, 감지음, 진행음, 약경고음, 강경고음)와 인지적으로 용이성과 직관성이 있는지 진행하였다. 추가적으로, 청각 피드백이 필요한 자율주행차량의 시나리오를 도출하기 위해서도 빈도분석을 진행하였다. 인지용이성과 직관성 평가를 통해 얻은 데이터는 분산분석(ANOVA)과 다중 비교를 위한 본페로니 사후 검정을 사용하여 분석하였다. 파일럿 테스트 결과, 진행음 피드백을 제외하고 모든 사운드 샘플이 의도된 정보로 직관적으로 설계된 것으로 확인하였다. 따라서, 본 논문에서 사용되는 진행음을 다시 설계 및 제작해야 하는 것이다. 또한, 개발된 27가지 시나리오 중 탑승객의 사용맥락에 기반한 청각 피드백이 필요한 상황이 15가지 필수 시나리오로 도출하였다. 파일럿 테스트를 이어서, 사운드 평가는 평균 연령이 37.15세(±11.4)인 총 125명의 참가자를 대상으로 온라인으로 대규모 수행되었으며, 7점 척도로 일관성/적절성 측정을 통해서 어떤 사운드 유형 (이어콘과 오디토리 아이콘의 혼합 또는 일련의 이어콘/오디토리 아이콘)을 선호하는지 조사하였다. 진행음은 올라가는 멜로, 내려가는 멜로, 변형 및 단순 음색으로 4가지 파라미터로 재제작하였다. 본 평가에서 얻어낸 일관성/적절성 데이터는 각 사운드 세트에 대한 쌍별 t-테스트 비교를 사용하여 분석하였다. 진행음은 만족도로 측정하여 분산분석(ANOVA)을 사용하여 분석하였다. 마지막으로, 참가자들의 의견들을 정성적 분석을 위해 텍스트 네트워크 분석으로 시각화를 하였다. 각 시나리오에 독립적인 표번 t-테스트의 결과에 따르면, 사용자가 자율주행차량 탑승의 시나리오에서 이어콘과 오디토리 아이콘의 혼합보다는 일관된 사운드 세트를 선호한다는 결과가 나타났다. 또한, 진행음의 결과는 내려가는 멜로디와 단순 음색의 파라미터로 높은 만족도가 보였다. 마지막으로, 본 논문의 토의 부분에서는 파일럿 테스트와 온라인 대규모 사운드 평가 실험을 통해서 얻은 결과를 바탕으로 연구 목표의 달성에 대해 토의하였다. 결론 부분에서는 본 논문의 한계점과 향후 연구 방향에 대해서도 논의하였다.The rise of autonomous technology that has been incorporated into vehicles allows the autonomous vehicles to shifted its functionality as an interactive system where providing interaction and feedback between the user and system is essential. In addition, auditory user interface has been used in vehicle technology to reduce cognitive workload and provide information to the drivers. However, autonomous vehicle is still regarded as a new technology domain, and it is necessary to investigate what type of in-vehicle signals feedback that should be designed to the passenger depending on the context-of-use and scenarios involved. In this thesis, the three main research aims are; (1) to present a design proposal for in-vehicle signals feedback for autonomous vehicles based on passenger’s perspective, (2) to explore the passenger’s sound preference for in-vehicle signals feedback used in autonomous vehicle, and (3) to suggest a fully derived scenario when designing an in-vehicle signals feedback used in autonomous vehicles based on user-centered design process. To achieve the research aim, this thesis focuses on investigating whether the design of in-vehicle signal types such as earcon and auditory icon, and temporal pattern of information signal types would affect the passenger’s preferences by measuring its perceivability, intuitiveness and consistency or appropriateness as an in-vehicle signal. This thesis includes two experiments; a pilot test and a large scale online sound evaluation study. Prior to the sound set evaluation, a pilot test was conducted on a total of 13 participants with an average age of 27.23(±7.53) to investigate whether the auditory sound sample that was created for sound evaluation has the congruity that matches with the intended information (confirmatory, error, detection, in progress, alert and warning), and to further develop the scenario for passengers in autonomous vehicles context. There were two measures used for the pilot test, which is perceivability and intuitiveness to determine if the designed sound sample with temporal pattern matches with the intended information as this paper suggested. The pilot test was conducted in an acoustic chamber, and participants were asked to give their evaluation in a 7-points Likert scale for perceivability and intuitiveness of the sound samples, and conducted survey of multiple choices to select the appropriate scenarios for each sound. The data obtained for perceivability and intuitiveness were analyzed using analysis of variance (ANOVA) and Bonferroni correction post-hoc test for multiple comparisons. Result of the pilot test shown that all sound samples are perceivable intuitively designed with the intended information, except for in progress type signal. Hence, in progress type signals will need to be re-created for this study. Also, out of the 27 scenarios that was developed prior to the pilot study, this study narrowed down 15 essential scenarios which in-vehicle signal feedbacks are imperative to autonomous vehicles based on passenger’s context. The sound set evaluation was conducted online with a total of 125 participants with an average age of 37.15(±11.4) to investigate which type of sounds (a mixture of earcons and auditory icons, or a set of earcon/auditory icon consecutively) they prefer by measuring consistency/appropriateness measure in 7-points Likert scale. In progress sounds were re-created in ascending, descending, variated and simple tone parameters, and were evaluated by its satisfaction measures. The data obtained for consistency/appropriateness were analyzed using pairwise t-test comparison for each sound sets. The in-progress sounds were analyzed using four-way analysis of variance (ANOVA). Lastly, all of the participants’ opinions were collected for qualitative analysis by performing text network analysis for visualization. Results from the independent samples t-tests for each scenario shown that users or listeners prefer a consistent ‘family’ of sounds, rather than a mixture of earcons and auditory icons in a scenario. The result from the in-progress sounds also shows that a descending-simple tone melody sounds has high satisfaction level. In the discussion, this study discussed whether the research aim is fulfilled based on the results obtained and added implications for the sound design. In summary and conclusion, this study also discussed the limitation of this study and the future direction.Chapter 1 Introduction 1 1.1 Research Background 1 1.2 Research Objectives 6 1.3 Organization of the Thesis 6 Chapter 2 Literature Review 7 2.1 Auditory Types 7 2.1.1 Earcons and Auditory Icons 7 2.1.2 Auditory Information Types 8 2.1.3 Acoustic Parameters 11 2.2 Auditory User Interface (AUI) 12 2.2.1 Auditory User Interface (AUI) in Autonomous Vehicles 12 2.2.2 Auditory User Experience Measurement 12 2.3 Ideation for Scenario Development for Autonomous Vehicles 14 2.3.1 Context-of-use of Autonomous Vehicles 14 2.3.2 Scenario Development 15 Chapter 3 Sound Experiment and Evaluation 19 3.1 Pilot Test 19 3.1.1 Overview and aim for pilot test 19 3.1.2 Participants 19 3.1.3 Stimuli 20석

Human-Computer Interaction

In this book the reader will find a collection of 31 papers presenting different facets of Human Computer Interaction, the result of research projects and experiments as well as new approaches to design user interfaces. The book is organized according to the following main topics in a sequential order: new interaction paradigms, multimodality, usability studies on several interaction mechanisms, human factors, universal design and development methodologies and tools

Naturalness framework for driver-car interaction

This thesis was submitted for the award of Doctor of Philosophy and was awarded by Brunel University LondonAutomobile dashboards are evolving into intelligent largely screen-based computer interfaces. Recent evidence suggests unnatural aspects of some secondary systems (including infotainment systems). Naturalness of interaction is a minority discipline not yet applied to the automobile; while automotive interface research is a mainly quantitative discipline that has not yet applied a naturalness approach. To advance the field, a measurement scale based on rigorous qualitative exploration of natural-feeling interaction with secondary controls was required. Study 1 used ethnographic interview with Contextual Inquiry inside 12 ordinary drivers’ cars, to investigate natural-feeling aspects of past, present and future driver-car interactions. Thematic analysis suggested a framework of ten characteristics. Half concerned control and physicality; half concerned perceived socio-intelligent behaviours of the car. Study 2 involved intensive exploratory workshops with ten drivers comprising Think Aloud, artefact modelling and focus groups, to explore natural-feeling interaction with secondary controls in different ways. The resulting thematic framework comprised 11 characteristics in four categories: familiarity/control, physical connection, low visual/cognitive demand, and humanlike intelligence and communication. Study 3 comprised two ethnographic participant observations. Eight drivers were observed interacting with their controls during long road journeys. Twenty-two drivers were observed interacting verbally with futuristic ‘intelligent’ secondary systems while driving on public roads. Design guidelines relating to physicality, usability, automation, and humanlike communication were formulated. Study 4 converted all the qualitative findings into a questionnaire comprising 46 bipolar five-point scales. Eighty-one drivers used it to rate one control in their cars. Correlation and factor analyses revealed three underlying factors and 14 items suitable for the first industrially applicable measurement scale for driver-car naturalness. These items concern perceived helpfulness, politeness, competence, predictability, control, ease, mental demands, intuitiveness, ‘realness’, instantaneousness, communication, logical location, mapping and 'affordance'. Initial testing found acceptable validity. The conclusion recommends further data collection, expanded validity testing, and potential applications to self-driving cars

PRESTK : situation-aware presentation of messages and infotainment content for drivers

The amount of in-car information systems has dramatically increased over the last few years. These potentially mutually independent information systems presenting information to the driver increase the risk of driver distraction. In a first step, orchestrating these information systems using techniques from scheduling and presentation planning avoid conflicts when competing for scarce resources such as screen space. In a second step, the cognitive capacity of the driver as another scarce resource has to be considered. For the first step, an algorithm fulfilling the requirements of this situation is presented and evaluated. For the second step, I define the concept of System Situation Awareness (SSA) as an extension of Endsley’s Situation Awareness (SA) model. I claim that not only the driver needs to know what is happening in his environment, but also the system, e.g., the car. In order to achieve SSA, two paths of research have to be followed: (1) Assessment of cognitive load of the driver in an unobtrusive way. I propose to estimate this value using a model based on environmental data. (2) Developing model of cognitive complexity induced by messages presented by the system. Three experiments support the claims I make in my conceptual contribution to this field. A prototypical implementation of the situation-aware presentation management toolkit PRESTK is presented and shown in two demonstrators.In den letzten Jahren hat die Menge der informationsanzeigenden Systeme im Auto drastisch zugenommen. Da sie potenziell unabhängig voneinander ablaufen, erhöhen sie die Gefahr, die Aufmerksamkeit des Fahrers abzulenken. Konflikte entstehen, wenn zwei oder mehr Systeme zeitgleich auf limitierte Ressourcen wie z. B. den Bildschirmplatz zugreifen. Ein erster Schritt, diese Konflikte zu vermeiden, ist die Orchestrierung dieser Systeme mittels Techniken aus dem Bereich Scheduling und Präsentationsplanung. In einem zweiten Schritt sollte die kognitive Kapazität des Fahrers als ebenfalls limitierte Ressource berücksichtigt werden. Der Algorithmus, den ich zu Schritt 1 vorstelle und evaluiere, erfüllt alle diese Anforderungen. Zu Schritt 2 definiere ich das Konzept System Situation Awareness (SSA), basierend auf Endsley’s Konzept der Situation Awareness (SA). Dadurch wird erreicht, dass nicht nur der Fahrer sich seiner Umgebung bewusst ist, sondern auch das System (d.h. das Auto). Zu diesem Zweck m¨ussen zwei Bereiche untersucht werden: (1) Die kognitive Belastbarkeit des Fahrers unaufdringlich ermitteln. Dazu schlage ich ein Modell vor, das auf Umgebungsinformationen basiert. (2) Ein weiteres Modell soll die Komplexität der präsentierten Informationen bestimmen. Drei Experimente stützen die Behauptungen in meinem konzeptuellen Beitrag. Ein Prototyp des situationsbewussten Präsentationsmanagement-Toolkits PresTK wird vorgestellt und in zwei Demonstratoren gezeigt

Autonomous interactive intermediaries : social intelligence for mobile communication agents

Thesis (Ph. D.)--Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2005.Includes bibliographical references (p. 151-167).Today's cellphones are passive communication portals. They are neither aware of our conversational settings, nor of the relationship between caller and callee, and often interrupt us at inappropriate times. This thesis is about adding elements of human style social intelligence to our mobile communication devices in order to make them more socially acceptable to both user and local others. I suggest the concept of an Autonomous Interactive Intermediary that assumes the role of an actively mediating party between caller, callee, and co-located people. In order to behave in a socially appropriate way, the Intermediary interrupts with non-verbal cues and attempts to harvest 'residual social intelligence' from the calling party, the called person, the people close by, and its current location. For example, the Intermediary obtains the user's conversational status from a decentralized network of autonomous body-worn sensor nodes. These nodes detect conversational groupings in real time, and provide the Intermediary with the user's conversation size and talk-to-listen ratio. The Intermediary can 'poll' all participants of a face-to-face conversation about the appropriateness of a possible interruption by slightly vibrating their wirelessly actuated finger rings.(cont.) Although the alerted people do not know if it is their own cellphone that is about to interrupt, each of them can veto the interruption anonymously by touching his/her ring. If no one vetoes, the Intermediary may interrupt. A user study showed significantly more vetoes during a collaborative group-focused setting than during a less group oriented setting. The Intermediary is implemented as a both a conversational agent and an animatronic device. The animatronics is a small wireless robotic stuffed animal in the form of a squirrel, bunny, or parrot. The purpose of the embodiment is to employ intuitive non-verbal cues such as gaze and gestures to attract attention, instead of ringing or vibration. Evidence suggests that such subtle yet public alerting by animatronics evokes significantly different reactions than ordinary telephones and are seen as less invasive by others present when we receive phone calls. The Intermediary is also a dual conversational agent that can whisper and listen to the user, and converse with a caller, mediating between them in real time.(cont.) The Intermediary modifies its conversational script depending on caller identity, caller and user choices, and the conversational status of the user. It interrupts and communicates with the user when it is socially appropriate, and may break down a synchronous phone call into chunks of voice instant messages.by Stefan Johannes Walter Marti.Ph.D

The ecological AUI (Auditory User Interface) design and evaluation of user acceptance for various tasks on smartphones

With the rapid development of the touch screen technology, some usability issues of smartphones have been reported [1]. To tackle those user experience issues, there has been research on the use of non-speech sounds on the mobile devices [e.g., 2, 3-7]. However, most of them have focused on a single specific task of the device. Given the varying functions of the smartphone, the present study designed plausibly integrated auditory cues for diverse functions and evaluated user acceptance levels from the ecological interface design perspective. Results showed that sophisticated auditory design could change users’ preference and acceptance of the interface and the extent depended on usage contexts. Overall, participants gave significantly higher scores on the functional satisfaction and the fun scales in the sonically-enhanced smartphones than in the no-sound condition. The balanced sound design may free users from auditory pollution and allow them to use their devices more pleasantly