Reference Resolution in Multi-modal Interaction: Position paper

In this position paper we present our research on multimodal interaction in and with virtual environments. The aim of this presentation is to emphasize the necessity to spend more research on reference resolution in multimodal contexts. In multi-modal interaction the human conversational partner can apply more than one modality in conveying his or her message to the environment in which a computer detects and interprets signals from different modalities. We show some naturally arising problems and how they are treated for different contexts. No generally applicable solutions are given

Reference resolution in multi-modal interaction: Preliminary observations

In this paper we present our research on multimodal interaction in and with virtual environments. The aim of this presentation is to emphasize the necessity to spend more research on reference resolution in multimodal contexts. In multi-modal interaction the human conversational partner can apply more than one modality in conveying his or her message to the environment in which a computer detects and interprets signals from different modalities. We show some naturally arising problems but do not give general solutions. Rather we decide to perform more detailed research on reference resolution in uni-modal contexts to obtain methods generalizable to multi-modal contexts. Since we try to build applications for a Dutch audience and since hardly any research has been done on reference resolution for Dutch, we give results on the resolution of anaphoric and deictic references in Dutch texts. We hope to be able to extend these results to our multimodal contexts later

Ambient Gestures

We present Ambient Gestures, a novel gesture-based system designed to support ubiquitous ‘in the environment’ interactions with everyday computing technology. Hand gestures and audio feedback allow users to control computer applications without reliance on a graphical user interface, and without having to switch from the context of a non-computer task to the context of the computer. The Ambient Gestures system is composed of a vision recognition software application, a set of gestures to be processed by a scripting application and a navigation and selection application that is controlled by the gestures. This system allows us to explore gestures as the primary means of interaction within a multimodal, multimedia environment. In this paper we describe the Ambient Gestures system, define the gestures and the interactions that can be achieved in this environment and present a formative study of the system. We conclude with a discussion of our findings and future applications of Ambient Gestures in ubiquitous computing

Looking towards the Future of Language Assessment: Usability of Tablet PCs in Language Testing

This research addresses the change in how the Spanish University Entrance Examination can be delivered in the future. There is a wide acknowledgement that computer tests are very demanding for the delivering institutions which makes computer language testing difficult to implement. However, the use of tablet PCs can facilitate the delivery at even lower cost than the regular computer based language testing. 183 students in their last year of high school took a computer based language test which included reading, writing, listening and speaking. The research aspects were 1) whether they feel at ease with the tablet PC exam; 2) if they felt that visual aspects were accessible and 3) whether the interface organization was clear. The paper first has a brief description of the OPENPAU® platform, after it addresses the field study based on questionnaires and observations with students during the test delivery, finally the results indicate that this means of language test delivery could be adequate not only for the Spanish University Entrance Examination but for most standardized tests

User Error Handling Strategies on a Non Visual Multimodal Interface: Preliminary Results from an Exploratory Study

The present study addresses two questions: On a non-visual multimodal interface for textual information browsing, (1) how prevalent is input modality switching as an error handling strategy, and (2) how much does an input modality need to fail before input modality switching occurs. The results indicate that although switching input modalities to correct errors is an expected practice on multimodal GUIs, it is not the prevalent strategy for non-visual multimodal interfaces. We believe that users are more likely to diversify their error handling strategies within a modality, if different strategies are possible, but we have not found conclusive evidence for this belief. However, our analysis suggests that the failure to switch modalities when errors occur may, in part, be due to the prevalence of alternative error handling strategies in a particular input modality, that is, the user prefers to stay in the same modality rather than assume the cognitive load of a switch

Age group differences in performance using diverse input modalities: insertion task evaluation

Novel input modalities such as touch, tangibles or gestures try to exploit human's innate skills rather than imposing new learning processes. However, no work has been reported that systematically evaluates how these interfaces influence users' performance, that is, assesses if one interface can be more or less appropriate for interaction regarding: (1) different age groups; and (2) different basic tasks, as content insertion or manipulation. This work presents itself as an exploratory evaluation about whether or not the users' efficiency is indeed influenced by different input modalities and age. We conducted a usability evaluation with 60 subjects to understand how different interfaces may influence the speed and accuracy of three specific age groups (children, young adults and older-adults) when dealing with a basic content insertion task. Four input modalities were considered to perform the task (keyboard, touch, tangibles and gestures) and the methodology was based on usability testing (speed, accuracy and user preference). Overall, results show that there is a statistically significant difference in speed of task completion between the age groups, and there may be indications that the type of interface that is used can indeed influence efficiency in insertion tasks, and not so much other factors like age. Also, the study raises new issues regarding the "old" mouse input versus the "new" input modalities.FCT – Fundação para a Ciência e a Tecnologia (SFRH/BD/81541/2011)COMPETE: POCI-01-0145- FEDER-007043 and FCT – Fundação para a Ciência e Tecnologia within the Project Scope: UID/CEC/00319/201

a human in the loop cyber physical system for collaborative assembly in smart manufacturing

Abstract Industry 4.0 rose with the introduction of cyber-physical systems (CPS) and Internet of things (IoT) inside manufacturing systems. CPS represent self-controlled physical processes, having tight networking capabilities and efficient interfaces for human interaction. The interactive dimension of CPS reaches its maximum when defined in terms of natural human-machine interfaces (NHMI), i.e., those reducing the technological barriers required for the interaction. This paper presents a NHMI bringing the human decision-making capabilities inside the cybernetic control loop of a smart manufacturing assembly system. The interface allows to control, coordinate and cooperate with an industrial cobot during the task execution

A Study on a Digital Reading Ecosystem based on Content for Flow Improvement

배경 : 디지털환경에서 멀티모달리티의 특성이 강조됨에 따라 읽기경험의 과정이 다양해졌다. 그로 인해 읽기의 과정을 복잡하게 느끼기도 하며 몰입이 어려워지기도 한다. 디지털 리딩에 있어서 몰입감은 읽기의 전 과정이 막힘없이 이루어질 때 나타난다고 가정하였다. 본 논문에서는 콘텐츠에 따라 달라지는 읽기단계와 각 단계에서 요구되는 멀티모달 요소를 밝혀 디지털 리딩 에코시스템 제안을 목적으로 한다. 방법 : 본 논문은 문헌연구를 통해 디지털 리딩의 몰입감에 대해서 정의하고 디지털 리딩의 단계 요소를 추출하였다. 설문조사에서는 논문과 뉴스 두 콘텐츠를 비교하여 읽기단계와 멀티모달 요소들의 차이를 밝혀 디지털 리딩 에코시스템을 제안하였다. 결과 : 문헌연구를 통해 디지털 리딩은 인지적, 정서적, 참여적 몰두가 서로 교차하면서 몰입(flow)이 이루어지는 것을 확인하였다. 또한 디지털 리딩의 과정을 이루는 항목으로 접근, 판단, 수집, 이해, 공유, 창조의 6가지항목이 추출 되었다. 설문조사에서 논문과 뉴스의 읽기단계와 단계별 고려 요소들을 비교한 결과 뉴스는 접근, 판단, 이해, 수집, 공유 순으로 논문은 접근, 판단 또는 수집, 이해, 창조로 응답이 나왔다. 논문의 경우 주로 검색과 북마크, 뉴스는 검색과 공유된 글을 통해 콘텐츠에 접근하고 이해단계에서는 두 콘텐츠모두 빠른 접근이 가장 중요하고 그 다음으로 논문은 표현도구, 종이로 읽기와 유사한 경험으로 응답하였고 뉴스의 경우 SNS통한 소통과 배경지식이 몰입에 도움을 준다고 답하였다. 결론 : 이러한 결과를 토대로 디지털 리딩에서 콘텐츠의 생산과 소비가 하나의 앱에서 이루어짐으로서 끊김 없는 경험을 위한 디지털 리딩 에코시스템을 제안하였다. 이 연구는 디지털 리딩의 몰입감을 향상시키는 독서앱을 디자인하기위해 기초자료로서 활용가치가 있을 것이라 기대한다.Background : As the concept and behavior of reading is augmented in the digital environment, the importance of multi-modality becomes apparent while reading steps grow increasingly diverse and complex. This paper claims that Flow occurs when users move through all of the reading steps seamlessly. In order to improve the Flow of digital reading, we aim to discover and examine the specific reading stages and accompanying multimodal elements supporting each stage depending on the content type and suggest a digital reading ecosystem to promote and realize seamless reading for the achievement of Flow. Methods : In this paper, we conducted a literature review and survey. The literature review was focused on the act of digital reading, multi-modality, Flow, and the derived reading stages. The survey investigated the discrepancies between a newspapers and a research papers reading stages and their multimodal elements. Based on the results we proposed a digital reading ecosystem. Results : Following the literature review, we discovered that digital reading consists of three types of immersion: cognitive immersion created by the readers imagination, affective immersion that is related to interest and motivation, and participatory immersion that relies on the readers engagement. We extracted six reading stages of digital reading: approaching, judging, collecting, comprehending, sharing and creating. The results reveal differences between journal papers and news articles mainly in the reading process and multimodal elements of each stage. The process of journal papers involves the order of approaching, judging & collecting, comprehending and sharing. The process of news articles involves the order of approaching, judging, comprehending, collecting, sharing and creating. In terms of immersion, fast access is the most important factor for both contents. Secondly, reproducing the experience of reading from printed text, and expression tools, are critical elements for journal papers, communications via SNS, and related information are observed as the important elements for news articles. Conclusion : Based on the results, we propose a digital reading ecosystem, which aids in realizing the seamless reading experience. We believe that this study provides useful insights in designing reading apps that encourage Flow in digital reading for knowledge acquisition.OAIID:oai:osos.snu.ac.kr:snu2013-01/102/0000025799/2SEQ:2PERF_CD:SNU2013-01EVAL_ITEM_CD:102USER_ID:0000025799ADJUST_YN:NEMP_ID:A075458DEPT_CD:611CITE_RATE:0FILENAME:첨부된 내역이 없습니다.DEPT_NM:디자인학부EMAIL:[email protected]_YN:NCONFIRM: