Naturalistic digital task modeling for personal information assistance via continuous screen monitoring

Peer reviewe

Data-Driven Evaluation of In-Vehicle Information Systems

Today’s In-Vehicle Information Systems (IVISs) are featurerich systems that provide the driver with numerous options for entertainment, information, comfort, and communication. Drivers can stream their favorite songs, read reviews of nearby restaurants, or change the ambient lighting to their liking. To do so, they interact with large center stack touchscreens that have become the main interface between the driver and IVISs. To interact with these systems, drivers must take their eyes off the road which can impair their driving performance. This makes IVIS evaluation critical not only to meet customer needs but also to ensure road safety. The growing number of features, the distraction caused by large touchscreens, and the impact of driving automation on driver behavior pose significant challenges for the design and evaluation of IVISs. Traditionally, IVISs are evaluated qualitatively or through small-scale user studies using driving simulators. However, these methods are not scalable to the growing number of features and the variety of driving scenarios that influence driver interaction behavior. We argue that data-driven methods can be a viable solution to these challenges and can assist automotive User Experience (UX) experts in evaluating IVISs. Therefore, we need to understand how data-driven methods can facilitate the design and evaluation of IVISs, how large amounts of usage data need to be visualized, and how drivers allocate their visual attention when interacting with center stack touchscreens. In Part I, we present the results of two empirical studies and create a comprehensive understanding of the role that data-driven methods currently play in the automotive UX design process. We found that automotive UX experts face two main conflicts: First, results from qualitative or small-scale empirical studies are often not valued in the decision-making process. Second, UX experts often do not have access to customer data and lack the means and tools to analyze it appropriately. As a result, design decisions are often not user-centered and are based on subjective judgments rather than evidence-based customer insights. Our results show that automotive UX experts need data-driven methods that leverage large amounts of telematics data collected from customer vehicles. They need tools to help them visualize and analyze customer usage data and computational methods to automatically evaluate IVIS designs. In Part II, we present ICEBOAT, an interactive user behavior analysis tool for automotive user interfaces. ICEBOAT processes interaction data, driving data, and glance data, collected over-the-air from customer vehicles and visualizes it on different levels of granularity. Leveraging our multi-level user behavior analysis framework, it enables UX experts to effectively and efficiently evaluate driver interactions with touchscreen-based IVISs concerning performance and safety-related metrics. In Part III, we investigate drivers’ multitasking behavior and visual attention allocation when interacting with center stack touchscreens while driving. We present the first naturalistic driving study to assess drivers’ tactical and operational self-regulation with center stack touchscreens. Our results show significant differences in drivers’ interaction and glance behavior in response to different levels of driving automation, vehicle speed, and road curvature. During automated driving, drivers perform more interactions per touchscreen sequence and increase the time spent looking at the center stack touchscreen. These results emphasize the importance of context-dependent driver distraction assessment of driver interactions with IVISs. Motivated by this we present a machine learning-based approach to predict and explain the visual demand of in-vehicle touchscreen interactions based on customer data. By predicting the visual demand of yet unseen touchscreen interactions, our method lays the foundation for automated data-driven evaluation of early-stage IVIS prototypes. The local and global explanations provide additional insights into how design artifacts and driving context affect drivers’ glance behavior. Overall, this thesis identifies current shortcomings in the evaluation of IVISs and proposes novel solutions based on visual analytics and statistical and computational modeling that generate insights into driver interaction behavior and assist UX experts in making user-centered design decisions

Behavioral Task Modeling for Entity Recommendation

Our everyday tasks involve interactions with a wide range of information. The information that we manage is often associated with a task context. However, current computer systems do not organize information in this way, do not help the user find information in task context, but require explicit user actions such as searching and information seeking. We explore the use of task context to guide the delivery of information to the user proactively, that is, to have the right information easily available at the right time. In this thesis, we used two types of novel contextual information: 24/7 behavioral recordings and spoken conversations for task modeling. The task context is created by monitoring the user's information behavior from temporal, social, and topical aspects; that can be contextualized by several entities such as applications, documents, people, time, and various keywords determining the task. By tracking the association amongst the entities, we can infer the user's task context, predict future information access, and proactively retrieve relevant information for the task at hand. The approach is validated with a series of field studies, in which altogether 47 participants voluntarily installed a screen monitoring system on their laptops 24/7 to collect available digital activities, and their spoken conversations were recorded. Different aspects of the data were considered to train the models. In the evaluation, we treated information sourced from several applications, spoken conversations, and various aspects of the data as different kinds of influence on the prediction performance. The combined influences of multiple data sources and aspects were also considered in the models. Our findings revealed that task information could be found in a variety of applications and spoken conversations. In addition, we found that task context models that consider behavioral information captured from the computer screen and spoken conversations could yield a promising improvement in recommendation quality compared to the conventional modeling approach that considered only pre-determined interaction logs, such as query logs or Web browsing history. We also showed how a task context model could support the users' work performance, reducing their effort in searching by ranking and suggesting relevant information. Our results and findings have direct implications for information personalization and recommendation systems that leverage contextual information to predict and proactively present personalized information to the user to improve the interaction experience with the computer systems.Jokapäiväisiin tehtäviimme kuuluu vuorovaikutusta monenlaisten tietojen kanssa. Hallitsemamme tiedot liittyvät usein johonkin tehtäväkontekstiin. Nykyiset tietokonejärjestelmät eivät kuitenkaan järjestä tietoja tällä tavalla tai auta käyttäjää löytämään tietoja tehtäväkontekstista, vaan vaativat käyttäjältä eksplisiittisiä toimia, kuten tietojen hakua ja etsimistä. Tutkimme, kuinka tehtäväkontekstia voidaan käyttää ohjaamaan tietojen toimittamista käyttäjälle ennakoivasti, eli siten, että oikeat tiedot olisivat helposti saatavilla oikeaan aikaan. Tässä väitöskirjassa käytimme kahdenlaisia uusia kontekstuaalisia tietoja: 24/7-käyttäytymistallenteita ja tehtävän mallintamiseen liittyviä puhuttuja keskusteluja. Tehtäväkonteksti luodaan seuraamalla käyttäjän tietokäyttäytymistä ajallisista, sosiaalisista ja ajankohtaisista näkökulmista katsoen; sitä voidaan kuvata useilla entiteeteillä, kuten sovelluksilla, asiakirjoilla, henkilöillä, ajalla ja erilaisilla tehtävää määrittävillä avainsanoilla. Tarkastelemalla näiden entiteettien välisiä yhteyksiä voimme päätellä käyttäjän tehtäväkontekstin, ennustaa tulevaa tiedon käyttöä ja hakea ennakoivasti käsillä olevaan tehtävään liittyviä asiaankuuluvia tietoja. Tätä lähestymistapaa arvioitiin kenttätutkimuksilla, joissa yhteensä 47 osallistujaa asensi vapaaehtoisesti kannettaviin tietokoneisiinsa näytönvalvontajärjestelmän, jolla voitiin 24/7 kerätä heidän saatavilla oleva digitaalinen toimintansa, ja joissa tallennettiin myös heidän puhutut keskustelunsa. Mallien kouluttamisessa otettiin huomioon datan eri piirteet. Arvioinnissa käsittelimme useista sovelluksista, puhutuista keskusteluista ja datan eri piirteistä saatuja tietoja erilaisina vaikutuksina ennusteiden toimivuuteen. Malleissa otettiin huomioon myös useiden tietolähteiden ja näkökohtien yhteisvaikutukset. Havaintomme paljastivat, että tehtävätietoja löytyi useista sovelluksista ja puhutuista keskusteluista. Lisäksi havaitsimme, että tehtäväkontekstimallit, joissa otetaan huomioon tietokoneen näytöltä ja puhutuista keskusteluista saadut käyttäytymistiedot, voivat parantaa suositusten laatua verrattuna tavanomaiseen mallinnustapaan, jossa tarkastellaan vain ennalta määritettyjä vuorovaikutuslokeja, kuten kyselylokeja tai verkonselaushistoriaa. Osoitimme myös, miten tehtäväkontekstimalli pystyi tukemaan käyttäjien suoritusta ja vähentämään heidän hakuihin tarvitsemaansa työpanosta järjestämällä hakutuloksia ja ehdottamalla heille asiaankuuluvia tietoja. Tuloksillamme ja havainnoillamme on suoria vaikutuksia tietojen personointi- ja suositusjärjestelmiin, jotka hyödyntävät kontekstuaalista tietoa ennustaakseen ja esittääkseen ennakoivasti personoituja tietoja käyttäjälle ja näin parantaakseen vuorovaikutuskokemusta tietokonejärjestelmien kanssa

Automotive UX design and data-driven development: Narrowing the gap to support practitioners

The development and evaluation of In-Vehicle Information Systems (IVISs) is strongly based on insights from qualitative studies conducted in artificial contexts (e.g., driving simulators or lab experiments). However, the growing complexity of the systems and the uncertainty about the context in which they are used, create a need to augment qualitative data with quantitative data, collected during real-world driving. In contrast to many digital companies that are already successfully using data-driven methods, Original Equipment Manufacturers (OEMs) are not yet succeeding in releasing the potentials such methods offer. We aim to understand what prevents automotive OEMs from applying data-driven methods, what needs practitioners formulate, and how collecting and analyzing usage data from vehicles can enhance UX activities. We adopted a Multiphase Mixed Methods approach comprising two interview studies with more than 15 UX practitioners and two action research studies conducted with two different OEMs. From the four studies, we synthesize the needs of UX designers, extract limitations within the domain that hinder the application of data-driven methods, elaborate on unleveraged potentials, and formulate recommendations to improve the usage of vehicle data. We conclude that, in addition to modernizing the legal, technical, and organizational infrastructure, UX and Data Science must be brought closer together by reducing silo mentality and increasing interdisciplinary collaboration. New tools and methods need to be developed and UX experts must be empowered to make data-based evidence an integral part of the UX design process

Crash/Near-Crash: Impact of Secondary Tasks and Real-Time Detection of Distracted Driving

The main goal of this dissertation is to investigate the problem of distracted driving from two different perspectives. First, the identification of possible sources of distraction and their associated crash/near-crash risk. That can assist government officials toward more informed decision-making process, allowing for optimized allocation of available resources to reduce roadway crashes and improve traffic safety. Second, actively counteracting the distracted driving phenomenon by quantitative evaluation of eye glance patterns. This dissertation research consists of two different parts. The first part provides an in-depth analysis for the increased crash/near-crash risk associated with different secondary task activities using the largest real-world naturalistic driving dataset (SHRP2 Naturalistic Driving Study). Several statistical and data mining techniques are developed to analyze the distracted driving and crash risk. More specifically, two different models were employed to quantify the increased risk associated with each secondary task: a baseline-category logit model, and a rule mining association model. The baseline-category logit model identified the increased risk in terms of odds ratios, while the A-priori association algorithm detected the associated risks in terms of rules. Each rule was then evaluated based on the lift index. The two models succeeded in ranking all the secondary task activities according to the associated increased crash/near-crash risk efficiently. To actively counteract to the distracted driving phenomenon, a new approach was developed to analyze eye glance patterns and quantify distracted driving behavior under safety and non-Safety Critical Events (SCEs). This approach is then applied to the Naturalistic Engagement in Secondary Tasks (NEST) dataset to investigate how drivers allocate their attention while driving, especially while distracted. The analysis revealed that distracted driving behavior can be well characterized using two new distraction risk indicators. Additional statistical analyses showed that the two indicators increase significantly for SCE compared to normal driving events. Consequently, an artificial neural network (ANN) model was developed to test the SCEs predictability power when accounting for the two new indicators. The ANN model was able to predict the SCEs with an overall accuracy of 96.1%. This outcome can help build reliable algorithms for in-vehicle driving assistance systems to alert drivers before SCEs

Social and interactional practices for disseminating current awareness information in an organisational setting.

Current awareness services are designed to keep users informed about recent developments based around user need profiles. In organisational settings, they may operate through both electronic and social interactions aimed at delivering information that is relevant, pertinent and current. Understanding these interactions can reveal the tensions in current awareness dissemination and help inform ways of making services more effective and efficient. We report an in-depth, observational study of electronic current awareness use within a large London law firm. The study found that selection, re-aggregation and forwarding of information by multiple actors gives rise to a complex sociotechnical distribution network. Knowledge management staff act as a layer of “intelligent filters” sensitive to complex, local information needs; their distribution decisions address multiple situational relevance factors in a situation fraught with information overload and restrictive time-pressures. Their decisions aim to optimise conflicting constraints of recall, precision and information quantity. Critical to this is the use of dynamic profile updates which propagate back through the network through formal and informal social interactions. This supports changes to situational relevance judgements and so allows the network to ‘self-tune’. These findings lead to design requirements, including that systems should support rapid assessment of information items against an individual’s interests; that it should be possible to organise information for different subsequent uses; and that there should be back-propagation from information consumers to providers, to tune the understanding of their information needs