A Natural Language Query Interface for Searching Personal Information on Smartwatches

Currently, personal assistant systems, run on smartphones and use natural language interfaces. However, these systems rely mostly on the web for finding information. Mobile and wearable devices can collect an enormous amount of contextual personal data such as sleep and physical activities. These information objects and their applications are known as quantified-self, mobile health or personal informatics, and they can be used to provide a deeper insight into our behavior. To our knowledge, existing personal assistant systems do not support all types of quantified-self queries. In response to this, we have undertaken a user study to analyze a set of “textual questions/queries” that users have used to search their quantified-self or mobile health data. Through analyzing these questions, we have constructed a light-weight natural language based query interface - including a text parser algorithm and a user interface - to process the users’ queries that have been used for searching quantified-self information. This query interface has been designed to operate on small devices, i.e. smartwatches, as well as augmenting the personal assistant systems by allowing them to process end users’ natural language queries about their quantified-self data

Elckerlyc goes mobile: enabling technology for ECAs in mobile applications

The fast growth of computational resources and speech technology available on mobile devices makes it pos- sible for users of these devices to interact with service sys- tems through natural dialogue. These systems are sometimes perceived as social agents and presented by means of an animated embodied conversational agent (ECA). To take the full advantage of the power of ECAs in service systems, it is important to support real-time, online and responsive interaction with the system through the ECA. The design of responsive animated conversational agents is a daunting task. Elckerlyc is a model-based platform for the specification and animation of synchronised multimodal responsive animated agents. This paper presents a new light-weight PictureEngine that allows this platform to embed an ECA in the user interface of mobile applications. The ECA can be specified by using the behavior markup language (BML). An application and user evaluations of Elckerlyc and the PictureEngine in a mobile embedded digital coach is presented

Rapid troubleshooting and management of a network using a chatbot

Managing a large and complex network is currently tedious and error-prone: administrators log in via lengthy authentication processes; look up and access a diversity of network devices, services, tools and other entities with differing communication protocols; issue relatively obscure, device-specific commands; etc. These problems of network management become especially manifest while troubleshooting critical issues under time constraints. This disclosure describes a fast, simple, and unified approach to network management via a chat interface. At the back end of the chat interface is a bot that is in communication with a variety of network entities in their native protocols. The bot receives aliased, nearly natural-language commands from the administrator via the chat interface, acts on such commands by communicating with relevant network entities, and returns responses via the chat interface. Per the disclosed techniques, the network can be managed over simple, widely available interfaces such as chat over mobile device, and without lengthy or complicated procedures

Overcoming Language Dichotomies: Toward Effective Program Comprehension for Mobile App Development

Mobile devices and platforms have become an established target for modern software developers due to performant hardware and a large and growing user base numbering in the billions. Despite their popularity, the software development process for mobile apps comes with a set of unique, domain-specific challenges rooted in program comprehension. Many of these challenges stem from developer difficulties in reasoning about different representations of a program, a phenomenon we define as a "language dichotomy". In this paper, we reflect upon the various language dichotomies that contribute to open problems in program comprehension and development for mobile apps. Furthermore, to help guide the research community towards effective solutions for these problems, we provide a roadmap of directions for future work.Comment: Invited Keynote Paper for the 26th IEEE/ACM International Conference on Program Comprehension (ICPC'18

Challenges in Developing Applications for Aging Populations

Elderly individuals can greatly benefit from the use of computer applications, which can assist in monitoring health conditions, staying in contact with friends and family, and even learning new things. However, developing accessible applications for an elderly user can be a daunting task for developers. Since the advent of the personal computer, the benefits and challenges of developing applications for older adults have been a hot topic of discussion. In this chapter, the authors discuss the various challenges developers who wish to create applications for the elderly computer user face, including age-related impairments, generational differences in computer use, and the hardware constraints mobile devices pose for application developers. Although these challenges are concerning, each can be overcome after being properly identified

Augmenting conversations through context-aware multimedia retrieval based on speech recognition

Future’s environments will be sensitive and responsive to the presence of people to support them carrying out their everyday life activities, tasks and rituals, in an easy and natural way. Such interactive spaces will use the information and communication technologies to bring the computation into the physical world, in order to enhance ordinary activities of their users. This paper describes a speech-based spoken multimedia retrieval system that can be used to present relevant video-podcast (vodcast) footage, in response to spontaneous speech and conversations during daily life activities. The proposed system allows users to search the spoken content of multimedia files rather than their associated meta-information and let them navigate to the right portion where queried words are spoken by facilitating within-medium searches of multimedia content through a bag-of-words approach. Finally, we have studied the proposed system on different scenarios by using vodcasts in English from various categories, as the targeted multimedia, and discussed how it would enhance people’s everyday life activities by different scenarios including education, entertainment, marketing, news and workplace