Recognizing and understanding user behaviors from screencasts

User interacts with computers or mobile devices, leading to user behaviors on screen. In the context of software engineering, analyzing user behavior enables many applications such as intelligent bug fix, code completion and knowledge recommendation for developers. Such technique can be extended to more general knowledge worker environment, in which users have to manipulate devices according to specific guidelines. Existing works rely heavily on software instrumentation to obtain user actions from operation systems, which is hard to deploy and maintain. In addition, considering the security and privacy of some scenarios, non-intrusive is the major requirement to be included in the system. In this work, we leverage Computer Vision and Natural Language Processing techniques to recognize and understand user behaviors from screencasts, which is a non-intrusive and cross-platform method. We first recognize 10 categories of low level user actions such as mouse moving and type text, then summarize them to higher level abstractions (i.e. line-granularity coding steps). We also try to interpret user interaction with applications by multi-task learning and generate structured language descriptions (i.e. command, widget and location). Finally, unsupervised learning method is introduced for GUI linting problem, which is taken as a case study of user behavior analysis. To train the deep neural networks, we collect diverse video data from YouTube, Twitch and Bugzilla, and manually label them to build the dataset. The experiment results demonstrate the high performance of proposed method, and the user study validate the practical applications of many downstream tasks

Investigating the potential for new media and new technologies in design and technology undergraduate education

Investigating Potential for New Media & New Technologies in Design & Technology Undergraduate Education This research explores potential for New Media and New Technology (NM & NT) in the Design School at Loughborough University. Using action research to investigate potential, this research develops a new way of managing inquiry based on Susman and Evered s five cycles of action research (Susman and Evered, 1978). In particular, it extends the double- helix metaphor (Dick, 2000) for action research. This new way of conducting action research looks at educational and IT- based aspects; in particular, developing strategies, guidelines and materials for implementing video podcasting (Vodcasting) and Really Simple Syndication (RSS) into Design School undergraduate modules. In looking at potential, the research involved 6 lecturer s interviews and thematic analysis. Findings suggest that limitations to the current uses of NM & NT related to lecturers lack of skills in NM & NT and scepticism about what the benefits might be. Some recognised potential for NM & NT to manage module administration. One lecturer wanted to stop students using dubious sources from the Internet for assessment on a Sustainable Design module. This led to using RSS to resolve this problem in a mobile learning scenario. In this research, 98 D and T students were surveyed to identify current uses of mobile technology. Results suggested that students would like module content streamed to their mobile device. Lecturers too could see benefits for NM & NT, if they stopped lecturers from having to repeat themselves to students. This led to using Vodcasting to resolve this problem in a mobile learning scenario. Video observational data was collected from 6 students using RSS to perform mobile learning tasks for a Sustainable Design module. The findings suggested that the technology at the time of study was not quite up to the task, although some NM & NT learning resources relating to Sustainable Design were found by students using RSS. Similarly, video observation data was collected from 4 students using Vodcasts to design electronic circuits. Findings showed more technological competence with this technology and students suggested future modules where this type of NM & NT would have further educational potential. Through exploring potential, this research develops new strategies, guidelines and materials for design and technology educators. This research reveals the educational benefits of Vodcasting and RSS in labs and workshops, and concludes that there is potential for NM & NT in D and T education

Systems and Methods for Measuring and Improving End-User Application Performance on Mobile Devices

In today's rapidly growing smartphone society, the time users are spending on their smartphones is continuing to grow and mobile applications are becoming the primary medium for providing services and content to users. With such fast paced growth in smart-phone usage, cellular carriers and internet service providers continuously upgrade their infrastructure to the latest technologies and expand their capacities to improve the performance and reliability of their network and to satisfy exploding user demand for mobile data. On the other side of the spectrum, content providers and e-commerce companies adopt the latest protocols and techniques to provide smooth and feature-rich user experiences on their applications. To ensure a good quality of experience, monitoring how applications perform on users' devices is necessary. Often, network and content providers lack such visibility into the end-user application performance. In this dissertation, we demonstrate that having visibility into the end-user perceived performance, through system design for efficient and coordinated active and passive measurements of end-user application and network performance, is crucial for detecting, diagnosing, and addressing performance problems on mobile devices. My dissertation consists of three projects to support this statement. First, to provide such continuous monitoring on smartphones with constrained resources that operate in such a highly dynamic mobile environment, we devise efficient, adaptive, and coordinated systems, as a platform, for active and passive measurements of end-user performance. Second, using this platform and other passive data collection techniques, we conduct an in-depth user trial of mobile multipath to understand how Multipath TCP (MPTCP) performs in practice. Our measurement study reveals several limitations of MPTCP. Based on the insights gained from our measurement study, we propose two different schemes to address the identified limitations of MPTCP. Last, we show how to provide visibility into the end- user application performance for internet providers and in particular home WiFi routers by passively monitoring users' traffic and utilizing per-app models mapping various network quality of service (QoS) metrics to the application performance.PHDComputer Science & EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/146014/1/ashnik_1.pd

Understanding and Evaluating User Interface Visibility

Technology dominates our lives, mobile technology in particular. In 2016 Apple sold their billionth iPhone. By 2018 they had sold their 2 billionth device based on the same underlying operating system. We access such technology through the user interface (UI) and concerns have been raised about the usability of such devices. The situation has been described by some as a “usability crisis”. One of the key issues raised is the lack of visibility of user interface elements, which is deemed to be a critical component of an effective UI. An initial investigation highlighted that UI visibility can be broken down into three key aspects: Firstly; some user interface elements are effectively ‘missing’; Secondly, they are ‘missed’ because they are not seen by the user; and thirdly, they are seen but ‘misunderstood’. Further analysis of the home screen of an iPhone revealed that only 8% of the available functions were visible at the top level, in other words, 92% were effectively ‘missing’. This raises key questions about how UI visibility can be evaluated, and such evaluation adopted into design practice. This research took a psychophysical perspective to better understand UI visibility. This led to the development of an evaluation framework and associated tool called vis-UI-lise. The tool represents UI visibility as a series of 5 hurdles between the user and the interface that have to be overcome for a successful interaction. This tool was applied to an everyday task on a mobile phone which resulted in highlighting a range of possible usability problems. Comparison of the predicted versus observed problems showed that the vis-UI-lise tool had predicted 74% of them, a score that compares well with other usability evaluation tools. A training and support package was also developed for the vis-UI-lise tool and evaluated with four different organisations. This provided key insights into how the tool could be improved to fit in with typical design practice. This thesis brings a new perspective to the understanding and evaluation of UI visibility that could have a real impact on the design of everyday user interfaces

A sound idea: An investigation into accessible video game design for the deaf and hard of hearing

A widely accepted, and incorrect, assumption towards hearing accessibility in video games is that deaf and hard of hearing (DHH) users are those who encounter the least barriers and are generally well catered for. Rapid advancement in video game technology has seen video game sound evolve from simple blips generated by internal circuitry to fully realised digital audio used to convey critical information. To accommodate the DHH, this information needs to be conveyed in an alternative manner. However, evidence suggests existing accessible design solutions for the DHH lack specificity and are insufficient. Thus, the inability to hear, or hear well, has historically resulted in DHH users left with impeded experience and gameplay. This thesis describes an investigation to address the primary research question: How might accessible video game design practices be facilitated to better accommodate the deaf and hard of hearing? To examine this question, an action research method as part of a transformative mixed methods methodology was used. Data collection procedures included critical analysis of literature, observations, and a cross-sectional self-administered survey for triangulation. The critical analysis of literature exposed issues relating to accessible video game design, particularly in relation to the identification of solutions and technical implementation. Further, issues related to the classification of video game software were identified. This posed potential problems with identification of game design methods and led to the development of a new video game classification model. The new model informed an analysis on the methods used for the design of video games, and results were visually represented and mapped to the different approaches to accessible design. Subsequent analysis determined that a game assessment framework is a suitable approach to facilitating accessible design. Further investigation identified visual feedback as the most suitable form of complementary feedback to game audio. This led to the development of a new model to classify visual feedback elements used in video games, and identification of audio feedback categories based on diegetic film theory. Through triangulation of results, a new game feedback model (GFM) was developed. The GFM was used for observational experimentation to identify and classify individual visual feedback elements used in video games. Each element was analysed and mapped to categories of game sound. The resulting model, with populated data, was used to determine what visual feedback elements may be used to complement specific categories of critical game audio. A survey was subsequently used for triangulation, and resulted in amendments to the final model. Through iterative development, and interpretation of findings, the research culminated in the development of a game assessment framework. The three-step framework aids in the classification of game sounds; assesses the impact of those game sounds; and provides recommendations for complementary visual feedback elements for sounds identified as having an adverse impact on user experience and gameplay if they were to be removed. The framework is innovative and has the potential to provide practical guidance for developers of video games. In addition, this research provides the foundation for future research, with the potential to influence accessible game design for the DHH