Sensing and visualizing spatial relations of mobile devices

Location information can be used to enhance interaction with mobile devices. While many location systems require instrumentation of the environment, we present a system that allows devices to measure their spatial relations in a true peer-to-peer fashion. The system is based on custom sensor hardware implemented as USB dongle, and computes spatial relations in real-time. In extension of this system we propose a set of spatialized widgets for incorporation of spatial relations in the user interface. The use of these widgets is illustrated in a number of applications, showing how spatial relations can be employed to support and streamline interaction with mobile devices

BlogForever D2.6: Data Extraction Methodology

This report outlines an inquiry into the area of web data extraction, conducted within the context of blog preservation. The report reviews theoretical advances and practical developments for implementing data extraction. The inquiry is extended through an experiment that demonstrates the effectiveness and feasibility of implementing some of the suggested approaches. More specifically, the report discusses an approach based on unsupervised machine learning that employs the RSS feeds and HTML representations of blogs. It outlines the possibilities of extracting semantics available in blogs and demonstrates the benefits of exploiting available standards such as microformats and microdata. The report proceeds to propose a methodology for extracting and processing blog data to further inform the design and development of the BlogForever platform

Variability of User Interaction with Multi-Platform News Feeds

The development of the World Wide Web (WWW) and proliferation of web enabled devices have allowed various news agencies to enrich their traditional method of distribution of news through TV, radio and print with simultaneous broadcast through the Web. The varying nature of devices through which the Web is accessed warrants different ways to feed the same content. This precipitates some variation in the way users interact with the news feeds. In this paper, we investigate how mental models and information scent affect this variation and user interaction on the whole. We present results from a preliminary survey conducted to capture the current news gathering behavior of general population and verify our assumptions. We then present observations from the study conducted using BBC news site over laptop, PDA and a cell phone

Learning the language of apps

To explore the functionality of an app, automated test generators systematically identify and interact with its user interface (UI) elements. A key challenge is to synthesize inputs which effectively and efficiently cover app behavior. To do so, a test generator has to choose which elements to interact with but, which interactions to do on each element and which input values to type. In summary, to better test apps, a test generator should know the app's language, that is, the language of its graphical interactions and the language of its textual inputs. In this work, we show how a test generator can learn the language of apps and how this knowledge is modeled to create tests. We demonstrate how to learn the language of the graphical input prior to testing by combining machine learning and static analysis, and how to refine this knowledge during testing using reinforcement learning. In our experiments, statically learned models resulted in 50\% less ineffective actions an average increase in test (code) coverage of 19%, while refining these through reinforcement learning resulted in an additional test (code) coverage of up to 20%. We learn the language of textual inputs, by identifying the semantics of input fields in the UI and querying the web for real-world values. In our experiments, real-world values increase test (code) coverage ~10%; Finally, we show how to use context-free grammars to integrate both languages into a single representation (UI grammar), giving back control to the user. This representation can then be: mined from existing tests, associated to the app source code, and used to produce new tests. 82% test cases produced by fuzzing our UI grammar can reach a UI element within the app and 70% of them can reach a specific code location.Automatisierte Testgeneratoren identifizieren systematisch Elemente der Benutzeroberfläche und interagieren mit ihnen, um die Funktionalität einer App zu erkunden. Eine wichtige Herausforderung besteht darin, Eingaben zu synthetisieren, die das App-Verhalten effektiv und effizient abdecken. Dazu muss ein Testgenerator auswählen, mit welchen Elementen interagiert werden soll, welche Interaktionen jedoch für jedes Element ausgeführt werden sollen und welche Eingabewerte eingegeben werden sollen. Um Apps besser testen zu können, sollte ein Testgenerator die Sprache der App kennen, dh die Sprache ihrer grafischen Interaktionen und die Sprache ihrer Texteingaben. In dieser Arbeit zeigen wir, wie ein Testgenerator die Sprache von Apps lernen kann und wie dieses Wissen modelliert wird, um Tests zu erstellen. Wir zeigen, wie die Sprache der grafischen Eingabe lernen vor dem Testen durch maschinelles Lernen und statische Analyse kombiniert und wie dieses Wissen weiter verfeinern beim Testen Verstärkung Lernen verwenden. In unseren Experimenten führten statisch erlernte Modelle zu 50% weniger ineffektiven Aktionen, was einer durchschnittlichen Erhöhung der Testabdeckung (Code) von 19% entspricht, während die Verfeinerung dieser durch verstärkendes Lernen zu einer zusätzlichen Testabdeckung (Code) von bis zu 20% führte. Wir lernen die Sprache der Texteingaben, indem wir die Semantik der Eingabefelder in der Benutzeroberfläche identifizieren und das Web nach realen Werten abfragen. In unseren Experimenten erhöhen reale Werte die Testabdeckung (Code) um ca. 10%; Schließlich zeigen wir, wie kontextfreien Grammatiken verwenden beide Sprachen in einer einzigen Darstellung (UI Grammatik) zu integrieren, wieder die Kontrolle an den Benutzer zu geben. Diese Darstellung kann dann: aus vorhandenen Tests gewonnen, dem App-Quellcode zugeordnet und zur Erstellung neuer Tests verwendet werden. 82% Testfälle, die durch Fuzzing unserer UI-Grammatik erstellt wurden, können ein UI-Element in der App erreichen, und 70% von ihnen können einen bestimmten Code-Speicherort erreichen

Using graphical representation of user interfaces as visual references

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2012.Cataloged from PDF version of thesis.Includes bibliographical references (p. 129-133).My thesis investigates using a graphical representation of user interfaces - screenshots - as a direct visual reference to support various kinds of applications. We have built several systems to demonstrate and validate this idea in domains like searching documentation, GUI automation and testing, and cross-device information migration. In particular, Sikuli Search enables users to search documentation using screenshots of GUI elements instead of keywords. Sikuli Script enables users to programmatically control GUIs without support from the underlying applications. Sikuli Test lets GUI developers and testers create test scripts without coding. Deep Shot introduces a framework and interaction techniques to migrate work states across heterogeneous devices in one action, taking a picture. We also discuss challenges inherent in screenshot-based interactions and propose potential solutions and directions of future research.by Tsung-Hsiang Chang.Ph.D

Design exploration: engaging a larger user population

Software designers must understand the domain, work practices, and user expectations before determining requirements or generating initial design mock-ups. Users and other stakeholders are a valuable source of information leading to that understanding. Much work has focused on design approaches that include users in the software development process. These approaches vary from surveys and questionnaires that garner responses from a population of potential users to participatory design processes where representative users are included in the design/development team. The Design Exploration approach retains the remote and asynchronous communication of surveys while making expression of feedback easier by providing users alternatives to textual communication for their suggestions and tacit understanding of the domain. To do this, visual and textual modes of expression are combined to facilitate communication from users to designers while allowing a broad user audience to contribute to software design. One challenge to such an approach is how software designers make use of the potentially overwhelming combination of text, graphics, and other content. The Design Exploration process provides users and other stakeholders the Design Exploration Builder, a construction kit where they create annotated partial designs. The Design Exploration Analyzer is an exploration tool that allows software designers to consolidate and explore partial designs. The Analyzer looks for patterns based on textual analysis of annotations and spatial analysis of graphical designs, to help identify interesting examples and patterns within the collection. Then software designers can use this tool to search and browse within the exploration set in order to better understand the task domain, user expectations and work practices. Evaluation of the tools has shown that users will often work to overcome expression constraints to convey information. Moreover, the mode of expression influences the types of information garnered. Furthermore, including more users results in greater coverage of the information gathered. These results provide evidence that Design Exploration is an approach that collects software and domain information from a large group of users that lies somewhere between questionnaires and face to face methods

Engaging end-user driven recommender systems: personalization through web augmentation

In the past decades recommender systems have become a powerful tool to improve personalization on the Web. Yet, many popular websites lack such functionality, its implementation usually requires certain technical skills, and, above all, its introduction is beyond the scope and control of end-users. To alleviate these problems, this paper presents a novel tool to empower end-users without programming skills, without any involvement of website providers, to embed personalized recommendations of items into arbitrary websites on client-side. For this we have developed a generic meta-model to capture recommender system configuration parameters in general as well as in a web augmentation context. Thereupon, we have implemented a wizard in the form of an easy-to-use browser plug-in, allowing the generation of so-called user scripts, which are executed in the browser to engage collaborative filtering functionality from a provided external rest service. We discuss functionality and limitations of the approach, and in a study with end-users we assess the usability and show its suitability for combining recommender systems with web augmentation techniques, aiming to empower end-users to implement controllable recommender applications for a more personalized browsing experience.Fil: Wischenbart, Martin. Johannes Kepler University Linz; AustriaFil: Firmenich, Sergio Damian. Universidad Nacional de La Plata. Facultad de Informática. Laboratorio de Investigación y Formación en Informática Avanzada; Argentina. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - La Plata; ArgentinaFil: Rossi, Gustavo Héctor. Universidad Nacional de La Plata. Facultad de Informática; Argentina. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - La Plata; ArgentinaFil: Bosetti, Gabriela Alejandra. Universidad Nacional de La Plata. Facultad de Informática; Argentina. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - La Plata; ArgentinaFil: Kapsammer, Elisabeth. Johannes Kepler University Linz; Austri

Stripes — A Conceptual Operating System User Interface

Perinteiset käyttöjärjestelmien graafiset käyttöliittymät suunniteltiin vuosikym- meniä sitten ja ovat muodostuneet osaksi jokapäiväistä elämäämme. Viime vuosien tietoteknillinen kehitys ja tietokoneiden muuttuneet käyttötarpeet ovat tuoneet esille nykyisten käyttöliittymien heikkoudet. Tämä lopputyö esittää vaihtoehtoisen käyttöliit- tymätoteutuksen, nimeltään Juovat, joka painottaa käytettävyyttä, yksinkertaisuutta, tuotteliaisuutta ja muita käyttäjäkeskeisiä arvoja. Työ kattaa ja analysoi nykyisten käyttöliittymien historian sekä viime vuosien tärkeimmät innovaatiot. Alan tutkiminen helpottaa nykyisten käyttöliittymien heik- kouksien ja vahvuuksien kartoittamisessa sekä auttaa muodostamaan selkeät kriteerit suunnitellulle käyttöliittymätoteutukselle. Juovat-käyttöliittymän kokonaisvaltainen suunnittelu ja useat yksityiskohdat esitel- lään seikkaperäisesti. Toimivuutta analysoidaan vertaamalla toteutuksen vahvuuksia ja heikkouksia olemassa oleviin käyttöliittymiin sekä testaamalla käytännön toimivuutta prototyypin avulla. Lopuksi testituloksia peilataan alkuperäisiin suunnittelukritee- reihin ja pohditaan työn onnistumista.Traditional computer operating system graphical user interfaces were designed decades ago have since become an integral part of our everyday lives. While fundamentally excellent, recent changes in the industry and use cases for computers are exposing the weaknesses in the implementations of current operating system user interfaces. This thesis is an attempt to propose an alternative, modernized user interface design — entitled Stripes — based on usability, simplicity, productivity, and other design guidelines. The history and recent innovative additions to the current implementations of WIMP- based operating system user interfaces are outlined and analyzed. This contextual mapping of the field helps in identifying key problems and strengths of modern computer user interfaces, and, in addition to the aforementioned guidelines, form a basis for the design of the proposed concept. The fundamental design choices, as well as several details of Stripes are described and explained. Its theoretical improvements and disadvantages over current user interface implementations are first discussed, then tested with the help of an interactive proto- type. The favorable and critical test outcomes are described in detail, and reflected against the original design guidelines of the concept. Finally, conclusions on the success of the proposed interface are provided