Probes and Sensors: The Design of Feedback Loops for Usability Improvements

The importance of user-centric design methods in the design of programming tools is now well accepted. These methods depend on creating a feedback loop between the designers and their users, providing data about developers, their needs and behaviour gathered through various means. These include controlled experiments, field observations, as well as analytical frameworks. However, whilst there have been a number of experiments detailed, quantitative data is rarely used as part of the design process. Part of the reason for this might be that such feedback loops are hard to design and use in practice. Still, we believe there is potential in this approach and opportunities in gathering this kind of ‘big data’. In this paper, we sketch a framework for reasoning about these feedback loops - when data gathering may make sense and for how to incorporate the results of such data gathering into the programming tool design process. We illustrate how to use the framework on two case studies and outline some of the challenges in instrumentation and in knowing when and how to act on signals

Noise Challenges in Monomodal Gaze Interaction

Modern graphical user interfaces (GUIs) are designed with able-bodied users in mind. Operating these interfaces can be impossible for some users who are unable to control the conventional mouse and keyboard. An eye tracking system offers possibilities for independent use and improved quality of life via dedicated interface tools especially tailored to the users ’ needs (e.g., interaction, communication, e-mailing, web browsing and entertainment). Much effort has been put towards robustness, accuracy and precision of modern eyetracking systems and there are many available on the market. Even though gaze tracking technologies have undergone dramatic improvements over the past years, the systems are still very imprecise. This thesis deals with current challenges of mono-modal gaze interaction and aims at improving access to technology and interface control for users who are limited to the eyes only. Low-cost equipment in eye tracking contributes toward improved affordability but potentially at the cost of introducing more noise in the system due to the lower quality of hardware. This implies that methods of dealing with noise and creative approaches towards getting the best out of the data stream are most wanted. The work in this thesis presents three contributions that may advance the use of low-cost mono-modal gaze tracking and research in the field:- An assessment of a low-cost open-source gaze tracker and two eye tracking systems through an accuracy and precision test and a performance evaluation.- Development and evaluation of a novel innovative 3D typing system with high tolerance to noise that is based on continuous panning and zooming.- Development and evaluation of novel selection tools that compensate for noisy input during small-target selections in modern GUIs. This thesis may be of particular interest for those working on the use of eye trackers for gaze interaction and how to deal with reduced data quality. The work in this thesis is accompanied by several software applications developed for the research projects that can be freely downloaded from the eyeInteract appstore 1

Addressing Situational and Physical Impairments and Disabilities with a Gaze-Assisted, Multi-Modal, Accessible Interaction Paradigm

Every day we encounter a variety of scenarios that lead to situationally induced impairments and disabilities, i.e., our hands are assumed to be engaged in a task, and hence unavailable for interacting with a computing device. For example, a surgeon performing an operation, a worker in a factory with greasy hands or wearing thick gloves, a person driving a car, and so on all represent scenarios of situational impairments and disabilities. In such cases, performing point-and-click interactions, text entry, or authentication on a computer using conventional input methods like the mouse, keyboard, and touch is either inefficient or not possible. Unfortunately, individuals with physical impairments and disabilities, by birth or due to an injury, are forced to deal with these limitations every single day. Generally, these individuals experience difficulty or are completely unable to perform basic operations on a computer. Therefore, to address situational and physical impairments and disabilities it is crucial to develop hands-free, accessible interactions. In this research, we try to address the limitations, inabilities, and challenges arising from situational and physical impairments and disabilities by developing a gaze-assisted, multi-modal, hands-free, accessible interaction paradigm. Specifically, we focus on the three primary interactions: 1) point-and-click, 2) text entry, and 3) authentication. We present multiple ways in which the gaze input can be modeled and combined with other input modalities to enable efficient and accessible interactions. In this regard, we have developed a gaze and foot-based interaction framework to achieve accurate “point-and-click" interactions and to perform dwell-free text entry on computers. In addition, we have developed a gaze gesture-based framework for user authentication and to interact with a wide range of computer applications using a common repository of gaze gestures. The interaction methods and devices we have developed are a) evaluated using the standard HCI procedures like the Fitts’ Law, text entry metrics, authentication accuracy and video analysis attacks, b) compared against the speed, accuracy, and usability of other gaze-assisted interaction methods, and c) qualitatively analyzed by conducting user interviews. From the evaluations, we found that our solutions achieve higher efficiency than the existing systems and also address the usability issues. To discuss each of these solutions, first, the gaze and foot-based system we developed supports point-and-click interactions to address the “Midas Touch" issue. The system performs at least as good (time and precision) as the mouse, while enabling hands-free interactions. We have also investigated the feasibility, advantages, and challenges of using gaze and foot-based point-and-click interactions on standard (up to 24") and large displays (up to 84") through Fitts’ Law evaluations. Additionally, we have compared the performance of the gaze input to the other standard inputs like the mouse and touch. Second, to support text entry, we developed a gaze and foot-based dwell-free typing system, and investigated foot-based activation methods like foot-press and foot gestures. We have demonstrated that our dwell-free typing methods are efficient and highly preferred over conventional dwell-based gaze typing methods. Using our gaze typing system the users type up to 14.98 Words Per Minute (WPM) as opposed to 11.65 WPM with dwell-based typing. Importantly, our system addresses the critical usability issues associated with gaze typing in general. Third, we addressed the lack of an accessible and shoulder-surfing resistant authentication method by developing a gaze gesture recognition framework, and presenting two authentication strategies that use gaze gestures. Our authentication methods use static and dynamic transitions of the objects on the screen, and they authenticate users with an accuracy of 99% (static) and 97.5% (dynamic). Furthermore, unlike other systems, our dynamic authentication method is not susceptible to single video iterative attacks, and has a lower success rate with dual video iterative attacks. Lastly, we demonstrated how our gaze gesture recognition framework can be extended to allow users to design gaze gestures of their choice and associate them to appropriate commands like minimize, maximize, scroll, etc., on the computer. We presented a template matching algorithm which achieved an accuracy of 93%, and a geometric feature-based decision tree algorithm which achieved an accuracy of 90.2% in recognizing the gaze gestures. In summary, our research demonstrates how situational and physical impairments and disabilities can be addressed with a gaze-assisted, multi-modal, accessible interaction paradigm

DAS Writeback: A Collaborative Annotation System for Proteins

We designed and developed a Collaborative Annotation System for Proteins called DAS Writeback, which extends the Distributed Annotation System (DAS) to provide the functionalities of adding, editing and deleting annotations. A great deal of effort has gone into gathering information about proteins over the last few years. By June 2009, UniProtKB/Swiss-Prot, a curated database, contained over four hundred thousand sequence entries and UniProtKB/TrEMBL, a database with automated annotation, contained over eight million sequence entries. Every protein is annotated with relevant information, which needs to be eciently captured and made available to other research groups. These include annotations about the structure, the function or the biochemical residues. Several research groups have taken on the task of making this information accessible to the community, however, information flow in the opposite direction has not been extensively explored. Users are currently passive actors that behave as consumers of one or several sources of protein annotations and they have no immediate way to provide feedback to the source if, for example, a mistake is detected or they want to add information. Any change has to be done by the owner of the database. The current lack of being able to feed information back to a database is tackled in this project. The solution consists of an extension of the DAS protocol that defines the communication rules between the client and the writeback server following the Uniform Interface of the RESTful architecture. A protocol extension was proposed to the DAS community and implementations of both server and client were created in order to have a fully functional system. For the development of the server, writing functionalities were added to MyDAS, which is a widely used DAS server. The writeback client is an extended version of the web-based protein client Dasty2. The involvement of the DAS community and other potential users was a fundamental component of this project. The architecture was designed with the insight of the DAS specialized forum, a prototype was then created and subsequently presented in the DAS workshop 2009. The feedback from the forum and workshop was used to redefine the architecture and implement the system. A usability experiment was performed using potential users of the system emulating a real annotation task. It demonstrated that DAS writeback is effective, usable and will provide the appropriate environment for the creation and evolution of a protein annotation community. Although the scope of this research is limited to protein annotations, the specification was defined in a general way. It can, therefore, be used for other types of information supported by DAS, implying that the server is versatile enough to be used in other scenarios without major modifications

Courtesy markers in requests: The case of pray and please in Late Modern English

This PhD dissertation focuses on the study of the two main courtesy markers in requests in the Late Modern English period, namely please and pray. Both of them are borrowings from French and came to replace native strategies (e.g. the Old English parenthetical ic bidde) in this pragmatic function. Pray had been the major courtesy marker in requests since the Early Modern English period, but it started to fall into disuse during the Late Modern English period, when a new form, please, started to gain ground. A preliminary analysis of the pragmatic markers please and pray in the multigenre corpus ARCHER (A Representative Corpus of Historical English Registers) showed that these features are only available in fiction, drama and letters. Following these results, I proceeded to the analysis of several single-genre corpora. As regards fiction, I resorted to a selection from Chadwyck Healey’s Eighteenth-Century Fiction (1700-1780) and Nineteenth-Century Fiction (1782-1903). For drama I used the drama section in A Corpus of Irish English. Finally, I paid attention to correspondence, and studied two epistolary corpora covering different periods within Late Modern English (the Corpus of Late Eighteenth-Century Prose (1761-1790) and A Corpus of Late Modern English Prose (1860-1919)) and a selection of letter-writing manuals extracted from ECCO (Eighteenth Century Collections Online) database. My study relies on corpus linguistics methodology, and gets insights from Historical Pragmatics, Politeness Theory and Speech-Act Theory, while the origin and development of the courtesy marker please is accounted for in terms of grammaticalisation. The thesis includes a revision of the literature on the different theoretical approaches and provides the accounts and descriptions of these two courtesy markers in the literature, both for Present-day English and for earlier periods. I also looked at Late Modern English reference works in order to gain insight as to how the speech act of requests was apprehended in the period. In my corpus analysis I explore the different sources which have been proposed in the literature as the origin of the courtesy marker please. In addition to conditional structures of the type if you please, in my study I draw special attention to imperative structures such as be pleased to, and please to, which constitute in my opinion the major source of the Present-day courtesy marker please. The process of grammaticalisation of please from these imperative structures would be as follows: Be pleased to > please to > please (verb) > please (courtesy marker). Thus, the courtesy marker please would have originated in a full matrix clause rather than in an already parenthetical conditional form. The grammaticalisation of please follows similar patterns to those identified in the development of other pragmatic markers, not only in English, but also cross-linguistically

Modelling the Role of Pragmatic Plasticity in the Evolution of Linguistic Communication

For a long time, human language has been assumed to be genetically determined and therefore the product of biological evolution. It is only within the last decade that researchers have begun to investigate more closely the domaingeneral cognitive mechanisms of cultural evolution as an alternative explanation for the origins of language. Most of this more recent work focuses on the role of imperfect cultural transmission and abstracts away from the mechanisms of communication. Specifically, models developed to study the cultural evolution of language—both theoretical and computational—often tacitly assume that linguistic signals fully specify the meaning they communicate. They imply that ignoring the fact that this is not the case in actual language use is a justified idealisation which can be made without significant consequences. In this thesis, I show that by making this idealisation, we miss out on the extensive explanatory potential of an empirically attested property of language: its pragmatic plasticity. The meaning that a signal comes to communicate in a specific context usually differs to a certain degree from its conventional meaning. This thesis (i) introduces a model of the cultural evolution of language that acknowledges and incorporates the fact that communication exhibits pragmatic plasticity and (ii) explores the explanatory potential of this fact with regard to language evolution. The thesis falls into two parts. In the first part, I develop the model conceptually. I begin by analysing the components of extant models of general cultural evolution and discuss how models of language change and linguistic evolution map onto them. Innovative use is identified as the motor of cultural evolution. I then conceptualise the cognitive mechanisms underlying innovative language use and argue that they originate in pre-linguistic forms of ostensive-inferential communication. In a next step, the identified mechanisms are employed to provide a unified account of the two main explananda of evolutionary linguistics, the emergence of symbolism and the emergence of grammar. Finally, I discuss the implications of the presented analysis for the so-called proto-language debate. In the second part of the thesis, I propose a computational implementation of the developed conceptual model. This computational implementation allows for the simulation of the cultural emergence and evolution of symbolic communication and provides a laboratory-like environment to study individual aspects of this process. I employ such computer simulations to explore the role that pragmatic plasticity plays in the development of the expressivity, signal economy and ambiguity of emerging and evolving symbolic communication systems. As its main contribution to the study of language evolution, this thesis shows that a model of linguistic cultural evolution that incorporates the notion of pragmatic plasticity has the potential to explain two crucial evolutionary puzzles, namely (i) how language can emerge from no language, and (ii) how language can come to exhibit the appearance of design for communication. The proposed usage-based model of language evolution bridges the evolutionary gap between no language and language by identifying ostensive-inferential communication as the continual aspect present in both stages, and by demonstrating that the cognitive mechanisms involved in ostensive-inferential communication are sufficient for the transition from one stage to the other

Language In My Mouth: Linguistic Variation in the Nmbo Speech Community of Southern New Guinea

This thesis is a mixed-methods investigation into the question of the sociolinguistics of linguistic diversity in Papua New Guinea. Social and cultural traits of New Guinean speech communities have been hypothesised as conducive to language differentiation and diversification (Laycock 1991, Thurston 1987, 1992, Foley 2000, Ross 2001), however there have been few empirical studies to support these hypotheses. In this thesis I investigate linguistic micro-variations within a contemporary New Guinean speech community, with the goal of identifying socio-cultural pressures that affect language variation and change. The community under investigation is the Nmbo speech community located in the Morehead area of Southern New Guinea. It is a highly multilingual community in the middle of the Nambu branch dialect chain, and consists primarily of the three villages Govav, Bevdvn, and Arovwe. The ideologically licensed speakers of Nmbo are the Kerake tribe people, but due to the practice of marriage exogamy, a large portion of non-Kerake people speak Nmbo as an additional language learnt from their parents or spouse. This thesis embraces the complexities of the multilingual ecology by including data from Kerake women who have married out of the Nmbo villages into the neighbouring Nen language village of Bimadbn. The empirical investigations bring data from three directions. First are the qualitative descriptions based on my own ethnographic fieldwork supported by prior ethnographic descriptions. The picture to emerge is of an egalitarian multilingual speech community. The qualitative descriptions also provide basic facts about demographics and social structures of the community. Second is the linguistic description of the Nmbo language. Nmbo is an under-described language without substantial prior description, and this thesis contains a sketch grammar covering the basics aspects of Nmbo grammar. Finally there are three quantitative studies of variation. The vowel sociophonetic study and the word initial [h]-drop study are classic Labovian variationist studies that investigate patterns of variation across a sample of speakers. The former is based of elicited word list data, and the latter on naturalistic speech data. The third quantitative study takes a grammaticalisation approach to an emergent topic marker in a topicalising construction from a relative clause construction. This is the first thesis ever produced providing qualitative, descriptive, and quantitative data from a New Guinean speech community within a language ecology of vital indigenous multilingualism. The contributions of the thesis are two fold. Firstly, this thesis brings grammatical and sociolinguistic descriptions from an under-studied language. It is a socio-grammar (Nagy 2009) that considers language ecology, sociolinguistics, and grammatical description. Secondly, this thesis contributes empirical data on the sociolinguistics of small-scale speech communities. The classic sociolinguistic variable of gender is not found to be particularly significant in the variables studied, despite the community being highly gendered in other social domains. Village, however, shows some significance. As far as the three variables are concerned, Nmbo speakers show little community-internal variation and paint a picture of a tight-knit society of intimates (Trudgill 2011). The conclusion to the question of the sociolinguistics of diversification is that while there is some evidence of sociolinguistic differentiation within the Nmbo speech community, the most important social groups to orient against are the other sister language groups in the Morehead area. The nascent variation within the Nmbo speech community, combined with the ethnographic evidence of a cluster of dense and multiplex social networks, suggest that should the social need to differentiate between other Kerake arise, linguistic differentiation may occur rapidly

Significance in Language

This book offers a unique perspective on meaning in language, broadening the scope of existing understanding of meaning by introducing a comprehensive and cohesive account of meaning that draws on a wide range of linguistic approaches. The volume seeks to build up a complete picture of what meaning is, different types of meaning, and different ways of structuring the same meaning across myriad forms and varieties of language across such domains, such as everyday speech, advertising, humour, and academic writing. Supported by data from psycholinguistic and neurolinguistic research, the book combines different approaches from scholarship in semantics, including formalist, structuralist, cognitive, functionalist, and semiotics to demonstrate the ways in which meaning is expressed in words but also in word order and intonation. The book argues for a revised conceptualisation of meaning toward presenting a new perspective on semantics and its wider study in language and linguistic research. This book will appeal to scholars interested in meaning in language in such fields as linguistics, semantics, and semiotics