Keyboardless Visual Programming Using Voice, Handwriting, and Gesture

Visual programming languages have facilitated the application development process, improving our ability to express programs, as well as our ability to view, edit and interact with them. Yet even in programming environments, productivity is restricted by the primary input sources: the mouse and the keyboard. As an alternative, we investigate a program development interface which responds to the most natural human communication technologies: voice, handwriting and gesture. Speech- and pen-based systems have yet to find broad acceptance in everyday life because they are insufficiently advantageous to overcome problems with reliability. However, we believe that a visual programming environment with a multimodal user interface properly constrained so as not to exceed the limits of the current technology has the potential to increase programming productivity for not only those people who are manually or visually impaired, but for the general population as well. In this paper we report on such a system

End-User Service Computing: Spreadsheets as a Service Composition Tool

In this paper, we show how spreadsheets, an end-user development paradigm proven to be highly productive and simple to learn and use, can be used for complex service compositions. We identify the requirements for spreadsheet-based service composition, and present our framework that implements these requirements. Our framework enables spreadsheets to send requests and retrieve results from various local and remote services. We show how our tools support different composition patterns, and how the style of declarative dependencies of spreadsheets can facilitate service composition. We also discuss novel issues identified by using the framework in several projects and education

Voice and Touch Diagrams (VATagrams) Diagrams for the Visually Impaired

If a picture is worth a thousand words would you rather read the two pages of text or simply view the image? Most would choose to view the image; however, for the visually impaired this isn’t always an option. Diagrams assist people in visualizing relationships between objects. Most often these diagrams act as a source for quickly referencing information about relationships. Diagrams are highly visual and as such, there are few tools to support diagram creation for visually impaired individuals. To allow the visually impaired the ability to share the same advantages in school and work as sighted colleagues, an accessible diagram tool is needed. A suitable tool for the visually impaired to create diagrams should allow these individuals to: 1. easily define the type of relationship based diagram to be created, 2. easily create the components of a relationship based diagram, 3. easily modify the components of a relationship based diagram, 4. quickly understand the structure of a relationship based diagram, 5. create a visual representation which can be used by the sighted, and 6. easily accesses reference points for tracking diagram components. To do this a series of prototypes of a tool were developed that allow visually impaired users the ability to read, create, modify and share relationship based diagrams using sound and gestural touches. This was accomplished by creating a series of applications that could be run on an iPad using an overlay that restricts the areas in which a user can perform gestures. These prototypes were tested for usability using measures of efficiency, effectiveness and satisfaction. The prototypes were tested with visually impaired, blindfolded and sighted participants. The results of the evaluation indicate that the prototypes contain the main building blocks that can be used to complete a fully functioning application to be used on an iPad

Design of interactive distance learning equipment.

Distance education focuses on the entitlement of children with limited learning opportunities to schooling experiences that are equivalent to those enjoyed by other students. The means for delivering the curriculum to these learners have been many and varied, but most of these are either unaffordable or deficient in their provision of interactive audio and visual enhancements which are necessary for the pupils' effective understanding of the lesson. The project documented in this report attempts to expand students' access to the curriculum, by providing a cost effective solution to the problems of teaching at a distance. The proposal builds on the cooperative sharing of educational resources within clusters of schools, through which pupils are enabled to study subjects not offered in their own campuses but available in other schools within the cluster. The proposed product employs the concept of a collaborative "electronic blackboard" interface, which allows teachers and remote students to interact with freehand notations on a shared screen. Using audiographics conferencing techniques, remote lessons with live voices and graphic information are transmitted simultaneously to various participating sites. The central focus of the product's design is on the digitiser screen, which accepts handwritten input directly on the display. This provides the user with better eye-hand coordination than was possible in previous systems. The convertibility of the screen from a writing tablet into a computer monitor recognises the students' twin needs for a remote communication device and a computer for other school computing applications. The report covers an extensive analysis of the current status of distance education in Australia, the various technologies used in curriculum delivery, the reactions of users to existing remote learning methods, and the market for distance education and teleconferencing. It documents the various stages of the concept development, and presents the final design in photographs and in line drawings. A study of the commercial viability of the proposal is also included

Keys to Play: Music as a Ludic Medium from Apollo to Nintendo

How do keyboards make music playable? Drawing on theories of media, systems, and cultural techniques, Keys to Play spans Greek myth and contemporary Japanese digital games to chart a genealogy of musical play and its animation via improvisation, performance, and recreation. As a paradigmatic digital interface, the keyboard forms a field of play on which the book’s diverse objects of inquiry—from clavichords to PCs and eighteenth-century musical dice games to the latest rhythm-action titles—enter into analogical relations. Remapping the keyboard’s topography by way of Mozart and Super Mario, who head an expansive cast of historical and virtual actors, Keys to Play invites readers to unlock ludic dimensions of music that are at once old and new

Social, Private, and Trusted Wearable Technology under Cloud-Aided Intermittent Wireless Connectivity

There has been an unprecedented increase in the use of smart devices globally, together with novel forms of communication, computing, and control technologies that have paved the way for a new category of devices, known as high-end wearables. While massive deployments of these objects may improve the lives of people, unauthorized access to the said private equipment and its connectivity is potentially dangerous. Hence, communication enablers together with highly-secure human authentication mechanisms have to be designed.In addition, it is important to understand how human beings, as the primary users, interact with wearable devices on a day-to-day basis; usage should be comfortable, seamless, user-friendly, and mindful of urban dynamics. Usually the connectivity between wearables and the cloud is executed through the user’s more power independent gateway: this will usually be a smartphone, which may have potentially unreliable infrastructure connectivity. In response to these unique challenges, this thesis advocates for the adoption of direct, secure, proximity-based communication enablers enhanced with multi-factor authentication (hereafter refereed to MFA) that can integrate/interact with wearable technology. Their intelligent combination together with the connection establishment automation relying on the device/user social relations would allow to reliably grant or deny access in cases of both stable and intermittent connectivity to the trusted authority running in the cloud.The introduction will list the main communication paradigms, applications, conventional network architectures, and any relevant wearable-specific challenges. Next, the work examines the improved architecture and security enablers for clusterization between wearable gateways with a proximity-based communication as a baseline. Relying on this architecture, the author then elaborates on the social ties potentially overlaying the direct connectivity management in cases of both reliable and unreliable connection to the trusted cloud. The author discusses that social-aware cooperation and trust relations between users and/or the devices themselves are beneficial for the architecture under proposal. Next, the author introduces a protocol suite that enables temporary delegation of personal device use dependent on different connectivity conditions to the cloud.After these discussions, the wearable technology is analyzed as a biometric and behavior data provider for enabling MFA. The conventional approaches of the authentication factor combination strategies are compared with the ‘intelligent’ method proposed further. The assessment finds significant advantages to the developed solution over existing ones.On the practical side, the performance evaluation of existing cryptographic primitives, as part of the experimental work, shows the possibility of developing the experimental methods further on modern wearable devices.In summary, the set of enablers developed here for wearable technology connectivity is aimed at enriching people’s everyday lives in a secure and usable way, in cases when communication to the cloud is not consistently available