Phrasing Bimanual Interaction for Visual Design

Architects and other visual thinkers create external representations of their ideas to support early-stage design. They compose visual imagery with sketching to form abstract diagrams as representations. When working with digital media, they apply various visual operations to transform representations, often engaging in complex sequences. This research investigates how to build interactive capabilities to support designers in putting together, that is phrasing, sequences of operations using both hands. In particular, we examine how phrasing interactions with pen and multi-touch input can support modal switching among different visual operations that in many commercial design tools require using menus and tool palettes—techniques originally designed for the mouse, not pen and touch. We develop an interactive bimanual pen+touch diagramming environment and study its use in landscape architecture design studio education. We observe interesting forms of interaction that emerge, and how our bimanual interaction techniques support visual design processes. Based on the needs of architects, we develop LayerFish, a new bimanual technique for layering overlapping content. We conduct a controlled experiment to evaluate its efficacy. We explore the use of wearables to identify which user, and distinguish what hand, is touching to support phrasing together direct-touch interactions on large displays. From design and development of the environment and both field and controlled studies, we derive a set methods, based upon human bimanual specialization theory, for phrasing modal operations through bimanual interactions without menus or tool palettes

Bag-and-Dump: Design and Evaluation of a User Interface for manipulating items across multiple contexts.

The copy-and-paste paradigm is a fundamental operation in graphical user interfaces. However, existing copy-and-paste techniques have limitations, in particular in terms of efficiency and robustness against interruptions. This thesis is focusing on improving the user interface used to copy-and-paste objects across different contexts, such as a series folders. To improve this fundamental operation, a new copy-and-paste technique, called Bag-and-Dump, is proposed, implemented and evaluated. Bag-and-Dump aims to substantially reduce mouse movement by allowing the user to gather up (‘bag’) source data across different folders before ‘dumping’ the whole load at the destination. Additionally, Bag-and-Dump provides constant visual feedback in the form of a bag-like semantic cursor to increase robustness against interruptions. Bag-and-Dump was eval- uated against two standard copy-and-paste techniques (Keyboard Shortcuts and Drag-and-Drop) under a different number of contexts (folders) and with and without inter- ruptions. Results from the experiment not only showed that Bag-and-Dump indeed significantly reduces mouse movement, it also confirmed that Bag-and-Dump was 9% faster than Keyboard Shortcuts, one of the most popular copy-paste techniques among “expert users”

Designing Hybrid Interactions through an Understanding of the Affordances of Physical and Digital Technologies

Two recent technological advances have extended the diversity of domains and social contexts of Human-Computer Interaction: the embedding of computing capabilities into physical hand-held objects, and the emergence of large interactive surfaces, such as tabletops and wall boards. Both interactive surfaces and small computational devices usually allow for direct and space-multiplex input, i.e., for the spatial coincidence of physical action and digital output, in multiple points simultaneously. Such a powerful combination opens novel opportunities for the design of what are considered as hybrid interactions in this work. This thesis explores the affordances of physical interaction as resources for interface design of such hybrid interactions. The hybrid systems that are elaborated in this work are envisioned to support specific social and physical contexts, such as collaborative cooking in a domestic kitchen, or collaborative creativity in a design process. In particular, different aspects of physicality characteristic of those specific domains are explored, with the aim of promoting skill transfer across domains. irst, different approaches to the design of space-multiplex, function-specific interfaces are considered and investigated. Such design approaches build on related work on Graspable User Interfaces and extend the design space to direct touch interfaces such as touch-sensitive surfaces, in different sizes and orientations (i.e., tablets, interactive tabletops, and walls). These approaches are instantiated in the design of several experience prototypes: These are evaluated in different settings to assess the contextual implications of integrating aspects of physicality in the design of the interface. Such implications are observed both at the pragmatic level of interaction (i.e., patterns of users' behaviors on first contact with the interface), as well as on user' subjective response. The results indicate that the context of interaction affects the perception of the affordances of the system, and that some qualities of physicality such as the 3D space of manipulation and relative haptic feedback can affect the feeling of engagement and control. Building on these findings, two controlled studies are conducted to observe more systematically the implications of integrating some of the qualities of physical interaction into the design of hybrid ones. The results indicate that, despite the fact that several aspects of physical interaction are mimicked in the interface, the interaction with digital media is quite different and seems to reveal existing mental models and expectations resulting from previous experience with the WIMP paradigm on the desktop PC

Touch Crossing-Based Selection and the Pin-and-Cross Technique

This thesis focuses on the evaluation, exploration and demonstration of crossing paradigm with touch modality. Under the scenario of crossing selection, the target is selected by stroking through a boundary 'goal' instead of pointing inside a perimeter. We present empirical evidence to validate crossing performance for touch. Inspired by the experimental results, we then develop, evaluate and demonstrate a new unimanual multi-touch interaction space called 'pin-and-cross'. It combines one or more static touches ('pins') with another touch to cross a radial target, all performed with one hand. Our work serves to provide necessary support for the exploration and evaluation of more expressive multi-touch crossing techniques

A Neural Signal Processor for Low-Latency Spike Inference

This thesis describes the development of a system that can assign identities to a population of single-units, in multi-electrode recordings, at single-spike resolution with low-latency. The system has two parts. The first is a Field-Programmable Gate Array (FPGA)-based Neural Signal Processor (NSP) that receives raw input and generates labelled spikes as output, a process referred to as real-time spike inference. The second is a piece of software (Spiketag) that runs on a PC, communicates with the NSP, and generates a spike-sorted model to guide the real-time spike inference. The NSP provides clocks and control signals to five 32-channel INTAN RHD2132 chips to manage the acquisition of 160 channels of raw neural data. In parallel, the NSP further filters, detects and extracts extracellular spike waveforms from the raw neural data recorded by tetrodes or silicon probes and assigns single-unit identity to each detected spike. A set of Python application programming interfaces (APIs) was developed in Spiketag to enable the communication between the NSP and the PC. These APIs allow the NSP to obtain a model from the PC, which holds parameters such as reference channels, spike detection thresholds, spike feature transformation matrix and vector quantized clusters generated by spike sorting a short recording session. Using the spike-sorted model, the NSP performs data acquisition and real-time spike inference simultaneously. Algorithmic modules were implemented in the FPGA and pipelined to compute during 40 ms acquisition intervals. At the output end of the FPGA NSP, the real-time assigned single-unit identity (spike-id) is packaged with the timestamp, the electrode group, and the spike features as a spike-id packet. Spike-id packets are asynchronously transmitted through a low-latency Peripheral Component Interconnect Express (PCIe) interface to the PC, producing the real-time spike trains. The real-time spike trains can be used for further processing, such as real-time decoding. Several types of ground-truth data, including intracellular/extracellular paired recordings, synthesized tetrode extracellular waveforms with ground-truth spike timing and high-channel-count silicon probe recordings with ground-truth animal positions during navigation were used to validate the low-latency (1 ms) and high-accuracy (as high as state-of-the-art offline sorting and decoding algorithms) of the NSP’s real-time spike inference and the NSP-based real-time population decoding performance

Bringing the Physical to the Digital

This dissertation describes an exploration of digital tabletop interaction styles, with the ultimate goal of informing the design of a new model for tabletop interaction. In the context of this thesis the term digital tabletop refers to an emerging class of devices that afford many novel ways of interaction with the digital. Allowing users to directly touch information presented on large, horizontal displays. Being a relatively young field, many developments are in flux; hardware and software change at a fast pace and many interesting alternative approaches are available at the same time. In our research we are especially interested in systems that are capable of sensing multiple contacts (e.g., fingers) and richer information such as the outline of whole hands or other physical objects. New sensor hardware enable new ways to interact with the digital. When embarking into the research for this thesis, the question which interaction styles could be appropriate for this new class of devices was a open question, with many equally promising answers. Many everyday activities rely on our hands ability to skillfully control and manipulate physical objects. We seek to open up different possibilities to exploit our manual dexterity and provide users with richer interaction possibilities. This could be achieved through the use of physical objects as input mediators or through virtual interfaces that behave in a more realistic fashion. In order to gain a better understanding of the underlying design space we choose an approach organized into two phases. First, two different prototypes, each representing a specific interaction style – namely gesture-based interaction and tangible interaction – have been implemented. The flexibility of use afforded by the interface and the level of physicality afforded by the interface elements are introduced as criteria for evaluation. Each approaches’ suitability to support the highly dynamic and often unstructured interactions typical for digital tabletops is analyzed based on these criteria. In a second stage the learnings from these initial explorations are applied to inform the design of a novel model for digital tabletop interaction. This model is based on the combination of rich multi-touch sensing and a three dimensional environment enriched by a gaming physics simulation. The proposed approach enables users to interact with the virtual through richer quantities such as collision and friction. Enabling a variety of fine-grained interactions using multiple fingers, whole hands and physical objects. Our model makes digital tabletop interaction even more “natural”. However, because the interaction – the sensed input and the displayed output – is still bound to the surface, there is a fundamental limitation in manipulating objects using the third dimension. To address this issue, we present a technique that allows users to – conceptually – pick objects off the surface and control their position in 3D. Our goal has been to define a technique that completes our model for on-surface interaction and allows for “as-direct-as possible” interactions. We also present two hardware prototypes capable of sensing the users’ interactions beyond the table’s surface. Finally, we present visual feedback mechanisms to give the users the sense that they are actually lifting the objects off the surface. This thesis contributes on various levels. We present several novel prototypes that we built and evaluated. We use these prototypes to systematically explore the design space of digital tabletop interaction. The flexibility of use afforded by the interaction style is introduced as criterion alongside the user interface elements’ physicality. Each approaches’ suitability to support the highly dynamic and often unstructured interactions typical for digital tabletops are analyzed. We present a new model for tabletop interaction that increases the fidelity of interaction possible in such settings. Finally, we extend this model so to enable as direct as possible interactions with 3D data, interacting from above the table’s surface

From collection to reflection : on designing Freed, a tool for free and flexible organization of designers' digital work

Designers collect a lot of information during the design process, such as background research, ideas, notes, sketches, photos, videos and feedback from various stakeholders. A large part of this information gets lost in folders on individual computers, inside documents and presentations, or on pages in designers' notebooks. This is wasteful, because this information can be used for reflection. Reflection enables designers to give meaning to their experience and to develop. When reflecting designers think about what, how and why they design, or more specifically: It allows them to gain overview of, gain insight in and give direction to their design process, ideas, designs, skills, knowledge, interests, ambitions, identity and community. Reflection concerns integration, i.e., to explore relations, and diversity, i.e., to explore new perspectives. Reflection has a dual nature. On the one hand, it is an explicit action that requires designers to step out of the flow of designing. On the other hand, it is an implicit process that happens automatically while designing. This dual nature also holds true for how reflection can be supported. On the one hand, one can specifically dedicate time for reflection. On the other hand, reflection can be captured ¿in the action', during or right after other activities that are part of the design process. This project adopts a Research-through-Design approach: By designing and evaluating a software application called Freed, insight is gained in how designers' reflection can be supported by means of their digital collections. Freed is discussed and evaluated with design students and designer-researchers at the department of Industrial Design at the Eindhoven University of Technology. This context, which has strongly inspired and influenced this work, is introduced in the first chapter. In the second chapter, the foundations, goals and approach of this research are outlined. Based on the goal of supporting both integration and diversity, the case is made for free and flexible organization. Freedom is defined as the possibility to let structure and meaning emerge during interaction, instead of being imposed by the structure of the application. It can also be referred to as the openness of the application, or its ability to be appropriated and used in diverse situations. Flexibility is defined as the possibility to easily reorganize and reuse design work and to switch between perspectives on this work. Related work concerning reflection, design and collection, is discussed in the third chapter. This chapter ends with the conclusion that design is about action and exploration, and that reflection cannot be seen independent from action. Opportunities for reflection can be provided by a flexible person- and context- dependent design process that allows for many switches between activities, and regular reframing of the design situation. A system for supporting reflection should fit this flexible nature, and give designers the freedom to use the system for their own purposes. This desired combination of freedom and flexibility is not found in existing tools and systems. For example, existing tools and systems include elements that may inhibit free and flexible organization of the collection, such as similarity criteria, IBIS notations, and hierarchical relations. The main process of design and evaluation is discussed chronologically in chapters four to seven. The fourth chapter introduces initial design concepts, and argues for a focus on software. A first software prototype called ¿The Magnetic Collage Software' is discussed, along with a personal reflection on the use of it. From this reflection is concluded that the initial prototype works well for gaining overview quickly, but that it needed to be improved in order to support more active exploration of relations and perspectives. In chapter five the initial version of Freed is discussed. The main elements of Freed are a zoomable unconstrained canvas, a forcebased layout, and the possibility to create multiple organizations of the same content. The purpose of the force-based layout, in which related content attracts each other and non-related content repulses each other, is to encourage the exploration of relations and different spatial organizations. These organizations, or ¿views', can for example be used for a specific design activity or project phase (e.g. presenting, mapping related work), for creating an overview of the entire design process, for a portfolio of multiple projects, or for explaining the perspective of a given designer or stakeholder. The chapter concludes with a discussion of first feedback from design students and a case study in which the software was used for building a presentation and collection of the research group in which this research is carried out. The case study showed how the activities of building a presentation and collection can support each other and how this active, integrated use can lead to reflection. Chapter six focuses on the use of Freed during the design process. It discusses a design iteration, an introductory workshop and questionnaire, and a semester-long evaluation during student design projects. This evaluation showed that Freed was valued as a tool for gaining overview of and revisiting design work and process. Additionally, it showed that in order to support more exploration and reflection during and after the design process, the threshold for documentation and communication needed to be lowered, a better balance between organization and visualization needed to be obtained, and the integration and overview of views needed to be improved. Chapter seven focuses on using Freed as a tool for exploring relations and perspectives. It discusses a final design iteration, an evaluation during which students used Freed to explore their personal views on design theory, a case study of designerresearchers using the software for organizing student projects, and a reflection on personal use of Freed. These cases showed how Freed provides the freedom to be used differently by various design students and how multiple views can help to integrate work and to explore relations and perspectives. They also showed that both freedom and structure are needed for reflection, and how Freed can be used complementary to other activities such as physical diagramming or clustering. For example, physical clustering (e.g. of Post-it notes or printed images) helps to quickly gain consensus among a group and to make decisions, while Freed provides input for more dynamic discussions, allows for personal exploration (i.e. to temporarily loose the group consensus), and allows for insight to develop gradually. Chapter 8 concerns a reflection on this research as a whole, and discusses ¿conditions for collection and reflection', future work, and Research-through-Design. The main conclusions are that reflection builds on active use of a digital collection, that active use benefits from having a rich, visual, integrated collection, that reflection requires both freedom and structure, that structure emerges from direct, expressive local interaction, and that using a digital collection for reflection requires time and skill. In future work, there's a need to move beyond the confinements of a single software application, and to explore how to design for systems that integrate diverse products and applications. Additionally, there's a need to explore the integration of collection and reflection in a collaborative setting (and) in design practice

Guide de conception d'une technique de désignation 3D

National audienceNous présentons un tour d'horizon des techniques de désignation existantes en IHM, aussi bien en 2D qu'en 3D. Leur analyse nous a permis de proposer un guide de conception d'une technique de désignation 3D pour les développeurs d'environnements virtuels. Une illustration de cette méthode est proposée avec le pointage sémantique en 3D