Accessibility and tangible interaction in distributed workspaces based on multi-touch surfaces

[EN] Traditional interaction mechanisms in distributed digital spaces often fail to consider the intrinsic properties of action, perception, and communication among workgroups, which may affect access to the common resources used to mutually organize information. By developing suitable spatial geometries and natural interaction mechanisms, distributed spaces can become blended where the physical and virtual boundaries of local and remote spaces merge together to provide the illusion of a single unified space. In this paper, we discuss the importance of blended interaction in distributed spaces and the particular challenges faced when designing accessible technology. We illustrate this discussion through a new tangible interaction mechanism for collaborative spaces based on tabletop system technology implemented with optical frames. Our tangible elements facilitate the exchange of digital information in distributed collaborative settings by providing a physical manifestation of common digital operations. The tangibles are designed as passive elements that do not require the use of any additional hardware or external power while maintaining a high degree of accuracy.This work was supported by the Spanish Ministry of Economy and Competitiveness and the European Regional Development Fund, through the ANNOTA Project (Ref. TIN2013-46036-C3-1-R).Salvador-Herranz, G.; Camba, J.; Contero, M.; Naya Sanchis, F. (2018). Accessibility and tangible interaction in distributed workspaces based on multi-touch surfaces. Universal Access in the Information Society. 17(2):247-256. https://doi.org/10.1007/s10209-017-0563-7S247256172Arkin, E.M., Chew, L.P., Huttenlocher, D.P., Kedem, K., Mitchell, J.S.B.: An efficiently computable metric for comparing polygonal shapes. IEEE Trans. Acoust. Speech Signal Process. 13(3), 209–216 (1991)Benyon, D.: Presence in blended spaces. Interact. Comput. 24(4), 219–226 (2012)Bhalla, M.R., Bhalla, A.V.: Comparative study of various touchscreen technologies. Int. J. Comput. Appl. 6(8), 12–18 (2010)Bradski, G., Kaehler, A.: Learning OpenCV: Computer Vision with the OpenCV Library. O’Reilly Media Inc., Newton (2008)Candela, E.S., Pérez, M.O., Romero, C.M., López, D.C.P., Herranz, G.S., Contero, M., Raya, M.A.: Humantop: a multi-object tracking tabletop. Multimed. Tools Appl. 70(3), 1837–1868 (2014)Cohen, J., Withgott, M., Piernot, P.: Logjam: a tangible multi-person interface for video logging. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 128–135. ACM (1999)Couture, N., Rivière, G., Reuter, P.: Geotui: a tangible user interface for geoscience. In: Proceedings of the 2nd International Conference on Tangible and Embedded Interaction, pp. 89–96. ACM (2008)de la Guía, E., Lozano, M.D., Penichet, V.R.: Cognitive rehabilitation based on collaborative and tangible computer games. In: 2013 7th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth), pp. 389–392. IEEE (2013)Dietz, P., Leigh, D.: Diamondtouch: a multi-user touch technology. In: Proceedings of the 14th Annual ACM Symposium on User Interface Software and Technology, pp. 219–226. ACM (2001)Falcão, T.P., Price, S.: What have you done! the role of ‘interference’ in tangible environments for supporting collaborative learning. In: Proceedings of the 9th International Conference on Computer Supported Collaborative Learning-Volume 1, pp. 325–334. International Society of the Learning Sciences (2009)Fallman, D.: Wear, point and tilt. In: Proceedings of the Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques, pp. 293–302. ACM Press (2002)Fishkin, K.P., Gujar, A., Harrison, B.L., Moran, T.P., Want, R.: Embodied user interfaces for really direct manipulation. Commun. ACM 43(9), 74–80 (2000)Fitzmaurice, G.W., Buxton, W.: An empirical evaluation of graspable user interfaces: towards specialized, space-multiplexed input. In: Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, pp. 43–50. ACM (1997)Fitzmaurice, G.W., Ishii, H., Buxton, W.A.: Bricks: laying the foundations for graspable user interfaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 442–449. ACM Press (1995)Graham, R.L., Yao, F.F.: Finding the convex hull of a simple polygon. J. Algorithms 4(4), 324–331 (1983)Hartigan, J.A., Wong, M.A.: Algorithm as 136: a k-means clustering algorithm. J. R. Stat. Soc.: Ser. C (Appl. Stat.) 28(1), 100–108 (1979)Higgins, S.E., Mercier, E., Burd, E., Hatch, A.: Multi-touch tables and the relationship with collaborative classroom pedagogies: a synthetic review. Int. J. Comput. Support. Collab. Learn. 6(4), 515–538 (2011)Hinckley, K., Pausch, R., Goble, J.C., Kassell, N.F.: Passive real-world interface props for neurosurgical visualization. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 452–458. ACM (1994)Hinske, S.: Determining the position and orientation of multi-tagged objects using RFID technology. In: 5th Annual IEEE International Conference on Pervasive Computing and Communications Workshops, 2007. PerCom Workshops’07, pp. 377–381. IEEE (2007)Hornecker, E.: A design theme for tangible interaction: embodied facilitation. In: ECSCW 2005, pp. 23–43. Springer (2005)Hoshi, K., Öhberg, F., Nyberg, A.: Designing blended reality space: conceptual foundations and applications. In: Proceedings of the 25th BCS Conference on Human–Computer Interaction, pp. 217–226. British Computer Society (2011)Ishii, H.: Tangible User Interfaces. CRC Press, Boca Raton (2007)Ishii, H., Ullmer, B.: Tangible bits: towards seamless interfaces between people, bits and atoms. In: Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, pp. 234–241. ACM (1997)Jacob, R.J., Girouard, A., Hirshfield, L.M., Horn, M.S., Shaer, O., Solovey, E.T., Zigelbaum, J.: Reality-based interaction: a framework for post-wimp interfaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 201–210. ACM (2008)Jetter, H.C., Dachselt, R., Reiterer, H., Quigley, A., Benyon, D., Haller, M.: Blended Interaction: Envisioning Future Collaborative Interactive Spaces. ACM, New York (2013)Jin, X., Han, J.: Quality threshold clustering. In: Sammut, C., Webb, G.I. (eds.) Encyclopedia of Machine Learning, pp. 820–820. Springer, Boston, MA (2011)Jordà, S., Geiger, G., Alonso, M., Kaltenbrunner, M.: The reactable: exploring the synergy between live music performance and tabletop tangible interfaces. In: Proceedings of the 1st International Conference on Tangible and Embedded Interaction, pp. 139–146. ACM (2007)Kaltenbrunner, M., Bovermann, T., Bencina, R., Costanza, E.: Tuio: a protocol for table-top tangible user interfaces. In: Proceedings of the 6th International Workshop on Gesture in Human–Computer Interaction and Simulation, pp. 1–5 (2005)Kirk, D., Sellen, A., Taylor, S., Villar, N., Izadi, S.: Putting the physical into the digital: issues in designing hybrid interactive surfaces. In: Proceedings of the 23rd British HCI Group Annual Conference on People and Computers: Celebrating People and Technology, pp. 35–44. British Computer Society (2009)Marques, T., Nunes, F., Silva, P., Rodrigues, R.: Tangible interaction on tabletops for elderly people. In: International Conference on Entertainment Computing, pp. 440–443. Springer (2011)Müller, D.: Mixed reality systems. iJOE 5(S2), 10–11 (2009)Newton-Dunn, H., Nakano, H., Gibson, J.: Block jam: a tangible interface for interactive music. In: Proceedings of the 2003 Conference on New Interfaces for Musical Expression, pp. 170–177. National University of Singapore (2003)Patten, J., Recht, B., Ishii, H.: Audiopad: a tag-based interface for musical performance. In: Proceedings of the 2002 Conference on New Interfaces for Musical Expression, pp. 1–6. National University of Singapore (2002)Patten, J., Recht, B., Ishii, H.: Interaction techniques for musical performance with tabletop tangible interfaces. In: Proceedings of the 2006 ACM SIGCHI International Conference on Advances in Computer Entertainment Technology, p. 27. ACM (2006)PQLabs: Inc. http://multitouch.com/ . Retrieved on 16 October 2016Ryokai, K., Marti, S., Ishii, H.: I/o brush: drawing with everyday objects as ink. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’04, pp. 303–310. ACM, New York (2004). doi: 10.1145/985692.985731Salvador, G., Bañó, M., Contero, M., Camba, J.: Evaluation of a distributed collaborative workspace as a creativity tool in the context of design education. In: 2014 IEEE Frontiers in Education Conference (FIE) Proceedings, pp. 1–7. IEEE (2014)Salvador-Herranz, G., Contero, M., Camba, J.: Use of tangible marks with optical frame interactive surfaces in collaborative design scenarios based on blended spaces. In: International Conference on Cooperative Design, Visualization and Engineering, pp. 253–260. Springer (2014)Salvador-Herranz, G., Camba, J.D., Naya, F., Contero, M.: On the integration of tangible elements with multi-touch surfaces for the collaborative creation of concept maps. In: International Conference on Learning and Collaboration Technologies, pp. 177–186. Springer (2016)Schöning, J., Hook, J., Bartindale, T., Schmidt, D., Oliver, P., Echtler, F., Motamedi, N., Brandl, P., von Zadow, U.: Building interactive multi-touch surfaces. In: Müller-Tomfelde, C. (ed.) Tabletops-Horizontal Interactive Displays, pp. 27–49. Springer, London, UK (2010)Shaer, O., Hornecker, E.: Tangible user interfaces: past, present, and future directions. Found. Trends Hum. Comput. Interact. 3(1–2), 1–137 (2010)Shen, C., Everitt, K., Ryall, K.: Ubitable: Impromptu face-to-face collaboration on horizontal interactive surfaces. In: International Conference on Ubiquitous Computing, pp. 281–288. Springer (2003)Suzuki, H., Kato, H.: Algoblock: a tangible programming language, a tool for collaborative learning. In: Proceedings of 4th European Logo Conference, pp. 297–303 (1993)Suzuki, H., Kato, H.: Interaction-level support for collaborative learning: Algoblockan open programming language. In: The 1st International Conference on Computer Support for Collaborative Learning, pp. 349–355. L. Erlbaum Associates Inc. (1995)Terrenghi, L., Kirk, D., Richter, H., Krämer, S., Hilliges, O., Butz, A.: Physical handles at the interactive surface: exploring tangibility and its benefits. In: Proceedings of the Working Conference on Advanced Visual Interfaces, pp. 138–145. ACM (2008)Veltkamp, R.C.: Shape matching: similarity measures and algorithms. In: SMI 2001 International Conference on Shape Modeling and Applications, pp. 188–197. IEEE (2001)Weinberg, G., Gan, S.L.: The squeezables: Toward an expressive and interdependent multi-player musical instrument. Comput. Music J. 25(2), 37–45 (2001)Weiser, M.: Some computer science issues in ubiquitous computing. Commun. ACM 36(7), 75–84 (1993)Wilson, F.: The hand: how its use shapes the brain, language, and human culture. Vintage Series. Vintage Books (1998). https://books.google.es/books?id=l_Boy_-NkwUCZuckerman, O., Arida, S., Resnick, M.: Extending tangible interfaces for education: digital montessori-inspired manipulatives. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 859–868. ACM (2005

Collaborative Human-Computer Interaction with Big Wall Displays - BigWallHCI 2013 3rd JRC ECML Crisis Management Technology Workshop

The 3rd JRC ECML Crisis Management Technology Workshop on Human-Computer Interaction with Big Wall Displays in Situation Rooms and Monitoring Centres was co-organised by the European Commission Joint Research Centre and the University of Applied Sciences St. Pölten, Austria. It took place in the European Crisis Management Laboratory (ECML) of the JRC in Ispra, Italy, from 18 to 19 April 2013. 40 participants from stakeholders in the EC, civil protection bodies, academia, and industry attended the workshop. The hardware of large display areas is on the one hand mature since many years and on the other hand changing rapidly and improving constantly. This high pace developments promise amazing new setups with respect to e.g., pixel density or touch interaction. On the software side there are two components with room for improvement: 1. the software provided by the display manufacturers to operate their video walls (source selection, windowing system, layout control) and 2. dedicated ICT systems developed to the very needs of crisis management practitioners and monitoring centre operators. While industry starts to focus more on the collaborative aspects of their operating software already, the customized and tailored ICT applications needed are still missing, unsatisfactory, or very expensive since they have to be developed from scratch many times. Main challenges identified to enhance big wall display systems in crisis management and situation monitoring contexts include: 1. Interaction: Overcome static layouts and/or passive information consumption. 2. Participatory Design & Development: Software needs to meet users’ needs. 3. Development and/or application of Information Visualisation & Visual Analytics principle to support the transition from data to information to knowledge. 4. Information Overload: Proper methods for attention management, automatic interpretation, incident detection, and alarm triggering are needed to deal with the ever growing amount of data to be analysed.JRC.G.2-Global security and crisis managemen

Analysing, visualising and supporting collaborative learning using interactive tabletops

The key contribution of this thesis is a novel approach to design, implement and evaluate the conceptual and technological infrastructure that captures student’s activity at interactive tabletops and analyses these data through Interaction Data Analytics techniques to provide support to teachers by enhancing their awareness of student’s collaboration. To achieve the above, this thesis presents a series of carefully designed user studies to understand how to capture, analyse and distil indicators of collaborative learning. We perform this in three steps: the exploration of the feasibility of the approach, the construction of a novel solution and the execution of the conceptual proposal, both under controlled conditions and in the wild. A total of eight datasets were analysed for the studies that are described in this thesis. This work pioneered in a number of areas including the application of data mining techniques to study collaboration at the tabletop, a plug-in solution to add user-identification to a regular tabletop using a depth sensor and the first multi-tabletop classroom used to run authentic collaborative activities associated with the curricula. In summary, while the mechanisms, interfaces and studies presented in this thesis were mostly explored in the context of interactive tabletops, the findings are likely to be relevant to other forms of groupware and learning scenarios that can be implemented in real classrooms. Through the mechanisms, the studies conducted and our conceptual framework this thesis provides an important research foundation for the ways in which interactive tabletops, along with data mining and visualisation techniques, can be used to provide support to improve teacher’s understanding about student’s collaboration and learning in small groups

Exploring The Impact Of Configuration And Mode Of Input On Group Dynamics In Computing

Objectives: Large displays and new technologies for interacting with computers offer a rich area for the development of new tools to facilitate collaborative concept mapping activities. In this thesis, WiiConcept is described as a tool designed to allow the use of multiple WiiRemotes for the collaborative creation of concept maps, with and without gestures. Subsequent investigation of participants' use of the system considers the effect of single and multiple input streams when using the software with and without gestures and the impact upon group concept mapping process outcomes and interactions when using a large display. Methods: Data is presented from an exploratory study of twenty two students who have used the tool. Half of the pairs used two WiiRemotes, while the remainder used one WiiRemote. All pairs created one map without gestures and one map with gestures. Data about their maps, interactions and responses to the tool were collected. Results: Analysis of coded transcripts indicates that one-controller afforded higher levels of interaction, with the use of gestures also increasing the number of interactions seen. Additionally, the result indicated that there were significantly more interactions of the 'shows solidarity', 'gives orientation', and 'gives opinion' categories (defined by the Bales' interaction processes assessment), when using one-controller as opposed to two. Furthermore, there were more interactions for the 'shows solidarity', 'tension release', 'gives orientation' and 'shows tension' categories when using gestures as opposed to the non-use of gestures. Additionally, there were no significant differences in the perceived dominance of individuals, as measured on the social dominance scales, for the amount of interaction displayed, however, there was a significant main effect of group conversational control score on the 'gives orientation' construct, with a higher number of interactions for low, mixed and high scores of this type when dyads had one-controller as opposed to two-controllers. There was also a significant interaction effect of group conversational control score on the 'shows solidarity' construct with a higher number of interactions for all scores of this type when dyads had one-controller as opposed to two-controllers. The results also indicate that for the WiiConcept there was no difference between number of controllers in the detail in the maps, and that all users found the tool to be useful for the collaborative creation of concept maps. At the same time, engaging in disagreement was related to the amount of nodes created with disagreement leading to more nodes being created. Conclusions: Use of one-controller afforded higher levels of interaction, with gestures also increasing the number of interactions seen. If a particular type of interaction is associated with more nodes, there might also be some argument for only using one-controller with gestures enabled to promote cognitive conflict within groups. All participants responded that the tool was relatively easy to use and engaging, which suggests that this tool could be integrated into collaborative concept mapping activities, allowing for greater collaborative knowledge building and sharing of knowledge, due to the increased levels of interaction for one-controller. As research has shown concept mapping can be useful for promoting the understanding of complex ideas, therefore the adoption of the WiiConcept tool as part of a small group learning activity may lead to deeper levels of understanding. Additionally, the use of gestures suggests that this mode of input does not affect the amount of words, nodes, and edges created in a concept map. Further research, over a longer period of time, may see improvement with this form of interaction, with increased mastery of gestural movement leading to greater detail of conceptual mapping

Interactive Visualization Lenses:: Natural Magic Lens Interaction for Graph Visualization

Information visualization is an important research field concerned with making sense and inferring knowledge from data collections. Graph visualizations are specific techniques for data representation relevant in diverse application domains among them biology, software-engineering, and business finance. These data visualizations benefit from the display space provided by novel interactive large display environments. However, these environments also cause new challenges and result in new requirements regarding the need for interaction beyond the desktop and according redesign of analysis tools. This thesis focuses on interactive magic lenses, specialized locally applied tools that temporarily manipulate the visualization. These may include magnification of focus regions but also more graph-specific functions such as pulling in neighboring nodes or locally reducing edge clutter. Up to now, these lenses have mostly been used as single-user, single-purpose tools operated by mouse and keyboard. This dissertation presents the extension of magic lenses both in terms of function as well as interaction for large vertical displays. In particular, this thesis contributes several natural interaction designs with magic lenses for the exploration of graph data in node-link visualizations using diverse interaction modalities. This development incorporates flexible switches between lens functions, adjustment of individual lens properties and function parameters, as well as the combination of lenses. It proposes interaction techniques for fluent multi-touch manipulation of lenses, controlling lenses using mobile devices in front of large displays, and a novel concept of body-controlled magic lenses. Functional extensions in addition to these interaction techniques convert the lenses to user-configurable, personal territories with use of alternative interaction styles. To create the foundation for this extension, the dissertation incorporates a comprehensive design space of magic lenses, their function, parameters, and interactions. Additionally, it provides a discussion on increased embodiment in tool and controller design, contributing insights into user position and movement in front of large vertical displays as a result of empirical investigations and evaluations.Informationsvisualisierung ist ein wichtiges Forschungsfeld, das das Analysieren von Daten unterstützt. Graph-Visualisierungen sind dabei eine spezielle Variante der Datenrepräsentation, deren Nutzen in vielerlei Anwendungsfällen zum Einsatz kommt, u.a. in der Biologie, Softwareentwicklung und Finanzwirtschaft. Diese Datendarstellungen profitieren besonders von großen Displays in neuen Displayumgebungen. Jedoch bringen diese Umgebungen auch neue Herausforderungen mit sich und stellen Anforderungen an Nutzerschnittstellen jenseits der traditionellen Ansätze, die dadurch auch Anpassungen von Analysewerkzeugen erfordern. Diese Dissertation befasst sich mit interaktiven „Magischen Linsen“, spezielle lokal-angewandte Werkzeuge, die temporär die Visualisierung zur Analyse manipulieren. Dabei existieren zum Beispiel Vergrößerungslinsen, aber auch Graph-spezifische Manipulationen, wie das Anziehen von Nachbarknoten oder das Reduzieren von Kantenüberlappungen im lokalen Bereich. Bisher wurden diese Linsen vor allem als Werkzeug für einzelne Nutzer mit sehr spezialisiertem Effekt eingesetzt und per Maus und Tastatur bedient. Die vorliegende Doktorarbeit präsentiert die Erweiterung dieser magischen Linsen, sowohl in Bezug auf die Funktionalität als auch für die Interaktion an großen, vertikalen Displays. Insbesondere trägt diese Dissertation dazu bei, die Exploration von Graphen mit magischen Linsen durch natürliche Interaktion mit unterschiedlichen Modalitäten zu unterstützen. Dabei werden flexible Änderungen der Linsenfunktion, Anpassungen von individuellen Linseneigenschaften und Funktionsparametern, sowie die Kombination unterschiedlicher Linsen ermöglicht. Es werden Interaktionstechniken für die natürliche Manipulation der Linsen durch Multitouch-Interaktion, sowie das Kontrollieren von Linsen durch Mobilgeräte vor einer Displaywand vorgestellt. Außerdem wurde ein neuartiges Konzept körpergesteuerter magischer Linsen entwickelt. Funktionale Erweiterungen in Kombination mit diesen Interaktionskonzepten machen die Linse zu einem vom Nutzer einstellbaren, persönlichen Arbeitsbereich, der zudem alternative Interaktionsstile erlaubt. Als Grundlage für diese Erweiterungen stellt die Dissertation eine umfangreiche analytische Kategorisierung bisheriger Forschungsarbeiten zu magischen Linsen vor, in der Funktionen, Parameter und Interaktion mit Linsen eingeordnet werden. Zusätzlich macht die Arbeit Vor- und Nachteile körpernaher Interaktion für Werkzeuge bzw. ihre Steuerung zum Thema und diskutiert dabei Nutzerposition und -bewegung an großen Displaywänden belegt durch empirische Nutzerstudien

An Abstraction Framework for Tangible Interactive Surfaces

This cumulative dissertation discusses - by the example of four subsequent publications - the various layers of a tangible interaction framework, which has been developed in conjunction with an electronic musical instrument with a tabletop tangible user interface. Based on the experiences that have been collected during the design and implementation of that particular musical application, this research mainly concentrates on the definition of a general-purpose abstraction model for the encapsulation of physical interface components that are commonly employed in the context of an interactive surface environment. Along with a detailed description of the underlying abstraction model, this dissertation also describes an actual implementation in the form of a detailed protocol syntax, which constitutes the common element of a distributed architecture for the construction of surface-based tangible user interfaces. The initial implementation of the presented abstraction model within an actual application toolkit is comprised of the TUIO protocol and the related computer-vision based object and multi-touch tracking software reacTIVision, along with its principal application within the Reactable synthesizer. The dissertation concludes with an evaluation and extension of the initial TUIO model, by presenting TUIO2 - a next generation abstraction model designed for a more comprehensive range of tangible interaction platforms and related application scenarios

Light on horizontal interactive surfaces: Input space for tabletop computing

In the last 25 years we have witnessed the rise and growth of interactive tabletop research, both in academic and in industrial settings. The rising demand for the digital support of human activities motivated the need to bring computational power to table surfaces. In this article, we review the state of the art of tabletop computing, highlighting core aspects that frame the input space of interactive tabletops: (a) developments in hardware technologies that have caused the proliferation of interactive horizontal surfaces and (b) issues related to new classes of interaction modalities (multitouch, tangible, and touchless). A classification is presented that aims to give a detailed view of the current development of this research area and define opportunities and challenges for novel touch- and gesture-based interactions between the human and the surrounding computational environment. © 2014 ACM.This work has been funded by Integra (Amper Sistemas and CDTI, Spanish Ministry of Science and Innovation) and TIPEx (TIN2010-19859-C03-01) projects and Programa de Becas y Ayudas para la Realización de Estudios Oficiales de Máster y Doctorado en la Universidad Carlos III de Madrid, 2010