Holmquist, Lars

Northumbria Research Link

Northumbria Research LinkCitation: Holmquist, Lars (2019) The future of tangible user interfaces. interactions, 26 (5). pp. 82-85. ISSN 1072-5520 Published by: Association for Computing MachineryURL: https://doi.org/10.1145/3352157 <https://doi.org/10.1145/3352157>This version was downloaded from Northumbria Research Link: http://nrl.northumbria.ac.uk/40763/Northumbria University has developed Northumbria Research Link (NRL) to enable users to access the University’s research output. Copyright © and moral rights for items on NRL are retained by the individual author(s) and/or other copyright owners.  Single copies of full items can be reproduced, displayed or performed, and given to third parties in any format or medium for personal research or study, educational, or not-for-profit purposes without prior permission or charge, provided the authors, title and full bibliographic details are given, as well as a hyperlink and/or URL to the original metadata page. The content must not be changed in any way. Full items must not be sold commercially in any  format or medium without formal permission of the copyright holder.  The full policy is available online: http://nrl.northumbria.ac.uk/pol  i cies.html  This  document  may differ  from the  final,  published version of  the research  and has been made available online in accordance with publisher policies. To read and/or cite from the published version of the research, please visit the publisher’s website (a subscription may be required.)                        The Future of Tangible User Interfaces Lars Erik Holmquist, Northumbria University  The notion of tangible user interfaces (TUI) was introduced by Hiroshi Ishii and Brygg Ullmer of the MIT Media Lab at the ACM CHI conference in 1997 [1]. The fundamental idea is that users should be able to “grasp and manipulate bits in the center of users’ attention by coupling the bits with everyday physical objects and architectural surfaces” [1]. TUI has been highly influential in the HCI research community, and to date the 1997 paper alone has been cited almost 5,000 times. Special issues of prominent journals have been devoted to tangibles, and there is an annual ACM conference on the topic: Tangible, Embedded, and Embodied Interaction (TEI). At CHI 2019, Hiroshi Ishii received the prestigious ACM CHI Lifetime Research Award for his work on tangibles, firmly establishing it as one of the cornerstones of modern HCI research. And, speaking personally, that original tangible bits talk at CHI ’97 (also my first CHI conference!) blew me away. To this day, it fundamentally influences how I think about user interfaces and interaction design.   That said, the real-world impact of TUI is still subject to some debate. Several products and interfaces have been launched that incorporate tangible interaction, but industry adoption is not as prominent as the research volume would indicate. In 2010, Gartner’s Emerging Technologies Hype Cycle listed tangible user interfaces at the Technology Trigger stage, which denotes promising new technologies where mainstream adoption is expected in “more than 10 years” [2]. TUI has not appeared on the chart since, whereas many other technologies listed as emerging at the same time in 2010, such as autonomous vehicles, social analytics, 3D printing, and speech-to-speech translation, have already become a commercial reality. And indeed, compared with other forms of interaction that have been commercialized in the past two decades, such as touchscreens and voice assistants, the amount of successful tangible interfaces is comparatively small. There could be multiple reasons, including the relatively high cost of producing bespoke tangible devices, the lack of general-purpose interaction paradigms, or even simply not enough marketing to the consumer market.  TUI Themes At the CHI 2019 conference in Glasgow, I co-organized a panel to discuss the past, present, and future of tangible user interfaces. In addition to Hiroshi Ishii himself, the panelists included Rafael “Tico” Ballagas of HP Labs, Kimiko Ryokai of UC Berkeley, Haiyan Zhang of Microsoft Research, and Oren Zuckerman of miLAB [3]. Representing a variety of viewpoints from academia and industry, we wanted to engage with the proven success as well as the untapped opportunities of TUI, and also understand what the potential impediments to its adoption might be.   A number of themes emerged during the panel that we argued would have unique implications for tangible interaction. It would have been impossible even for the great minds assembled (both on stage and in the audience) to give any definitive answers to these questions, but having an open dialogue helped crystalize some of the most important issues for TUI going forward. The following reflects a synthesis of the some of the main panel themes and my own thoughts on the subject:  Users. Who is the recipient or customer of future tangible user interfaces, and what settings will they be in? When TUI was introduced, personal computers were still limited mostly to desktops. Even laptops were comparatively rare, and WiFi in public places was basically unheard of. With the explosion of the mobile consumer Internet, and aided by portable devices such as smartphones and tablets, computer use has thoroughly dissipated out into the world—everything from social media to reading, watching movies, wayfinding, and shopping. To find a place for tangibles in this wider world of computer use—that is already per definition beyond the desktop—it will be important to create tangible experiences that engage people in their personal, social, and emotional lives.  Embodiment. How do we then engage all of the six human senses—including proprioception, the sense of one’s own body? The senses of touch, smell, and other modalities are still undervalued in commercial user interface design, which has largely focused on the visual, more recently including voice. Even paradigms such as augmented and virtual reality still put the emphasis on pixels before the real world. By its nature, TUI thus has a unique opportunity to engage the senses and provide a more embodied experience of digital systems, which will help interactive systems become more naturally integrated into human activities.  Materiality. Following on from embodiment, TUI potentially offers a much richer materiality than that of graphical interfaces. TUIs can be made from substances like plastic, wood, glass, metal, or even dynamic materials such as liquids or living organisms like plants—each offering a completely different tactile and sensory experience. They can also be hot or cold to the touch, light or heavy, and even moist and squishy versus dry and brittle. In other words, TUI offers a whole new world of materials to build interactive experiences, truly beyond pixels. But realizing this potential will require equally new forms of input, output, and sensor technology to truly take advantage of such non-digital materials.  Process. How do we invent, construct, and, perhaps most important, evaluate tangible experiences? While Internet companies can quickly produce new user experiences in apps or on the Web, and perform A-B testing with thousands of users literally at the flick of a switch, producing and studying tangible devices is much more laborious. Constructing a brand-new TUI requires much more in the way of engineering than just launching yet another interactive button on the Internet; consequently, neither design principles nor usability standards for TUIs have had the same chance to emerge. One way forward would be a move toward more standardized tangible UI components from which physical interfaces can be built quickly, just like GUI builders have made it easier to quickly produce graphical interfaces.  Scalability. How can we make tangible user interfaces—or embodied interaction in general—scale economically and realistically? After all, pixels are cheap, but atoms are comparatively expensive. This means that the cost for a tangible user interface may not equal the perceived benefit, and a screen-based one may win out even if the tangible user experience is arguably better. However, users are prepared to pay large sums to own the latest phone or the newest smartwatch, indicating that if the value proposition is right, people will pay.   Sustainability. From an environmental perspective, the tech industry is rapidly becoming a worldwide problem, with greatly increasing amounts of electronic waste being produced every year. To just take one example, the battery life in wireless earbuds such as Apple’s AirPods limits them to only a few years’ practical use, and replacing a worn-out battery is not cost effective. Thus, the whole item most likely gets thrown away, dangerous lithium-ion cells and all. The same goes for mobile phones and other battery-powered devices. This will obviously have a drastic impact on the environment. Yet TUI by its very nature implies the production and distribution of more physical devices. Would a world where TUI is widely popular see an even greater increase in electronic junk, or can we create interfaces using sustainable and recyclable components and materials?  Impact. What creates the need and push for tangible user interfaces? In many cases, the research community will come up with a novel vision or technology paradigm, then work on it for decades to experiment, validate, and come up with solutions, before industry eventually picks it up. This has been the case, for instance, with the notion of ubiquitous computing, which arguably was a major influence both on the Internet of Things and the consumer smartphone and tablet market [4]. Another example is virtual and augmented reality, both of which are finally seeing mainstream adoption after being well known and explored in the research community for decades. Given its undeniable influence in the academic world, TUI should be well positioned to have a similar impact. On the other hand, some would argue that research also has value for its own sake and does not have to have any direct commercial impact to be valuable. Which takes us to the last point…  Inspiration. There is one aspect of TUI research that is perhaps more important than all else: the steady creation of new ideas and new impulses in human-computer interaction. Ishii himself once explained his creative approach to research as putting on new glasses that let you see the world in a different way. If the research community can inspire young students to see the world differently and try new things, this in itself has a great value.   What’s Next? All things said and done, where will TUI go in the immediate future? One way forward that was suggested during the panel is the creation of bespoke tangible interfaces for very specific situations. These would be interfaces to computer systems that do only one thing but do them very well. One striking example in recent years is Microsoft’s Surface Dial, introduced in 2017. Billed as a “completely new way to interact with technology and create in the most natural, immersive way,” the Dial is a repurposable tangible interface that can be adapted for a variety of functions. It can be configured as a “physical shortcut” to perform a single specific function, for instance scrolling, which is performed very often yet requires cumbersome precision maneuvers when using a GUI (and in fact, scrolling has already been incorporated in some mouse models as a dedicated separate control). However, there are perhaps other generic functions that could potentially be “tangiblized,” so to speak, to unload specific burdens when operating a complex computer system.  An example of this bespoke approach was recently demonstrated by Sam Nemeth, a Ph.D. student at Northumbria University. He has been working on the task of constructing a tangible controller for an embodied telepresence system—essentially, a two-way video communication system to be used in a building reception. The concept is that a receptionist should be able to virtually greet and help visitors at a reception in a different building, for instance covering for regular staff when they need to step away from the reception to solve some urgent issue, or simply go on break (Figure 1).  A novel feature of Nemeth’s system is that the operator can physically move the screen of the remote system. This means that the screen—located in another building—can be turned to see different parts of the lobby or to better interact with a visitor. However, while a standard controller for such a system might be a joystick or a keypad, in early studies Nemeth found that this was not suitable for the staff that would be using the system.  Instead, he constructed a highly specialized interface, where all functions were combined and immediately available (Figure 2). A single box includes an on/off switch for the live video, a microphone for communicating with visitors, and, finally, a dedicated controller for the motion of the remote screen. This takes the form of a half-sphere, made of polished wood. The controller is limited so that it can only turn the screen left or right, avoiding the risk of losing control of the remote device. It also provides a one-to-one correspondence between the controller in the operator’s hand and the position of the remote screen. Finally, the natural material gives a pleasant analog feel that makes the system more naturally incorporated in the everyday task.  Another bespoke tangible interface can be found at the visitors’ end: an old-fashioned hotel bell that is used to call the attention of the remote operator (Figure 3). This has also proven to be an intuitive and simple way of performing a single, dedicated function. At its core, this is simply a switch connected with USB and hidden inside the bell, but visitors immediately grasp the function, since it exactly corresponds to the usage you would expect —while seamlessly interfacing with a digital system.   This bespoke tangible interface solution has become a substantial success. The staff at the two buildings where the system is installed are using it literally every workday to cover for each other, to greet and help visitors remotely, and even to just keep an eye on the other reception, letting the screen act as a remote window in the background. We believe that a large factor in the success of the system is the naturalness of the interaction, which does not force users to learn an unfamiliar generic controller, but rather allows them to use one that is exactly right for the task.  Thus, while there are many questions still to be answered on the future of tangible interfaces—a few of which I have indicated above—there are also clear paths to success. By solving well-defined problems in a way that has a clear benefit to users, tangible user interfaces can show their value and fully enter the mainstream.  Endnotes 1. Ishii, H. and Ullmer, B. Tangible bits: Towards seamless interfaces between people, bits and atoms. Proc. of the ACM SIGCHI Conference on Human Factors in Computing Systems.  ACM, New York, 1997, 234–241; http://dx.doi.org/10.1145/258549.258715  2. Gartner Research. Hype Cycle for Emerging Technologies, 2010.  Aug. 2, 2010; https://www.gartner.com/en/documents/1414917/hype-cycle-for-emerging-technologies-2010  3. Holmquist, L.E., Zuckerman, O., Ballagas, R., Ishii, H., Ryokai, K., and Zhang, H. The future of tangible user interfaces. Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, New York, 2019, Paper panel02; https://doi.org/10.1145/3290607.3311741  4. Weiser, M. The computer for the 21st century. SIGMOBILE Mob. Comput. Commun. Rev. 3, 3 (July 1999), 3–11; http://dx.doi.org/10.1145/329124.329126  Lars Erik Holmquist is Professor of Innovation at Northumbria University. He is an internationally leading researcher in human-computer interaction, interaction design, and ubiquitous computing. His book Grounded Innovation: Strategies for Creating Digital Products (Morgan Kaufmann, 2012) provides a practical guide to the design-driven innovation process, with many examples drawn from his own research and elsewhere. Lars.holmquist@northumbria.ac.uk Insights  Tangible user interfaces (TUI) have a rich history in academia and is one of the cornerstones of modern HCI research  TUI has many benefits and opportunities, but there are also impediments to its mainstream adoption  One way forward could be bespoke, specialized tangible interfaces that perform a specific task really well   Images  Figure 1. A visitor is greeted by the virtual receptionist.  Figure 2. A bespoke tangible interface controls the remote screen.  Figure 3. An old-fashioned hotel bell is used to call attention.  

The future of tangible user interfaces

http://nrl.northumbria.ac.uk/40763/1/Holmquist_Review%20final.pdf

The future of tangible user interfaces

Abstract

Similar works

Full text

Available Versions

Northumbria Research Link