THE UNIVERSAL MEDIA BOOK

We explore the integration of projected imagery with a physical book that acts as a tangible interface to multimedia data. Using a camera and projector pair, a tracking framework is presented wherein the 3D position of planar pages are monitored as they are turned back and forth by a user, and data is correctly warped and projected onto each page at interactive rates to provide the user with an intuitive mixed-reality experience. The book pages are blank, so traditional camera-based approaches to tracking physical features on the display surface do not apply. Instead, in each frame, feature points are independently extracted from the camera and projector images, and matched to recover the geometry of the pages in motion. The book can be loaded with multimedia content, including images and videos. In addition, volumetric datasets can be explored by removing a page from the book and using it as a tool to navigate through a virtual 3D volume

RealitySketch: Embedding Responsive Graphics and Visualizations in AR through Dynamic Sketching

We present RealitySketch, an augmented reality interface for sketching interactive graphics and visualizations. In recent years, an increasing number of AR sketching tools enable users to draw and embed sketches in the real world. However, with the current tools, sketched contents are inherently static, floating in mid air without responding to the real world. This paper introduces a new way to embed dynamic and responsive graphics in the real world. In RealitySketch, the user draws graphical elements on a mobile AR screen and binds them with physical objects in real-time and improvisational ways, so that the sketched elements dynamically move with the corresponding physical motion. The user can also quickly visualize and analyze real-world phenomena through responsive graph plots or interactive visualizations. This paper contributes to a set of interaction techniques that enable capturing, parameterizing, and visualizing real-world motion without pre-defined programs and configurations. Finally, we demonstrate our tool with several application scenarios, including physics education, sports training, and in-situ tangible interfaces.Comment: UIST 202

Supporting Tangible User Interaction with Integrated Paper and Electronic Document Management Systems

Although electronic technology has had a significant impact on the way that offices manage documents, in most cases electronic documents have not completely replaced paper documents. As a result, many present-day offices use a combination of paper and electronic documents in their normal work-flow. The problem with this, however, is that it results in information and document management becoming fragmented between the paper and electronic forms. There is, therefore, a need to provide better integration of the management of paper and electronic documents in order to reduce this fragmentation and, where possible, bring the advantages of electronic document management to paper documents. Previous research has investigated methods of incorporating management and tracking of paper documents into electronic document management systems. However, better integration between paper and electronic document management is still needed, and could potentially be achieved by augmenting elements of the physical document management system with electronic circuitry so they can support tangible user interaction with the integrated document management system. Therefore, the aim of this thesis has been to investigate this. The approach that was taken began by identifying the requirements of such integrated systems by studying the document management needs of a number of real-world offices. This was followed by the development of a series of prototype systems designed to function as tangible user interfaces to the integrated document management system. These prototypes were then evaluated against the identified requirements, and a user study was conducted in order to evaluate their usability. The results of these evaluations demonstrate that it is possible to develop systems systems that can utilise tangible user interaction techniques to enhance the integration of paper and electronic document management, and thus better bridge the divide between the physical and virtual worlds of documents

Design and Analysis of Security Schemes for Low-cost RFID Systems

With the remarkable progress in microelectronics and low-power semiconductor technologies, Radio Frequency IDentification technology (RFID) has moved from obscurity into mainstream applications, which essentially provides an indispensable foundation to realize ubiquitous computing and machine perception. However, the catching and exclusive characteristics of RFID systems introduce growing security and privacy concerns. To address these issues are particularly challenging for low-cost RFID systems, where tags are extremely constrained in resources, power and cost. The primary reasons are: (1) the security requirements of low-cost RFID systems are even more rigorous due to large operation range and mass deployment; and (2) the passive tags' modest capabilities and the necessity to keep their prices low present a novel problem that goes beyond the well-studied problems of traditional cryptography. This thesis presents our research results on the design and the analysis of security schemes for low-cost RFID systems. Motivated by the recent attention on exploiting physical layer resources in the design of security schemes, we investigate how to solve the eavesdropping, modification and one particular type of relay attacks toward the tag-to-reader communication in passive RFID systems without requiring lightweight ciphers. To this end, we propose a novel physical layer scheme, called Backscatter modulation- and Uncoordinated frequency hopping-assisted Physical Layer Enhancement (BUPLE). The idea behind it is to use the amplitude of the carrier to transmit messages as normal, while to utilize its periodically varied frequency to hide the transmission from the eavesdropper/relayer and to exploit a random sequence modulated to the carrier's phase to defeat malicious modifications. We further improve its eavesdropping resistance through the coding in the physical layer, since BUPLE ensures that the tag-to-eavesdropper channel is strictly noisier than the tag-to-reader channel. Three practical Wiretap Channel Codes (WCCs) for passive tags are then proposed: two of them are constructed from linear error correcting codes, and the other one is constructed from a resilient vector Boolean function. The security and usability of BUPLE in conjunction with WCCs are further confirmed by our proof-of-concept implementation and testing. Eavesdropping the communication between a legitimate reader and a victim tag to obtain raw data is a basic tool for the adversary. However, given the fundamentality of eavesdropping attacks, there are limited prior work investigating its intension and extension for passive RFID systems. To this end, we firstly identified a brand-new attack, working at physical layer, against backscattered RFID communications, called unidirectional active eavesdropping, which defeats the customary impression that eavesdropping is a ``passive" attack. To launch this attack, the adversary transmits an un-modulated carrier (called blank carrier) at a certain frequency while a valid reader and a tag interacts at another frequency channel. Once the tag modulates the amplitude of reader's signal, it causes fluctuations on the blank carrier as well. By carefully examining the amplitude of the backscattered versions of the blank carrier and the reader's carrier, the adversary could intercept the ongoing reader-tag communication with either significantly lower bit error rate or from a significantly greater distance away. Our concept is demonstrated and empirically analyzed towards a popular low-cost RFID system, i.e., EPC Gen2. Although active eavesdropping in general is not trivial to be prohibited, for a particular type of active eavesdropper, namely a greedy proactive eavesdropper, we propose a simple countermeasure without introducing extra cost to current RFID systems. The needs of cryptographic primitives on constraint devices keep increasing with the growing pervasiveness of these devices. One recent design of the lightweight block cipher is Hummingbird-2. We study its cryptographic strength under a novel technique we developed, called Differential Sequence Attack (DSA), and present the first cryptanalytic result on this cipher. In particular, our full attack can be divided into two phases: preparation phase and key recovery phase. During the key recovery phase, we exploit the fact that the differential sequence for the last round of Hummingbird-2 can be retrieved by querying the full cipher, due to which, the search space of the secret key can be significantly reduced. Thus, by attacking the encryption (decryption resp.) of Hummingbird-2, our algorithm recovers 36-bit (another 28-bit resp.) out of 128-bit key with $2^{68}$ ($2^{60}$ resp.) time complexity if particular differential conditions of the internal states and of the keys at one round can be imposed. Additionally, the rest 64-bit of the key can be exhaustively searched and the overall time complexity is dominated by $2^{68}$. During the preparation phase, by investing $2^{81}$ effort in time, the adversary is able to create the differential conditions required in the key recovery phase with at least 0.5 probability. As an additional effort, we examine the cryptanalytic strength of another lightweight candidate known as A2U2, which is the most lightweight cryptographic primitive proposed so far for low-cost tags. Our chosen-plaintext-attack fully breaks this cipher by recovering its secret key with only querying the encryption twice on the victim tag and solving 32 sparse systems of linear equations (where each system has 56 unknowns and around 28 unknowns can be directly obtained without computation) in the worst case, which takes around 0.16 second on a Thinkpad T410 laptop

An inertial motion capture framework for constructing body sensor networks

Motion capture is the process of measuring and subsequently reconstructing the movement of an animated object or being in virtual space. Virtual reconstructions of human motion play an important role in numerous application areas such as animation, medical science, ergonomics, etc. While optical motion capture systems are the industry standard, inertial body sensor networks are becoming viable alternatives due to portability, practicality and cost. This thesis presents an innovative inertial motion capture framework for constructing body sensor networks through software environments, smartphones and web technologies. The first component of the framework is a unique inertial motion capture software environment aimed at providing an improved experimentation environment, accompanied by programming scaffolding and a driver development kit, for users interested in studying or engineering body sensor networks. The software environment provides a bespoke 3D engine for kinematic motion visualisations and a set of tools for hardware integration. The software environment is used to develop the hardware behind a prototype motion capture suit focused on low-power consumption and hardware-centricity. Additional inertial measurement units, which are available commercially, are also integrated to demonstrate the functionality the software environment while providing the framework with additional sources for motion data. The smartphone is the most ubiquitous computing technology and its worldwide uptake has prompted many advances in wearable inertial sensing technologies. Smartphones contain gyroscopes, accelerometers and magnetometers, a combination of sensors that is commonly found in inertial measurement units. This thesis presents a mobile application that investigates whether the smartphone is capable of inertial motion capture by constructing a novel omnidirectional body sensor network. This thesis proposes a novel use for web technologies through the development of the Motion Cloud, a repository and gateway for inertial data. Web technologies have the potential to replace motion capture file formats with online repositories and to set a new standard for how motion data is stored. From a single inertial measurement unit to a more complex body sensor network, the proposed architecture is extendable and facilitates the integration of any inertial hardware configuration. The Motion Cloud’s data can be accessed through an application-programming interface or through a web portal that provides users with the functionality for visualising and exporting the motion data

Content creation for seamless augmented experiences with projection mapping

This dissertation explores systems and methods for creating projection mapping content that seamlessly merges virtual and physical. Most virtual reality and augmented reality technologies rely on screens for display and interaction, where a mobile device or head mounted display mediates the user's experience. In contrast, projection mapping uses off-the-shelf video projectors to augment the appearance of physical objects, and with projection mapping there is no screen to mediate the experience. The physical world simply becomes the display. Projection mapping can provide users with a seamless augmented experience, where virtual and physical become indistinguishable in an apparently unmediated way. Projection mapping is an old concept dating to Disney's 1969 Haunted Mansion. The core technical foundations were laid back in 1999 with UNC's Office of the Future and Shader Lamps projects. Since then, projectors have gotten brighter, higher resolution, and drastically decreased in price. Yet projection mapping has not crossed the chasm into mainstream use. The largest remaining challenge for projection mapping is that content creation is very difficult and time consuming. Content for projection mapping is still created via a tedious manual process by warping a 2D video file onto a 3D physical object using existing tools (e.g. Adobe Photoshop) which are not made for defining animated interactive effects on 3D object surfaces. With existing tools, content must be created for each specific display object, and cannot be re-used across experiences. For each object the artist wants to animate, the artist must manually create a custom texture for that specific object, and warp the texture to the physical object. This limits projection mapped experiences to controlled environments and static scenes. If the artist wants to project onto a different object from the original, they must start from scratch creating custom content for that object. This manual content creation process is time consuming, expensive and doesn't scale. This thesis explores new methods for creating projection mapping content. Our goal is to make projection mapping easier, cheaper and more scalable. We explore methods for adaptive projection mapping, which enables artists to create content once, and that content adapts based on the color and geometry of the display surface. Content can be created once, and re-used on any surface. This thesis is composed of three proof-of-concept prototypes, exploring new methods for content creation for projection mapping. IllumiRoom expands video game content beyond the television screen and into the physical world using a standard video projector to surround a television with projected light. IllumiRoom works in any living room, the projected content dynamically adapts based on the color and geometry of the room. RoomAlive expands on this idea, using multiple projectors to cover an entire living room in input/output pixels and dynamically adapts gaming experiences to fill an entire room. Finally, Projectibles focuses on the physical aspect of projection mapping. Projectibles optimizes the display surface color to increase the contrast and resolution of the overall experience, enabling artists to design the physical object along with the virtual content. The proof-of-concept prototypes presented in this thesis are aimed at the not-to-distant future. The projects in this thesis are not theoretical concepts, but fully working prototype systems that demonstrate the practicality of projection mapping to create immersive experiences. It is the sincere hope of the author that these experiences quickly move of the lab and into the real world

Abstraction of representation in live theater

Thesis (S.M.)--Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2009.Cataloged from PDF version of thesis.Includes bibliographical references (p. 151-158).Early in Tod Machover's opera Death and the Powers, the main character, Simon Powers, is subsumed into a technological environment of his own creation. The theatrical set comes alive in the form of robotic, visual, and sonic elements that allow the actor to extend his range and influence across the stage in unique and dynamic ways. The environment must compellingly assume the behavior and expression of the absent Simon. This thesis presents a new approach called Disembodied Performance that adapts ideas from affective psychology, cognitive science, and the theatrical tradition to create a framework for thinking about the translation of stage presence. An implementation of a system informed by this methodology is demonstrated. In order to distill the essence of this character, we recover performance parameters in real-time from physiological sensors, voice, and vision systems. This system allows the offstage actor to express emotion and interact with others onstage. The Disembodied Performance approach takes a new direction in augmented performance by employing a nonrepresentational abstraction of a human presence that fully translates a character into an environment. The technique and theory presented also have broad-reaching applications outside of theater for personal expression, telepresence, and storytelling.Peter Alexander Torpey.S.M

Photodiodes and Image Sensors on Mechanically Flexible Ultra-Thin Silicon Chips-in-Foil

CMOS-Bildsensoren haben in den letzten zwei Jahrzehnten enorme technologische Fortschritte erfahren und sich als eine wettbewerbsfähige Alternative gegenüber CCDBildsensoren auf dem Markt etabliert. Reduziert man die Chipdicke von CMOSBildsensoren von normal 725 μm auf ≤ 30 μm, erhält man mechanisch flexible Bildaufnehmer. Gewölbte CMOS-Bildsensoren würden für die optische Wahrnehmung völlig neue Möglichkeiten eröffnen (wie z. B. bei Insektenaugen). Betrachtet man die auf dem Chip integrierten Bauelemente und Schaltungen unter mechanischen Spannungen, stellt man fest, dass ihre elektrischen und optoelektronischen Eigenschaften von der ausgeübten mechanischen Spannung beeinflusst werden. Für den technischen Einsatz ist eine vom mechanischen Zustand des Bildsensors unbeinflusste Funktion erforderlich. Der Einfluss von mechanischer Spannung auf die Bauelemente- und Schaltungs-Charakteristiken und seine Minimierung bzw. Kompensation sind daher von besonderem Interesse. In dieser Arbeit wurden die optischen und elektrischen Eigenschaften von passiven und aktiven Bauelementen, sowie integrierten Schaltungen auf monokristallinen gedünnten flexiblen Siliziumchips unter mechanischen Spannungen untersucht. Der Einfluss von mechanischen Spannungen auf optische Eigenschaften (spektrale Lichtempfindlichkeit, Dunkelstrom und elektronisches Rauschen) einzelner p-n-Übergang basierter Photodioden und Bildsensorarrays auf (100)-Siliziumwafern wurde theoretisch modelliert und experimentell charakterisiert. Weiterhin wurden die elektrischen Eigenschaften (Ladungsträgerbeweglichkeit, Schwellenspannung, 1/f Rauschen) von MOSFeldeffekttransistoren in Bezug auf mechanischen Spannungen charakterisiert und ihre Abhängigkeit von der Orientierung zur Kristallorientierung des Substrats untersucht. Integrierte Schaltungen, wie Bandgap-Referenzspannungsquellen, Operationsverstärker und SC-basierte Schaltungen wurden unter mechanischen Spannungen theoretisch betrachtet, entworfen, gefertigt und experimentell charakterisiert. Mit Hilfe des in dieser Arbeit vorgeschlagenen und eingesetzten Simulationskonzeptes, ist die Schaltungssimulation der obengenannten Abhängigkeiten möglich. Dadurch hat der Schaltungsentwickler die Möglichkeit Schaltungskonzepte zur Kompensation oder Minimierung der von der mechanischen Spannung hervorgerufenen Einflüsse zu simulieren. In dieser Hinsicht werden Schaltungskonzepte und Design-Regeln präsentiert, die den Einfluss von mechanischen Spannungen auf Bildsensorchips berücksichtigen und minimieren. Im Rahmen dieser Arbeit wurde darüber hinaus ein mechanisch flexibler Bildsensorchip entworfen, simuliert und gefertigt, dessen Betrieb unabhängig von der ausgeübten mechanischen Spannung ist. Der ultra-dünne 20 μm Bildsensorchip ist geeignet auf zylindrisch gewölbte Oberflächen aufgebracht zu werden und erlaubt die Aufnahme raumrichtungsselektiver optischer Informationen im Sinne eines Panoramablicks.CMOS image sensors (CIS) have experienced the last two decades tremendous technological advances rendering them a viable alternative to charged couple devices (CCDs) not only in high volume applications but also in applications which require high spatial and temporal resolution, high dynamic range, low noise or high sensitivity levels. CISs are employed due to their increased chip thickness (ca. 750 μm) solely in the traditional planar image acquisition. If the chip thickness could be reduced down to or less than 30 μm, the silicon chips would become mechanically flexible. Such flexible CISs could substantially extend the application spectrum of image sensors in non-conventional imaging systems (e.g. imitating insect vision). However, the on-chip integrated devices and circuits exhibit stress-induced changes on their electrical and optoelectronic characteristics. Since a stress independent operation is striven, the minimization or compensation of the influence of mechanical stress on the characteristics of devices and circuits is of great interest. In this work optical and electrical properties of passive and active devices as well as integrated circuits on ultra-thin monolithic flexible silicon chips have been investigated under the application of mechanical stress. The influence of mechanical stress on the optical characteristics (spectral sensitivity, dark current and electronic noise) of p-n junction based photodiodes and image sensor chips on (100)-silicon wafers have been theoretically modeled and experimentally characterized. Moreover, the electrical characteristics (carrier mobility, threshold voltage and 1/f noise) of mechanically strained MOS field-effect transistors and their dependence on the channel orientation on the substrate have been investigated. Integrated circuits such as bandgap reference voltage sources, operational amplifiers and switched capacitor (SC) based circuits have been theoretically treated, designed, fabricated and experimentally characterized. Within this framework a simulation technique has been proposed and deployed, which allows the simulation of the above mentioned stress dependence on device and circuit level. The analog circuit designer can employ the simulation technique toward the proposal of circuit topologies or techniques, which minimize or compensate the strain-induced changes on the circuit operation. In this direction, circuit concepts and design rules are proposed, which minimize the influence of mechanical stress on flexible CIS chips. Within the scope of this work, a mechanically flexible CMOS image sensor chip has been designed, simulated and fabricated, which operation is stress independent. The developed ultra-thin 20 μm CIS chip can be wrapped around a cylindrically curved surface and thus record panoramic optical information

Freeform 3D interactions in everyday environments

PhD ThesisPersonal computing is continuously moving away from traditional input using mouse and keyboard, as new input technologies emerge. Recently, natural user interfaces (NUI) have led to interactive systems that are inspired by our physical interactions in the real-world, and focus on enabling dexterous freehand input in 2D or 3D. Another recent trend is Augmented Reality (AR), which follows a similar goal to further reduce the gap between the real and the virtual, but predominately focuses on output, by overlaying virtual information onto a tracked real-world 3D scene. Whilst AR and NUI technologies have been developed for both immersive 3D output as well as seamless 3D input, these have mostly been looked at separately. NUI focuses on sensing the user and enabling new forms of input; AR traditionally focuses on capturing the environment around us and enabling new forms of output that are registered to the real world. The output of NUI systems is mainly presented on a 2D display, while the input technologies for AR experiences, such as data gloves and body-worn motion trackers are often uncomfortable and restricting when interacting in the real world. NUI and AR can be seen as very complimentary, and bringing these two fields together can lead to new user experiences that radically change the way we interact with our everyday environments. The aim of this thesis is to enable real-time, low latency, dexterous input and immersive output without heavily instrumenting the user. The main challenge is to retain and to meaningfully combine the positive qualities that are attributed to both NUI and AR systems. I review work in the intersecting research fields of AR and NUI, and explore freehand 3D interactions with varying degrees of expressiveness, directness and mobility in various physical settings. There a number of technical challenges that arise when designing a mixed NUI/AR system, which I will address is this work: What can we capture, and how? How do we represent the real in the virtual? And how do we physically couple input and output? This is achieved by designing new systems, algorithms, and user experiences that explore the combination of AR and NUI