PhD ThesisPersonal computing is continuously moving away from traditional input using
mouse and keyboard, as new input technologies emerge. Recently, natural user interfaces
(NUI) have led to interactive systems that are inspired by our physical interactions
in the real-world, and focus on enabling dexterous freehand input in 2D or 3D. Another
recent trend is Augmented Reality (AR), which follows a similar goal to further reduce
the gap between the real and the virtual, but predominately focuses on output, by overlaying
virtual information onto a tracked real-world 3D scene.
Whilst AR and NUI technologies have been developed for both immersive 3D output as
well as seamless 3D input, these have mostly been looked at separately. NUI focuses on
sensing the user and enabling new forms of input; AR traditionally focuses on capturing
the environment around us and enabling new forms of output that are registered to the
real world. The output of NUI systems is mainly presented on a 2D display, while
the input technologies for AR experiences, such as data gloves and body-worn motion
trackers are often uncomfortable and restricting when interacting in the real world.
NUI and AR can be seen as very complimentary, and bringing these two fields together
can lead to new user experiences that radically change the way we interact with
our everyday environments. The aim of this thesis is to enable real-time, low latency,
dexterous input and immersive output without heavily instrumenting the user. The
main challenge is to retain and to meaningfully combine the positive qualities that are
attributed to both NUI and AR systems.
I review work in the intersecting research fields of AR and NUI, and explore freehand
3D interactions with varying degrees of expressiveness, directness and mobility
in various physical settings. There a number of technical challenges that arise when
designing a mixed NUI/AR system, which I will address is this work: What can we capture,
and how? How do we represent the real in the virtual? And how do we physically
couple input and output? This is achieved by designing new systems, algorithms, and
user experiences that explore the combination of AR and NUI