Over the past several decades, technological advancements have introduced new modes of communication
with the computers, introducing a shift from traditional mouse and keyboard interfaces.
While touch based interactions are abundantly being used today, latest developments in computer
vision, body tracking stereo cameras, and augmented and virtual reality have now enabled communicating
with the computers using spatial input in the physical 3D space. These techniques are now
being integrated into several design critical tasks like sketching, modeling, etc. through sophisticated
methodologies and use of specialized instrumented devices. One of the prime challenges in
design research is to make this spatial interaction with the computer as intuitive as possible for the
users.
Drawing curves in mid-air with fingers, is a fundamental task with applications to 3D sketching,
geometric modeling, handwriting recognition, and authentication. Sketching in general, is a
crucial mode for effective idea communication between designers. Mid-air curve input is typically
accomplished through instrumented controllers, specific hand postures, or pre-defined hand gestures,
in presence of depth and motion sensing cameras. The user may use any of these modalities
to express the intention to start or stop sketching. However, apart from suffering with issues like
lack of robustness, the use of such gestures, specific postures, or the necessity of instrumented
controllers for design specific tasks further result in an additional cognitive load on the user.
To address the problems associated with different mid-air curve input modalities, the presented
research discusses the design, development, and evaluation of data driven models for intent recognition
in non-instrumented, gesture-free, bare-hand mid-air drawing tasks.
The research is motivated by a behavioral study that demonstrates the need for such an approach
due to the lack of robustness and intuitiveness while using hand postures and instrumented
devices. The main objective is to study how users move during mid-air sketching, develop qualitative
insights regarding such movements, and consequently implement a computational approach to
determine when the user intends to draw in mid-air without the use of an explicit mechanism (such
as an instrumented controller or a specified hand-posture). By recording the user’s hand trajectory,
the idea is to simply classify this point as either hover or stroke. The resulting model allows for
the classification of points on the user’s spatial trajectory.
Drawing inspiration from the way users sketch in mid-air, this research first specifies the necessity
for an alternate approach for processing bare hand mid-air curves in a continuous fashion.
Further, this research presents a novel drawing intent recognition work flow for every recorded
drawing point, using three different approaches. We begin with recording mid-air drawing data
and developing a classification model based on the extracted geometric properties of the recorded
data. The main goal behind developing this model is to identify drawing intent from critical geometric
and temporal features. In the second approach, we explore the variations in prediction
quality of the model by improving the dimensionality of data used as mid-air curve input. Finally,
in the third approach, we seek to understand the drawing intention from mid-air curves using
sophisticated dimensionality reduction neural networks such as autoencoders. Finally, the broad
level implications of this research are discussed, with potential development areas in the design
and research of mid-air interactions
