To Draw or Not to Draw: Recognizing Stroke-Hover Intent in Gesture-Free Bare-Hand Mid-Air Drawing Tasks

Over the past several decades, technological advancements have introduced new modes of communication with the computers, introducing a shift from traditional mouse and keyboard interfaces. While touch based interactions are abundantly being used today, latest developments in computer vision, body tracking stereo cameras, and augmented and virtual reality have now enabled communicating with the computers using spatial input in the physical 3D space. These techniques are now being integrated into several design critical tasks like sketching, modeling, etc. through sophisticated methodologies and use of specialized instrumented devices. One of the prime challenges in design research is to make this spatial interaction with the computer as intuitive as possible for the users. Drawing curves in mid-air with fingers, is a fundamental task with applications to 3D sketching, geometric modeling, handwriting recognition, and authentication. Sketching in general, is a crucial mode for effective idea communication between designers. Mid-air curve input is typically accomplished through instrumented controllers, specific hand postures, or pre-defined hand gestures, in presence of depth and motion sensing cameras. The user may use any of these modalities to express the intention to start or stop sketching. However, apart from suffering with issues like lack of robustness, the use of such gestures, specific postures, or the necessity of instrumented controllers for design specific tasks further result in an additional cognitive load on the user. To address the problems associated with different mid-air curve input modalities, the presented research discusses the design, development, and evaluation of data driven models for intent recognition in non-instrumented, gesture-free, bare-hand mid-air drawing tasks. The research is motivated by a behavioral study that demonstrates the need for such an approach due to the lack of robustness and intuitiveness while using hand postures and instrumented devices. The main objective is to study how users move during mid-air sketching, develop qualitative insights regarding such movements, and consequently implement a computational approach to determine when the user intends to draw in mid-air without the use of an explicit mechanism (such as an instrumented controller or a specified hand-posture). By recording the user’s hand trajectory, the idea is to simply classify this point as either hover or stroke. The resulting model allows for the classification of points on the user’s spatial trajectory. Drawing inspiration from the way users sketch in mid-air, this research first specifies the necessity for an alternate approach for processing bare hand mid-air curves in a continuous fashion. Further, this research presents a novel drawing intent recognition work flow for every recorded drawing point, using three different approaches. We begin with recording mid-air drawing data and developing a classification model based on the extracted geometric properties of the recorded data. The main goal behind developing this model is to identify drawing intent from critical geometric and temporal features. In the second approach, we explore the variations in prediction quality of the model by improving the dimensionality of data used as mid-air curve input. Finally, in the third approach, we seek to understand the drawing intention from mid-air curves using sophisticated dimensionality reduction neural networks such as autoencoders. Finally, the broad level implications of this research are discussed, with potential development areas in the design and research of mid-air interactions

Symbol Emergence in Robotics: A Survey

Humans can learn the use of language through physical interaction with their environment and semiotic communication with other people. It is very important to obtain a computational understanding of how humans can form a symbol system and obtain semiotic skills through their autonomous mental development. Recently, many studies have been conducted on the construction of robotic systems and machine-learning methods that can learn the use of language through embodied multimodal interaction with their environment and other systems. Understanding human social interactions and developing a robot that can smoothly communicate with human users in the long term, requires an understanding of the dynamics of symbol systems and is crucially important. The embodied cognition and social interaction of participants gradually change a symbol system in a constructive manner. In this paper, we introduce a field of research called symbol emergence in robotics (SER). SER is a constructive approach towards an emergent symbol system. The emergent symbol system is socially self-organized through both semiotic communications and physical interactions with autonomous cognitive developmental agents, i.e., humans and developmental robots. Specifically, we describe some state-of-art research topics concerning SER, e.g., multimodal categorization, word discovery, and a double articulation analysis, that enable a robot to obtain words and their embodied meanings from raw sensory--motor information, including visual information, haptic information, auditory information, and acoustic speech signals, in a totally unsupervised manner. Finally, we suggest future directions of research in SER.Comment: submitted to Advanced Robotic

2016 Annual Research Symposium Abstract Book

2016 annual volume of abstracts for science research projects conducted by students at Trinity Colleg

Deriving Motor Primitives Through Action Segmentation

The purpose of the present experiment is to further understand the effect of levels of processing (top-down vs. bottom-up) on the perception of movement kinematics and primitives for grasping actions in order to gain insight into possible primitives used by the mirror system. In the present study, we investigated the potential of identifying such primitives using an action segmentation task. Specifically, we investigated whether or not segmentation was driven primarily by the kinematics of the action, as opposed to high-level top-down information about the action and the object used in the action. Participants in the experiment were shown 12 point-light movies of object-centered hand/arm actions that were either presented in their canonical orientation together with the object in question (top-down condition) or upside down (inverted) without information about the object (bottom-up condition). The results show that (1) despite impaired high-level action recognition for the inverted actions participants were able to reliably segment the actions according to lower-level kinematic variables, (2) segmentation behavior in both groups was significantly related to the kinematic variables of change in direction, velocity, and acceleration of the wrist (thumb and finger tips) for most of the included actions. This indicates that top-down activation of an action representation leads to similar segmentation behavior for hand/arm actions compared to bottom-up, or local, visual processing when performing a fairly unconstrained segmentation task. Motor primitives as parts of more complex actions may therefore be reliably derived through visual segmentation based on movement kinematics

Managerial Segmentation of Service Offerings in Work Commuting, MTI Report WP 12-02

Methodology to efficiently segment markets for public transportation offerings has been introduced and exemplified in an application to an urban travel corridor in which high tech companies predominate. The principal objective has been to introduce and apply multivariate methodology to efficiently identify segments of work commuters and their demographic identifiers. A set of attributes in terms of which service offerings could be defined was derived from background studies and focus groups of work commuters in the county. Adaptive choice conjoint analysis was used to derive the importance weights of these attributes in available service offering to these commuters. A two-stage clustering procedure was then used to explore the grouping of individual’s subsets into homogeneous sub-groups of the sample. These subsets are commonly a basis for differentiation in service offerings that can increase total ridership in public transportation while approximating cost neutrality in service delivery. Recursive partitioning identified interactions between demographic predictors that significantly contributed to the discrimination of segments in demographics. Implementation of the results is discussed

Spatiotemporal Learning of Multivehicle Interaction Patterns in Lane-Change Scenarios

Interpretation of common-yet-challenging interaction scenarios can benefit well-founded decisions for autonomous vehicles. Previous research achieved this using their prior knowledge of specific scenarios with predefined models, limiting their adaptive capabilities. This paper describes a Bayesian nonparametric approach that leverages continuous (i.e., Gaussian processes) and discrete (i.e., Dirichlet processes) stochastic processes to reveal underlying interaction patterns of the ego vehicle with other nearby vehicles. Our model relaxes dependency on the number of surrounding vehicles by developing an acceleration-sensitive velocity field based on Gaussian processes. The experiment results demonstrate that the velocity field can represent the spatial interactions between the ego vehicle and its surroundings. Then, a discrete Bayesian nonparametric model, integrating Dirichlet processes and hidden Markov models, is developed to learn the interaction patterns over the temporal space by segmenting and clustering the sequential interaction data into interpretable granular patterns automatically. We then evaluate our approach in the highway lane-change scenarios using the highD dataset collected from real-world settings. Results demonstrate that our proposed Bayesian nonparametric approach provides an insight into the complicated lane-change interactions of the ego vehicle with multiple surrounding traffic participants based on the interpretable interaction patterns and their transition properties in temporal relationships. Our proposed approach sheds light on efficiently analyzing other kinds of multi-agent interactions, such as vehicle-pedestrian interactions. View the demos via https://youtu.be/z_vf9UHtdAM.Comment: for the supplements, see https://chengyuan-zhang.github.io/Multivehicle-Interaction

Single Trial Decoding of Movement Intentions Using Functional Ultrasound Neuroimaging

Brain-machine interfaces (BMI) are powerful devices for restoring function to people living with paralysis. Leveraging significant advances in neurorecording technology, computational power, and understanding of the underlying neural signals, BMI have enabled severely paralyzed patients to control external devices, such as computers and robotic limbs. However, high-performance BMI currently require highly invasive recording techniques, and are thus only available to niche populations. Here, we show that a minimally invasive neuroimaging approach based on functional ultrasound (fUS) imaging can be used to detect and decode movement intention signals usable for BMI. We trained non-human primates to perform memory-guided movements while using epidural fUS imaging to record changes in cerebral blood volume from the posterior parietal cortex, a brain area important for spatial perception, multisensory integration, and movement planning. Using hemodynamic signals acquired during movement planning, we classified left-cued vs. right-cued movements, establishing the feasibility of ultrasonic BMI. These results demonstrate the ability of fUS-based neural interfaces to take advantage of the excellent spatiotemporal resolution, sensitivity, and field of view of ultrasound without breaching the dura or physically penetrating brain tissue

Summer 2012 Research Symposium Abstract Book

Summer 2012 volume of abstracts for science research projects conducted by Trinity College students

Nägemistaju automaatsete protsesside eksperimentaalne uurimine

Väitekirja elektrooniline versioon ei sisalda publikatsiooneVäitekiri keskendub nägemistaju protsesside eksperimentaalsele uurimisele, mis on suuremal või vähemal määral automaatsed. Uurimistöös on kasutatud erinevaid eksperimentaalseid katseparadigmasid ja katsestiimuleid ning nii käitumuslikke- kui ka ajukuvamismeetodeid. Esimesed kolm empiirilist uurimust käsitlevad liikumisinformatsiooni töötlust, mis on evolutsiooni käigus kujunenud üheks olulisemaks baasprotsessiks nägemistajus. Esmalt huvitas meid, kuidas avastatakse liikuva objekti suunamuutusi, kui samal ajal toimub ka taustal liikumine (Uurimus I). Nägemistaju uurijad on pikka aega arvanud, et liikumist arvutatakse alati mõne välise objekti või tausta suhtes. Meie uurimistulemused ei kinnitanud taolise suhtelise liikumise printsiibi paikapidavust ning toetavad pigem seisukohta, et eesmärkobjekti liikumisinformatsiooni töötlus on automaatne protsess, mis tuvastab silma põhjas toimuvaid nihkeid, ja taustal toimuv seda eriti ei mõjuta. Teise uurimuse tulemused (Uurimus II) näitasid, et nägemissüsteem töötleb väga edukalt ka seda liikumisinformatsiooni, millele vaatleja teadlikult tähelepanu ei pööra. See tähendab, et samal ajal, kui inimene on mõne tähelepanu hõlmava tegevusega ametis, suudab tema aju taustal toimuvaid sündmusi automaatselt registreerida. Igapäevaselt on inimese nägemisväljas alati palju erinevaid objekte, millel on erinevad omadused, mistõttu järgmiseks huvitas meid (Uurimus III), kuidas ühe tunnuse (antud juhul värvimuutuse) töötlemist mõjutab mõne teise tunnusega toimuv (antud juhul liikumiskiiruse) muutus. Näitasime, et objekti liikumine parandas sama objekti värvimuutuse avastamist, mis viitab, et nende kahe omaduse töötlemine ajus ei ole päris eraldiseisev protsess. Samuti tähendab taoline tulemus, et hoolimata ühele tunnusele keskendumisest ei suuda inimene ignoreerida teist tähelepanu tõmbavat tunnust (liikumine), mis viitab taas kord automaatsetele töötlusprotsessidele. Neljas uurimus keskendus emotsionaalsete näoväljenduste töötlusele, kuna need kannavad keskkonnas hakkamasaamiseks vajalikke sotsiaalseid signaale, mistõttu on alust arvata, et nende töötlus on kujunenud suuresti automaatseks protsessiks. Näitasime, et emotsiooni väljendavaid nägusid avastati kiiremini ja kergemini kui neutraalse ilmega nägusid ning et vihane nägu tõmbas rohkem tähelepanu kui rõõmus (Uurimus IV). Väitekirja viimane osa puudutab visuaalset lahknevusnegatiivsust (ingl Visual Mismatch Negativity ehk vMMN), mis näitab aju võimet avastada automaatselt erinevusi enda loodud mudelist ümbritseva keskkonna kohta. Selle automaatse erinevuse avastamise mehhanismi uurimisse andsid oma panuse nii Uurimus II kui Uurimus IV, mis mõlemad pakuvad välja tõendusi vMMN tekkimise kohta eri tingimustel ja katseparadigmades ning ka vajalikke metodoloogilisi täiendusi. Uurimus V on esimene kogu siiani ilmunud temaatilist teadustööd hõlmav ülevaateartikkel ja metaanalüüs visuaalsest lahknevusnegatiivsusest psühhiaatriliste ja neuroloogiliste haiguste korral, mis panustab oluliselt visuaalse lahknevusnegatiivsuse valdkonna arengusse.The research presented and discussed in the thesis is an experimental exploration of processes in visual perception, which all display a considerable amount of automaticity. These processes are targeted from different angles using different experimental paradigms and stimuli, and by measuring both behavioural and brain responses. In the first three empirical studies, the focus is on motion detection that is regarded one of the most basic processes shaped by evolution. Study I investigated how motion information of an object is processed in the presence of background motion. Although it is widely believed that no motion can be perceived without establishing a frame of reference with other objects or motion on the background, our results found no support for relative motion principle. This finding speaks in favour of a simple and automatic process of detecting motion, which is largely insensitive to the surrounding context. Study II shows that the visual system is built to automatically process motion information that is outside of our attentional focus. This means that even if we are concentrating on some task, our brain constantly monitors the surrounding environment. Study III addressed the question of what happens when multiple stimulus qualities (motion and colour) are present and varied, which is the everyday reality of our visual input. We showed that velocity facilitated the detection of colour changes, which suggests that processing motion and colour is not entirely isolated. These results also indicate that it is hard to ignore motion information, and processing it is rather automatically initiated. The fourth empirical study focusses on another example of visual input that is processed in a rather automatic way and carries high survival value – emotional expressions. In Study IV, participants detected emotional facial expressions faster and more easily compared with neutral facial expressions, with a tendency towards more automatic attention to angry faces. In addition, we investigated the emergence of visual mismatch negativity (vMMN) that is one of the most objective and efficient methods for analysing automatic processes in the brain. Study II and Study IV proposed several methodological gains for registering this automatic change-detection mechanism. Study V is an important contribution to the vMMN research field as it is the first comprehensive review and meta-analysis of the vMMN studies in psychiatric and neurological disorders