Understanding egocentric human actions with temporal decision forests

Understanding human actions is a fundamental task in computer vision with a wide range of applications including pervasive health-care, robotics and game control. This thesis focuses on the problem of egocentric action recognition from RGB-D data, wherein the world is viewed through the eyes of the actor whose hands describe the actions. The main contributions of this work are its findings regarding egocentric actions as described by hands in two application scenarios and a proposal of a new technique that is based on temporal decision forests. The thesis first introduces a novel framework to recognise fingertip writing in mid-air in the context of human-computer interaction. This framework detects whether the user is writing and tracks the fingertip over time to generate spatio-temporal trajectories that are recognised by using a Hough forest variant that encourages temporal consistency in prediction. A problem with using such forest approach for action recognition is that the learning of temporal dynamics is limited to hand-crafted temporal features and temporal regression, which may break the temporal continuity and lead to inconsistent predictions. To overcome this limitation, the thesis proposes transition forests. Besides any temporal information that is encoded in the feature space, the forest automatically learns the temporal dynamics during training, and it is exploited in inference in an online and efficient manner achieving state-of-the-art results. The last contribution of this thesis is its introduction of the first RGB-D benchmark to allow for the study of egocentric hand-object actions with both hand and object pose annotations. This study conducts an extensive evaluation of different baselines, state-of-the art approaches and temporal decision forest models using colour, depth and hand pose features. Furthermore, it extends the transition forest model to incorporate data from different modalities and demonstrates the benefit of using hand pose features to recognise egocentric human actions. The thesis concludes by discussing and analysing the contributions and proposing a few ideas for future work.Open Acces

Integrating passive ubiquitous surfaces into human-computer interaction

Mobile technologies enable people to interact with computers ubiquitously. This dissertation investigates how ordinary, ubiquitous surfaces can be integrated into human-computer interaction to extend the interaction space beyond the edge of the display. It turns out that acoustic and tactile features generated during an interaction can be combined to identify input events, the user, and the surface. In addition, it is shown that a heterogeneous distribution of different surfaces is particularly suitable for realizing versatile interaction modalities. However, privacy concerns must be considered when selecting sensors, and context can be crucial in determining whether and what interaction to perform.Mobile Technologien ermöglichen den Menschen eine allgegenwärtige Interaktion mit Computern. Diese Dissertation untersucht, wie gewöhnliche, allgegenwärtige Oberflächen in die Mensch-Computer-Interaktion integriert werden können, um den Interaktionsraum über den Rand des Displays hinaus zu erweitern. Es stellt sich heraus, dass akustische und taktile Merkmale, die während einer Interaktion erzeugt werden, kombiniert werden können, um Eingabeereignisse, den Benutzer und die Oberfläche zu identifizieren. Darüber hinaus wird gezeigt, dass eine heterogene Verteilung verschiedener Oberflächen besonders geeignet ist, um vielfältige Interaktionsmodalitäten zu realisieren. Bei der Auswahl der Sensoren müssen jedoch Datenschutzaspekte berücksichtigt werden, und der Kontext kann entscheidend dafür sein, ob und welche Interaktion durchgeführt werden soll

Drawing, Handwriting Processing Analysis: New Advances and Challenges

International audienceDrawing and handwriting are communicational skills that are fundamental in geopolitical, ideological and technological evolutions of all time. drawingand handwriting are still useful in defining innovative applications in numerous fields. In this regard, researchers have to solve new problems like those related to the manner in which drawing and handwriting become an efficient way to command various connected objects; or to validate graphomotor skills as evident and objective sources of data useful in the study of human beings, their capabilities and their limits from birth to decline

Advanced Parameterisation of Online Handwriting in Writers with Graphomotor Disabilities

Grafomotorick© obt­e (GD) vraznÄ ovlivuj­ kvalitu ivota koln­m vÄkem poÄ­naj­c, kde se vyv­jej­ grafomotorick© schopnosti, a do dchodov©ho vÄku. VÄasn diagnza tÄchto obt­­ a terapeutick zsah maj­ velk vznam k jejich zlepen­. Vzhledem k tomu, e GD souvis­ z v­cermi symptomy v oblasti kinematiky, zkladn­ kinematick© parametry jako rychlost, zrychlen­ a vih prokzaly efektivn­ kvantizaci tÄchto symptom. Objektivn­ vpoÄetn­ syst©m podpory rozhodovn­ pro identifikaci a vyeten­ GD vak nen­ dostupn. A proto je hlavn­m c­lem m© disertaÄn­ prce vzkum pokroÄil© metody parametrizace online p­sma pro analzu GD se speciln­m zamÄen­m na vyuit­ metod zlomkov©ho kalkulu. Tato prce je prvn­, kter experimentuje s vyuit­m derivac­ neceloÄ­seln©ho du (FD) pro analzu GD pomoc­ online p­sma z­skan©ho od pacient s Parkinsonovou nemoc­ a u dÄt­ koln­ho vÄku. Byla navrena a evaluovna nov metoda parametrizace online p­sma zaloena na FD vyuit­m Grnwald-Letnikova p­stupu. Bylo dokzno, e navren metoda vznamnÄ zlepuje diskriminaÄn­ s­lu a deskriptivn­ schopnosti v oblasti Parkinsonick© dysgrafie. StejnÄ tak metoda pozitivnÄ ovlivnila i nejmodernÄj­ techniky v oblasti analzy GD u dÄt­ koln­ho vÄku. Vyvinut parametrizace byla optimalizovna s ohledem na vpoÄetn­ nroÄnost (a o 80 %) a tak© na vyladÄn­ du FD. Ke konci prce byly porovnny v­cer© p­stupy vpoÄtu FD, jmenovitÄ Riemann-Liouvillv, Caputv spoleÄnÄ z Grnwald-Letnikovm p­stupem za Äelem identifikace tÄch nejvhodnÄj­ch pro jednotliv© oblasti analzy GD.Graphomotor disabilities (GD) significantly affect the quality of life beginning from the school-age, when the graphomotor skills are developed, until the elderly age. The timely diagnosis of these difficulties and therapeutic interventions are of great importance. As GD are associated with several symptoms in the field of kinematics, the basic kinematic features such as velocity, acceleration, and jerk were proved to effectively quantify these symptoms. Nevertheless, an objective computerized decision support system for the identification and assessment of GD is still missing. Therefore, the main objective of my dissertation is the research of an advanced online handwriting parametrization utilized in the field of GD analysis, with a special focus on methods based on fractional calculus. This work is the first to experiment with fractional-order derivatives (FD) in the GD analysis by online handwriting of Parkinsonâs disease (PD) patients and school-age children. A new online handwriting parametrization technique based on the Grnwald-Letnikov approach of FD has been proposed and evaluated. In the field of PD dysgraphia, a significant improvement in the discrimination power and descriptive abilities was proven. Similarly, the proposed methodology improved current state-of-the-art techniques of GD analysis in school-aged children. The newly designed parametrization has been optimized in the scope of the computational performance (up to 80 %) as well as in FD order fine-tuning. Finally, various FD-approaches were compared, namely Riemann-Liouville, Caputoâs, together with Grnwald-Letnikov approximation to identify the most suitable approach for particular areas of GD analysis.

Novel neural approaches to data topology analysis and telemedicine

1noL'abstract è presente nell'allegato / the abstract is in the attachmentopen676. INGEGNERIA ELETTRICAnoopenRandazzo, Vincenz

Effect of visual feedback on the static and kinetic individual characteristics of handwriting

It has been previously established that handwriting is a motor skill defined in a two dimensional spatial domain, consisted of three major levels through which the motor units that contain the letter trajectories are retrieved from their motor memory storage and translated into a process of muscle commands via muscle adjustments. As soon as individuals start learning how to write they are introduced to a writing system common to a group of writers connected by geographic, academic, temporal, national or occupational links. As the writing ability evolves, writers distance themselves from the class system, that they were taught, develop peculiarities in handwriting and acquire personal writing characteristics, the so called individual characteristics of handwriting, which are considered the backbone of forensic handwriting identification. Handwriting is influenced by a number of genetic, physiological and biomechanical factors. Some factors can change the individual's writing so drastically that it may be impossible to make an accurate comparison of the person's normal writing with the person's abnormal writing causing serious problems for forensic document examiners. However the research regarding the visual feedback is partially contradictory regarding the degree of its influence on the individual characteristics. A two-pronged approach was designed in order to investigate the degree of this influence: Samples of signatures, cursive and block handwriting written with and without visual feedback were collected by 40 volunteers and were imported in a PC via an opaque pen tablet using an electronic inking pen. The data was stored and analyzed in a handwriting movement analysis software module specially designed for this research, that was attached in the software MovAlyzeR by Neuroscript LLC. Peer reviewed forensic comparison by a forensic document examined (FDE) between the two groups (that is the group of samples executed with normal visual feedback versus the group of samples executed without visual feedback) shows total lack of significant differences between samples of the two different conditions and the existence of a large corpus of similarities in the design and the pictorial aspect, regardless of the complexity of the samples. Focusing on the cursive and block handwriting, six traits linked to the absence of visual feedback where found: change of overall size, non uniformity of left margins, change of slant, avoidance of pen lifts, inclusion of extra trajectories and decrease of line quality. Furthermore, it was established that the absence of visual feedback by itself cannot lead a trained FDE to an erroneous conclusion. The statistical analysis shows that visual feedback significant influences the duration and average absolute velocity of the signature execution, since the signature is executed more slowly under no visual feedback. Further analysis of the cursive handwriting shows that without visual feedback there is a significant increase in absolute and horizontal size as well as average pen pressure and a decrease in slant and vertical size while in block handwriting there is a significant increase in absolute and horizontal size, average pen pressure as well as duration and a decrease in slant, average absolute velocity and vertical size. The comparative analysis suggests that the factors of gender, educational level and handedness creates an insignificant influence during the comparison of the two conditions of the researched individual characteristics, with the only notable exception of the relationship between signature duration and educational level due to automation and its results in the memory retrieval program of the allographs. The combination of the above findings suggests that all types of writing (signature, cursive and block handwriting) are governed by a single major open loop motor program, which is not significantly influenced by visual feedback -no evidence was found that visual feedback intervenes significantly in the procedure of allograph execution, but is mainly linked with the auxiliary order of macro-managing, inspection and possibly correction of the overall outcome of the combination of the above allographs

Neural Networks for Indoor Human Activity Reconstructions

Low cost, ubiquitous, tagless, and privacy aware indoor monitoring is essential to many existing or future applications, such as assisted living of elderly persons. We explore how well different types of neural networks in basic configurations can extract location and movement information from noisy experimental data (with both high-pitch and slow drift noise) obtained from capacitive sensors operating in loading mode at ranges much longer that the diagonal of their plates. Through design space exploration, we optimize and analyze the location and trajectory tracking inference performance of multilayer perceptron (MLP), autoregressive feedforward, 1D Convolutional (1D-CNN), and Long-Short Term Memory (LSTM) neural networks on experimental data collected using four capacitive sensors with 16 cm x 16 cm plates deployed on the boundaries of a 3 m x 3 m open space in our laboratory. We obtain the minimum error using a 1D-CNN [0.251 m distance Root Mean Square Error (RMSE) and 0.307 m Average Distance Error (ADE)] and the smoothest trajectory inference using an LSTM, albeit with higher localization errors (0.281 m RMSE and 0.326 m ADE). 1D Convolutional and window-based neural networks have best inference accuracy and smoother trajectory reconstruction. LSTMs seem to infer best the person movement dynamics

Verificaciónn de firma y gráficos manuscritos: Características discriminantes y nuevos escenarios de aplicación biométrica

Tesis doctoral inédita leída en la Escuela Politécnica Superior, Departamento de Tecnología Electrónica y de las Comunicaciones. Fecha de lectura: Febrero 2015The proliferation of handheld devices such as smartphones and tablets brings a new scenario for biometric authentication, and in particular to automatic signature verification. Research on signature verification has been traditionally carried out using signatures acquired on digitizing tablets or Tablet-PCs. This PhD Thesis addresses the problem of user authentication on handled devices using handwritten signatures and graphical passwords based on free-form doodles, as well as the effects of biometric aging on signatures. The Thesis pretends to analyze: (i) which are the effects of mobile conditions on signature and doodle verification, (ii) which are the most distinctive features in mobile conditions, extracted from the pen or fingertip trajectory, (iii) how do different similarity computation (i.e. matching) algorithms behave with signatures and graphical passwords captured on mobile conditions, and (iv) what is the impact of aging on signature features and verification performance. Two novel datasets have been presented in this Thesis. A database containing free-form graphical passwords drawn with the fingertip on a smartphone is described. It is the first publicly available graphical password database to the extent of our knowledge. A dataset containing signatures from users captured over a period 15 months is also presented, aimed towards the study of biometric aging. State-of-the-art local and global matching algorithms are used, namely Hidden Markov Models, Gaussian Mixture Models, Dynamic Time Warping and distance-based classifiers. A large proportion of features presented in the research literature is considered in this Thesis. The experimental contribution of this Thesis is divided in three main topics: signature verification on handheld devices, the effects of aging on signature verification, and free-form graphical password-based authentication. First, regarding signature verification in mobile conditions, we use a database captured both on a handheld device and digitizing tablet in an office-like scenario. We analyze the discriminative power of both global and local features using discriminant analysis and feature selection techniques. The effects of the lack of pen-up trajectories on handheld devices (when the stylus tip is not in contact with the screen) are also studied. We then analyze the effects of biometric aging on the signature trait. Using three different matching algorithms, Hidden Markov Models (HMM), Dynamic Time Warping (DTW), and distance-based classifiers, the impact in verification performance is studied. We also study the effects of aging on individual users and individual signature features. Template update techniques are analyzed as a way of mitigating the negative impact of aging. Regarding graphical passwords, the DooDB graphical password database is first presented. A statistical analysis is performed comparing the database samples (free-form doodles and simplified signatures) with handwritten signatures. The sample variability (inter-user, intra-user and inter-session) is also analyzed, as well as the learning curve for each kind of trait. Benchmark results are also reported using state of the art classifiers. Graphical password verification is afterwards studied using features and matching algorithms from the signature verification state of the art. Feature selection is also performed and the resulting feature sets are analyzed. The main contributions of this work can be summarized as follows. A thorough analysis of individual feature performance has been carried out, both for global and local features and on signatures acquired using pen tablets and handheld devices. We have found which individual features are the most robust and which have very low discriminative potential (pen inclination and pressure among others). It has been found that feature selection increases verification performance dramatically, from example from ERRs (Equal Error Rates) over 30% using all available local features, in the case of handheld devices and skilled forgeries, to rates below 20% after feature selection. We study the impact of the lack of trajectory information when the pen tip is not in contact with the acquisition device surface (which happens when touchscreens are used for signature acquisitions), and we have found that the lack of pen-up trajectories negatively affects verification performance. As an example, the EER for the local system increases from 9.3% to 12.1% against skilled forgeries when pen-up trajectories are not available. We study the effects of biometric aging on signature verification and study a number of ways to compensate the observed performance degradation. It is found that aging does not affect equally all the users in the database and that features related to signature dynamics are more degraded than static features. Comparing the performance using test signatures from the first months with the last months, a variable effect of aging on the EER against random forgeries is observed in the three systems that are evaluated, from 0.0% to 0.5% in the DTW system, from 1.0% to 5.0% in the distance-based system using global features, and from 3.2% to 27.8% in the HMM system. A new graphical password database has been acquired and made publicly available. Verification algorithms for finger-drawn graphical passwords and simplified signatures are compared and feature analysis is performed. We have found that inter-session variability has a highly negative impact on verification performance, but this can be mitigated performing feature selection and applying fusion of different matchers. It has also been found that some feature types are prevalent in the optimal feature vectors and that classifiers have a very different behavior against skilled and random forgeries. An EER of 3.4% and 22.1% against random and skilled forgeries is obtained for free-form doodles, which is a promising performance

AutoGraff: towards a computational understanding of graffiti writing and related art forms.

The aim of this thesis is to develop a system that generates letters and pictures with a style that is immediately recognizable as graffiti art or calligraphy. The proposed system can be used similarly to, and in tight integration with, conventional computer-aided geometric design tools and can be used to generate synthetic graffiti content for urban environments in games and in movies, and to guide robotic or fabrication systems that can materialise the output of the system with physical drawing media. The thesis is divided into two main parts. The first part describes a set of stroke primitives, building blocks that can be combined to generate different designs that resemble graffiti or calligraphy. These primitives mimic the process typically used to design graffiti letters and exploit well known principles of motor control to model the way in which an artist moves when incrementally tracing stylised letter forms. The second part demonstrates how these stroke primitives can be automatically recovered from input geometry defined in vector form, such as the digitised traces of writing made by a user, or the glyph outlines in a font. This procedure converts the input geometry into a seed that can be transformed into a variety of calligraphic and graffiti stylisations, which depend on parametric variations of the strokes