Contributions to Pen & Touch Human-Computer Interaction

[EN] Computers are now present everywhere, but their potential is not fully exploited due to some lack of acceptance. In this thesis, the pen computer paradigm is adopted, whose main idea is to replace all input devices by a pen and/or the fingers, given that the origin of the rejection comes from using unfriendly interaction devices that must be replaced by something easier for the user. This paradigm, that was was proposed several years ago, has been only recently fully implemented in products, such as the smartphones. But computers are actual illiterates that do not understand gestures or handwriting, thus a recognition step is required to "translate" the meaning of these interactions to computer-understandable language. And for this input modality to be actually usable, its recognition accuracy must be high enough. In order to realistically think about the broader deployment of pen computing, it is necessary to improve the accuracy of handwriting and gesture recognizers. This thesis is devoted to study different approaches to improve the recognition accuracy of those systems. First, we will investigate how to take advantage of interaction-derived information to improve the accuracy of the recognizer. In particular, we will focus on interactive transcription of text images. Here the system initially proposes an automatic transcript. If necessary, the user can make some corrections, implicitly validating a correct part of the transcript. Then the system must take into account this validated prefix to suggest a suitable new hypothesis. Given that in such application the user is constantly interacting with the system, it makes sense to adapt this interactive application to be used on a pen computer. User corrections will be provided by means of pen-strokes and therefore it is necessary to introduce a recognizer in charge of decoding this king of nondeterministic user feedback. However, this recognizer performance can be boosted by taking advantage of interaction-derived information, such as the user-validated prefix. Then, this thesis focuses on the study of human movements, in particular, hand movements, from a generation point of view by tapping into the kinematic theory of rapid human movements and the Sigma-Lognormal model. Understanding how the human body generates movements and, particularly understand the origin of the human movement variability, is important in the development of a recognition system. The contribution of this thesis to this topic is important, since a new technique (which improves the previous results) to extract the Sigma-lognormal model parameters is presented. Closely related to the previous work, this thesis study the benefits of using synthetic data as training. The easiest way to train a recognizer is to provide "infinite" data, representing all possible variations. In general, the more the training data, the smaller the error. But usually it is not possible to infinitely increase the size of a training set. Recruiting participants, data collection, labeling, etc., necessary for achieving this goal can be time-consuming and expensive. One way to overcome this problem is to create and use synthetically generated data that looks like the human. We study how to create these synthetic data and explore different approaches on how to use them, both for handwriting and gesture recognition. The different contributions of this thesis have obtained good results, producing several publications in international conferences and journals. Finally, three applications related to the work of this thesis are presented. First, we created Escritorie, a digital desk prototype based on the pen computer paradigm for transcribing handwritten text images. Second, we developed "Gestures à Go Go", a web application for bootstrapping gestures. Finally, we studied another interactive application under the pen computer paradigm. In this case, we study how translation reviewing can be done more ergonomically using a pen.[ES] Hoy en día, los ordenadores están presentes en todas partes pero su potencial no se aprovecha debido al "miedo" que se les tiene. En esta tesis se adopta el paradigma del pen computer, cuya idea fundamental es sustituir todos los dispositivos de entrada por un lápiz electrónico o, directamente, por los dedos. El origen del rechazo a los ordenadores proviene del uso de interfaces poco amigables para el humano. El origen de este paradigma data de hace más de 40 años, pero solo recientemente se ha comenzado a implementar en dispositivos móviles. La lenta y tardía implantación probablemente se deba a que es necesario incluir un reconocedor que "traduzca" los trazos del usuario (texto manuscrito o gestos) a algo entendible por el ordenador. Para pensar de forma realista en la implantación del pen computer, es necesario mejorar la precisión del reconocimiento de texto y gestos. El objetivo de esta tesis es el estudio de diferentes estrategias para mejorar esta precisión. En primer lugar, esta tesis investiga como aprovechar información derivada de la interacción para mejorar el reconocimiento, en concreto, en la transcripción interactiva de imágenes con texto manuscrito. En la transcripción interactiva, el sistema y el usuario trabajan "codo con codo" para generar la transcripción. El usuario valida la salida del sistema proporcionando ciertas correcciones, mediante texto manuscrito, que el sistema debe tener en cuenta para proporcionar una mejor transcripción. Este texto manuscrito debe ser reconocido para ser utilizado. En esta tesis se propone aprovechar información contextual, como por ejemplo, el prefijo validado por el usuario, para mejorar la calidad del reconocimiento de la interacción. Tras esto, la tesis se centra en el estudio del movimiento humano, en particular del movimiento de las manos, utilizando la Teoría Cinemática y su modelo Sigma-Lognormal. Entender como se mueven las manos al escribir, y en particular, entender el origen de la variabilidad de la escritura, es importante para el desarrollo de un sistema de reconocimiento, La contribución de esta tesis a este tópico es importante, dado que se presenta una nueva técnica (que mejora los resultados previos) para extraer el modelo Sigma-Lognormal de trazos manuscritos. De forma muy relacionada con el trabajo anterior, se estudia el beneficio de utilizar datos sintéticos como entrenamiento. La forma más fácil de entrenar un reconocedor es proporcionar un conjunto de datos "infinito" que representen todas las posibles variaciones. En general, cuanto más datos de entrenamiento, menor será el error del reconocedor. No obstante, muchas veces no es posible proporcionar más datos, o hacerlo es muy caro. Por ello, se ha estudiado como crear y usar datos sintéticos que se parezcan a los reales. Las diferentes contribuciones de esta tesis han obtenido buenos resultados, produciendo varias publicaciones en conferencias internacionales y revistas. Finalmente, también se han explorado tres aplicaciones relaciones con el trabajo de esta tesis. En primer lugar, se ha creado Escritorie, un prototipo de mesa digital basada en el paradigma del pen computer para realizar transcripción interactiva de documentos manuscritos. En segundo lugar, se ha desarrollado "Gestures à Go Go", una aplicación web para generar datos sintéticos y empaquetarlos con un reconocedor de forma rápida y sencilla. Por último, se presenta un sistema interactivo real bajo el paradigma del pen computer. En este caso, se estudia como la revisión de traducciones automáticas se puede realizar de forma más ergonómica.[CA] Avui en dia, els ordinadors són presents a tot arreu i es comunament acceptat que la seva utilització proporciona beneficis. No obstant això, moltes vegades el seu potencial no s'aprofita totalment. En aquesta tesi s'adopta el paradigma del pen computer, on la idea fonamental és substituir tots els dispositius d'entrada per un llapis electrònic, o, directament, pels dits. Aquest paradigma postula que l'origen del rebuig als ordinadors prové de l'ús d'interfícies poc amigables per a l'humà, que han de ser substituïdes per alguna cosa més coneguda. Per tant, la interacció amb l'ordinador sota aquest paradigma es realitza per mitjà de text manuscrit i/o gestos. L'origen d'aquest paradigma data de fa més de 40 anys, però només recentment s'ha començat a implementar en dispositius mòbils. La lenta i tardana implantació probablement es degui al fet que és necessari incloure un reconeixedor que "tradueixi" els traços de l'usuari (text manuscrit o gestos) a alguna cosa comprensible per l'ordinador, i el resultat d'aquest reconeixement, actualment, és lluny de ser òptim. Per pensar de forma realista en la implantació del pen computer, cal millorar la precisió del reconeixement de text i gestos. L'objectiu d'aquesta tesi és l'estudi de diferents estratègies per millorar aquesta precisió. En primer lloc, aquesta tesi investiga com aprofitar informació derivada de la interacció per millorar el reconeixement, en concret, en la transcripció interactiva d'imatges amb text manuscrit. En la transcripció interactiva, el sistema i l'usuari treballen "braç a braç" per generar la transcripció. L'usuari valida la sortida del sistema donant certes correccions, que el sistema ha d'usar per millorar la transcripció. En aquesta tesi es proposa utilitzar correccions manuscrites, que el sistema ha de reconèixer primer. La qualitat del reconeixement d'aquesta interacció és millorada, tenint en compte informació contextual, com per exemple, el prefix validat per l'usuari. Després d'això, la tesi se centra en l'estudi del moviment humà en particular del moviment de les mans, des del punt de vista generatiu, utilitzant la Teoria Cinemàtica i el model Sigma-Lognormal. Entendre com es mouen les mans en escriure és important per al desenvolupament d'un sistema de reconeixement, en particular, per entendre l'origen de la variabilitat de l'escriptura. La contribució d'aquesta tesi a aquest tòpic és important, atès que es presenta una nova tècnica (que millora els resultats previs) per extreure el model Sigma- Lognormal de traços manuscrits. De forma molt relacionada amb el treball anterior, s'estudia el benefici d'utilitzar dades sintètiques per a l'entrenament. La forma més fàcil d'entrenar un reconeixedor és proporcionar un conjunt de dades "infinit" que representin totes les possibles variacions. En general, com més dades d'entrenament, menor serà l'error del reconeixedor. No obstant això, moltes vegades no és possible proporcionar més dades, o fer-ho és molt car. Per això, s'ha estudiat com crear i utilitzar dades sintètiques que s'assemblin a les reals. Les diferents contribucions d'aquesta tesi han obtingut bons resultats, produint diverses publicacions en conferències internacionals i revistes. Finalment, també s'han explorat tres aplicacions relacionades amb el treball d'aquesta tesi. En primer lloc, s'ha creat Escritorie, un prototip de taula digital basada en el paradigma del pen computer per realitzar transcripció interactiva de documents manuscrits. En segon lloc, s'ha desenvolupat "Gestures à Go Go", una aplicació web per a generar dades sintètiques i empaquetar-les amb un reconeixedor de forma ràpida i senzilla. Finalment, es presenta un altre sistema inter- actiu sota el paradigma del pen computer. En aquest cas, s'estudia com la revisió de traduccions automàtiques es pot realitzar de forma més ergonòmica.Martín-Albo Simón, D. (2016). Contributions to Pen & Touch Human-Computer Interaction [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/68482TESI

Teaching and learning writing at primary school: an exploration of writing environments, transcription and text generation

Introduction: Children’s writing skill continues to cause concern. While research into interventions is on-going, little is known about writing in natural classroom environments or the effects of individual differences on everyday performance. This project examined real-life handwritten work in primary school, focussing on transcription and its relationship with text-generation in different classroom writing environments (i.e. whether content is teacherdetermined, child-determined, or generated jointly by teachers and children). Method: Nine Year 5 teachers were interviewed about their classroom practice, training, and beliefs relating to writing tuition. All handwritten work by 135 children from one week was photographed and transcribed. Amount written and spelling accuracy were compared between children, classes and writing environments. Relationships between transcription and word-level text generation were examined. Compositional quality of child-generated writing was scored and factors drawn from the entire project evaluated as predictors of quality. Results: The teachers felt that handwriting tuition should occur throughout primary school. Though compositional quality was considered to be more associated with handwriting speed rather than its neatness, teachers emphasised neatness. The most productive child wrote 16 times more than the least. Lower-productivity classes were typified by a greater proportion of teachergenerated writing. Compositional quality and lexical richness of childgenerated writing were positively associated with amount of teacher + childgenerated writing, but the link with amount of teacher-generated writing was non-significant. Spelling-errors in copying tended to be phonologically implausible whereas in child-generated writing plausible errors were more frequent. Better genre-writing scores were achieved by children who had written more word-types during preparation for the tasks. Strongest predictors of scores were teachers feeling well-prepared for writing tuition and more recent qualification, and larger amounts of teacher + childgenerated writing carried out. Discussion: The national curriculum for handwriting does not require tuition throughout primary school, contrary to motor learning research. More recently qualified teachers were less critical of writing performance being judged against specified criteria than those qualified for longer. Many teachers were unaware of how much copying occurred and copying may be an ineffective means of acquiring vocabulary knowledge; increasing the amount of teacher + child-generated writing may be beneficial. Other theoretical and practical implications are discussed and limitations and future research considered

AutoGraff: towards a computational understanding of graffiti writing and related art forms.

The aim of this thesis is to develop a system that generates letters and pictures with a style that is immediately recognizable as graffiti art or calligraphy. The proposed system can be used similarly to, and in tight integration with, conventional computer-aided geometric design tools and can be used to generate synthetic graffiti content for urban environments in games and in movies, and to guide robotic or fabrication systems that can materialise the output of the system with physical drawing media. The thesis is divided into two main parts. The first part describes a set of stroke primitives, building blocks that can be combined to generate different designs that resemble graffiti or calligraphy. These primitives mimic the process typically used to design graffiti letters and exploit well known principles of motor control to model the way in which an artist moves when incrementally tracing stylised letter forms. The second part demonstrates how these stroke primitives can be automatically recovered from input geometry defined in vector form, such as the digitised traces of writing made by a user, or the glyph outlines in a font. This procedure converts the input geometry into a seed that can be transformed into a variety of calligraphic and graffiti stylisations, which depend on parametric variations of the strokes

The Dollar General: Continuous Custom Gesture Recognition Techniques At Everyday Low Prices

Humans use gestures to emphasize ideas and disseminate information. Their importance is apparent in how we continuously augment social interactions with motion—gesticulating in harmony with nearly every utterance to ensure observers understand that which we wish to communicate, and their relevance has not escaped the HCI community\u27s attention. For almost as long as computers have been able to sample human motion at the user interface boundary, software systems have been made to understand gestures as command metaphors. Customization, in particular, has great potential to improve user experience, whereby users map specific gestures to specific software functions. However, custom gesture recognition remains a challenging problem, especially when training data is limited, input is continuous, and designers who wish to use customization in their software are limited by mathematical attainment, machine learning experience, domain knowledge, or a combination thereof. Data collection, filtering, segmentation, pattern matching, synthesis, and rejection analysis are all non-trivial problems a gesture recognition system must solve. To address these issues, we introduce The Dollar General (TDG), a complete pipeline composed of several novel continuous custom gesture recognition techniques. Specifically, TDG comprises an automatic low-pass filter tuner that we use to improve signal quality, a segmenter for identifying gesture candidates in a continuous input stream, a classifier for discriminating gesture candidates from non-gesture motions, and a synthetic data generation module we use to train the classifier. Our system achieves high recognition accuracy with as little as one or two training samples per gesture class, is largely input device agnostic, and does not require advanced mathematical knowledge to understand and implement. In this dissertation, we motivate the importance of gestures and customization, describe each pipeline component in detail, and introduce strategies for data collection and prototype selection

Détection de la fatigue neuromusculaire de l’épaule au moyen de la Théorie Cinématique des mouvements humains rapides

Les troubles musculo-squelettiques de l’épaule, et plus particulièrement ceux impactant la coiffe des rotateurs sont omniprésents dans notre société. Ils sont handicapants dans la vie de tous les jours et également très coûteux. Les coûts directs et indirects associés sont considérables. Il est alors important de les prévenir. Les causes de blessures à la coiffe des rotateurs sont multifactorielles. Parmi elles se trouve la fatigue musculaire, qui peut être soit centrale (du système nerveux central) ou périphérique (des muscles). Lors de mouvements répétitifs à hauteur d’épaule ou au-dessus de la tête, de la fatigue apparait, qui suite à une réaction en chaine peut entrainer une blessure. Les sportifs, dont les gestes impliquent de nombreuses répétitions de mouvement du bras, sont souvent sujets à des risques de blessures de l’épaule. Une façon de prévenir les blessures est alors d’étudier la fatigue musculaire à la coiffe des rotateurs. Ceci permettrait d’adapter à terme les entrainements pour chaque sportif et d’avoir un traitement personnalisé à chacun par des professionnels de la santé. Pour ce faire, un test clinique de détection de la fatigue doit être mis en place. Il doit être facile d’utilisation, peu coûteux et avec des données fiables, répétables d’un jour à l’autre et d’un évaluateur à l’autre. Cependant, à ce jour, il semble que les outils existants de détection de la fatigue musculaire à l’épaule peuvent difficilement être instaurés en clinique et différencier une fatigue centrale d’une fatigue périphérique. L’objectif de ce projet est d’innover dans les tests cliniques de détection de la fatigue de l’épaule. La Théorie Cinématique des mouvements humains rapides semble être une alternative efficace à l’étude du contrôle moteur. Il s’agit d’une méthode permettant de reconstruire le mouvement humain à partir de fonctions lognormales. Ses paramètres décrivent l’état physiologique du participant. Le dispositif expérimental constitué d’une tablette est ergonomique et peu coûteux. Les gestes à effectuer se rapprochent de l’écriture et sont faciles à exécuter, ce qui fait de cette méthode un bon candidat pour son utilisation en tant que test clinique. Pour ce faire, nous avons donc réalisé une première étude visant à étudier l’efficacité de détection de la fatigue neuromusculaire au moyen de la Théorie Cinématique. Vingt participants sains et sportifs ont réalisé deux séances avec une procédure similaire : exécuter des traits sur une tablette (traits simples, triangles, oscillations horizontales et oscillations verticales), fatiguer un muscle de l’épaule (le sous-scapulaire ou l’infra-épineux en fonction de la séance) et exécuter de nouveaux traits sur tablette. Des différences significatives avec taille d’effet de modérée à très large ont été constatées dans les paramètres de la théorie, suite aux deux séances. D’un point de vue théorique, il a été possible de différencier une fatigue centrale d’une fatigue périphérique. Un profil de réponse général suite à la fatigue a également été dénoté. Par ailleurs, un suivi de fatigue individualisé est également possible dans un souci d’accompagnement personnalisé. Tandis que le test des traits simples permettait de savoir plus facilement si la fatigue était centrale ou périphérique, le test des oscillations était le plus sensible. La différence de réaction suite au muscle fatigué n’a pas encore été établie. La deuxième étude visait à analyser la répétabilité des données d’un jour à l’autre. Pour cela, les mêmes traits que ceux exécutés pendant l’étude de la fatigue ont été réalisés par 40 participants deux fois à un jour d’intervalle minimum entre chaque séance. Une bonne répétabilité des données a été établie pour la plupart des paramètres des tests. Néanmoins, le changement minimal détectable était plus élevé que les différences avant et après fatigue de la première étude. Ceci nous amène à penser qu’il n’est à ce jour pas utilisable dans notre étude de cas. Afin de se conformer aux exigences d’un nouveau test clinique, il faudrait que des études de sensibilité et de spécificité soient entreprises afin de s’assurer de la bonne détection des personnes fatiguées. En dépit des limitations, l’utilisation de la Théorie Cinématique semble être une approche innovante pour détecter la fatigue neuromusculaire, et utilisable en clinique. Ces connaissances sont nécessaires afin de mieux guider les sportifs dans leur entrainement et éviter l’apparition à plus long terme de blessure.----------ABSTRACT Shoulder musculoskeletal disorders, and particularly those affecting the rotator cuff, are omnipresent in our society. They are incapacitating in the daily life and are also expensive. Direct and indirect costs associated with them are huge. Their prevention is therefore important. The causes of rotator cuff injuries are multifactorial. One of them is muscle fatigue, which can be either central (from the central nervous system) or peripheral (from the muscles). Fatigue appears when repetitive movements are made at shoulder height or overhead. As a result, it can lead to injuries. Sportsmen and women, whose movements involve many repetitive arm movements, are often at risk of shoulder injuries. Studying fatigue at the rotator cuff is a good way to prevent injuries. This would help in the long term to adapt training for each athlete and to have a personalized treatment for each one by health professionals. In that way, a clinical test to detect fatigue must be set up. It must be easy to use, inexpensive and with data that are reliable from one day/examiner to the other. However, to date, it appears that the existing tools for detecting shoulder muscle fatigue are difficult to implement in clinics and cannot easily differentiate between central and peripheral fatigue. The objective of this project is to innovate in clinical tests for the detection of shoulder fatigue. The Kinematic Theory of rapid human movement seems to be an effective alternative to study motor control. Human movement is reconstructed through lognormal functions. Their parameters describe the physiological state of the participant. The experimental device consisting of a tablet is ergonomic and inexpensive. The gestures to be performed are similar to writing and are easy to perform, making this method a good choice for clinical uses. We therefore carried out an initial study to investigate the effectiveness of detecting neuromuscular fatigue using the Kinematic Theory. Twenty healthy and active participants carried out two sessions with a similar procedure: performing strokes on a tablet (simple strokes, triangles, horizontal oscillations and vertical oscillations), fatiguing a shoulder muscle (subscapularis or infraspinatus depending on the session) and performing the strokes again on a tablet. Significant differences with moderate to good effect sizes were found in the parameters of the theory, following the two sessions. From a theoretical point of view it was possible to differentiate between central and peripheral fatigue. A general response profile to fatigue was also highlighted. In addition, an individualized fatigue follow-up is also possible. The distinction of the type of fatigue (central vs peripheral) is more easily detectable with the test of the simple strokes. The oscillation test was the most sensitive to fatigue. The difference in reaction to the fatigued muscle has not been established yet. The second study was designed to analyze the data reliability from one day to the other. For this purpose, the same strokes as those performed for the study of fatigue were performed twice with a minimum of one day interval between each session, by 40 participants. Good data reliability was ascertained for most of the parameters. Nevertheless, the minimal detectable change was greater than the pre- and post-fatigue differences in the first study. This leads us to believe that it is not usable yet in our case study. In order to comply with the requirements of a new clinical test, sensitivity and specificity studies would need to be undertaken to ensure the proper detection of fatigued participants. Despite some limitations, the use of the Kinematic Theory appears to be an innovative approach for detecting neuromuscular fatigue, that is clinically applicable. This knowledge is necessary to better guide athletes in their training and, in the long run, to avoid the occurrence of an injury

Gaze-Based Human-Robot Interaction by the Brunswick Model

We present a new paradigm for human-robot interaction based on social signal processing, and in particular on the Brunswick model. Originally, the Brunswick model copes with face-to-face dyadic interaction, assuming that the interactants are communicating through a continuous exchange of non verbal social signals, in addition to the spoken messages. Social signals have to be interpreted, thanks to a proper recognition phase that considers visual and audio information. The Brunswick model allows to quantitatively evaluate the quality of the interaction using statistical tools which measure how effective is the recognition phase. In this paper we cast this theory when one of the interactants is a robot; in this case, the recognition phase performed by the robot and the human have to be revised w.r.t. the original model. The model is applied to Berrick, a recent open-source low-cost robotic head platform, where the gazing is the social signal to be considered