Tracking objects with point clouds from vision and touch

We present an object-tracking framework that fuses point cloud information from an RGB-D camera with tactile information from a GelSight contact sensor. GelSight can be treated as a source of dense local geometric information, which we incorporate directly into a conventional point-cloud-based articulated object tracker based on signed-distance functions. Our implementation runs at 12 Hz using an online depth reconstruction algorithm for GelSight and a modified second-order update for the tracking algorithm. We present data from hardware experiments demonstrating that the addition of contact-based geometric information significantly improves the pose accuracy during contact, and provides robustness to occlusions of small objects by the robot's end effector

Practical and precise projector-camera calibration

International audienceProjectors are important display devices for large scale augmented reality applications. However, precisely calibrating projectors with large focus distances implies a trade-off between practicality and accuracy. People either need a huge calibration board or a precise 3D model [12]. In this paper, we present a practical projector-camera calibration method to solve this problem. The user only needs a small calibration board to calibrate the system regardless of the focus distance of the projector. Results show that the root-mean-squared re-projection error (RMSE) for a 450cm projection distance is only about 4mm, even though it is calibrated using a small B4 (250 × 353mm) calibration board

Automatic annotation of tennis games: an integration of audio, vision, and learning

Fully automatic annotation of tennis game using broadcast video is a task with a great potential but with enormous challenges. In this paper we describe our approach to this task, which integrates computer vision, machine listening, and machine learning. At the low level processing, we improve upon our previously proposed state-of-the-art tennis ball tracking algorithm and employ audio signal processing techniques to detect key events and construct features for classifying the events. At high level analysis, we model event classification as a sequence labelling problem, and investigate four machine learning techniques using simulated event sequences. Finally, we evaluate our proposed approach on three real world tennis games, and discuss the interplay between audio, vision and learning. To the best of our knowledge, our system is the only one that can annotate tennis game at such a detailed level

Automatic annotation of tennis games: an integration of audio, vision, and learning

Fully automatic annotation of tennis game using broadcast video is a task with a great potential but with enormous challenges. In this paper we describe our approach to this task, which integrates computer vision, machine listening, and machine learning. At the low level processing, we improve upon our previously proposed state-of-the-art tennis ball tracking algorithm and employ audio signal processing techniques to detect key events and construct features for classifying the events. At high level analysis, we model event classification as a sequence labelling problem, and investigate four machine learning techniques using simulated event sequences. Finally, we evaluate our proposed approach on three real world tennis games, and discuss the interplay between audio, vision and learning. To the best of our knowledge, our system is the only one that can annotate tennis game at such a detailed level

iCub visual memory inspector: Visualising the iCub’s thoughts

This paper describes the integration of multiple sensory recognition models created by a Synthetic Autobiographical Memory into a structured system. This structured system provides high level control of the overall architecture and interfaces with an iCub simulator based in Unity which provides a virtual space for the display of recollected events

Using gaze data to predict multiword expressions

In recent years gaze data has been increasingly used to improve and evaluate NLP models due to the fact that it carries information about the cognitive processing of linguistic phenomena. In this paper we conduct a preliminary study towards the automatic identification of multiword expressions based on gaze features from native and non-native speakers of English. We report comparisons between a part-ofspeech (POS) and frequency baseline to: i) a prediction model based solely on gaze data and ii) a combined model of gaze data, POS and frequency. In spite of the challenging nature of the task, best performance was achieved by the latter. Furthermore, we explore how the type of gaze data (from native versus non-native speakers) affects the prediction, showing that data from the two groups is discriminative to an equal degree. Finally, we show that late processing measures are more predictive than early ones, which is in line with previous research on idioms and other formulaic structures.Na

The ORKG R Package and Its Use in Data Science

Research infrastructures and services provide access to (meta)data via user interfaces and APIs. The more advanced services also support access through (Python, R, etc.) packages that users can use in computational environments. For scientific information as a particular kind of research data, the Open Research Knowledge Graph (ORKG) is an example of an advanced service that also supports accessing data from Python scripts. Since many research communities use R as the statistical language of choice, we have developed the ORKG R package to support accessing and processing ORKG data directly from R scripts. Inspired by the Python library, the ORKG R package supports a comparable set of features through a similar programmatic interface. Having developed the ORKG R package, we demonstrate its use in various applications grounded in life science and soil science research fields. As an additional key contribution of this work, we show how the ORKG R package can be used in combination with ORKG templates to support the pre-publication production and publication of machine-readable scientific information, during the data analysis phase of the research life cycle and directly in the scripts that produce scientific information. This new mode of machine-readable scientific information production complements the post-publication Crowdsourcing-based manual and NLP-based automated approaches with the major advantages of unmatched high accuracy and fine granularity

Visuospatial encoding deficits and compensatory strategies in schizophrenia revealed by eye movement analysis during a working memory task

Objective: To investigate scanpath abnormalities during the encoding of static stimuli in schizophrenia and their interaction with visuospatial working memory (VSWM) dysfunction. Methods: Outpatients with schizophrenia and control subjects were asked to encode a static pattern for subsequent recognition after a short delay. We measured the number of correct and incorrect choices. We also assessed the number and the distribution of fixations, the scanning time in specific regions of interest (ROIs) and the head movements during the encoding of the stimuli. The distributions of fixations and scanning time in definite ROIs during the discrimination of the correct pattern from the foils were also measured. Results: Patients recognised fewer correct patterns than controls. Correct trials in patients were characterised by a specific exploration of the central part of the stimulus during its presentation, whereas this feature was absent in incorrect trials. However, the scanning time and the numbers of fixations and head movements during encoding were similar in both groups and unrelated to recognition accuracy. In both groups, correct trials were associated with a selective exploration of the correct pattern amongst the six possibilities during recognition. Furthermore, patients gave more attention to incorrect patterns with a leftmost element identical to that of the correct response and also those approximating its global structure. Conclusion: Patients showed a VSWM deficit independent of oculomotor dysfunctions and head movements during encoding. Patients' correct trials were related to specific scanning during encoding and discrimination phases. Analysis of these patterns suggests that patients try to compensate for reduced VSWM ability by using specific encoding strategie