Low latency vision-based control for robotics : a thesis presented in partial fulfilment of the requirements for the degree of Master of Engineering in Mechatronics at Massey University, Manawatu, New Zealand

In this work, the problem of controlling a high-speed dynamic tracking and interception system using computer vision as the measurement unit was explored. High-speed control systems alone present many challenges, and these challenges are compounded when combined with the high volume of data processing required by computer vision systems. A semi-automated foosball table was chosen as the test-bed system because it combines all the challenges associated with a vision-based control system into a single platform. While computer vision is extremely useful and can solve many problems, it can also introduce many problems such as latency, the need for lens and spatial calibration, potentially high power consumption, and high cost. The objective of this work is to explore how to implement computer vision as the measurement unit in a high-speed controller, while minimising latencies caused by the vision itself, communication interfaces, data processing/strategy, instruction execution, and actuator control. Another objective was to implement the solution in one low-latency, low power, low cost embedded system. A field programmable gate array (FPGA) system on chip (SoC), which combines programmable digital logic with a dual core ARM processor (HPS) on the same chip, was hypothesised to be capable of running the described vision-based control system. The FPGA was used to perform streamed image pre-processing, concurrent stepper motor control and provide communication channels for user input, while the HPS performed the lens distortion mapping, intercept calculation and “strategy” control tasks, as well as controlling overall function of the system. Individual vision systems were compared for latency performance. Interception performance of the semi-automated foosball table was then tested for straight, moderate-speed shots with limited view time, and latency was artificially added to the system and the interception results for the same, centre-field shot tested with a variety of different added latencies. The FPGA based system performed the best in both steady-state latency, and novel event detection latency tests. The developed stepper motor control modules performed well in terms of speed, smoothness, resource consumption, and versatility. They are capable of constant velocity, constant acceleration and variable acceleration profiles, as well as being completely parameterisable. The interception modules on the foosball table achieved a 100% interception rate, with a confidence interval of 95%, and reliability of 98.4%. As artificial latency was added to the system, the performance dropped in terms of overall number of successful intercepts. The decrease in performance was roughly linear with a 60% in reduction in performance caused by 100 ms of added latency. Performance dropped to 0% successful intercepts when 166 ms of latency was added. The implications of this work are that FPGA SoC technology may, in future, enable computer vision to be used as a general purpose, high-speed measurement system for a wide variety of control problems

Engineering data compendium. Human perception and performance. User's guide

The concept underlying the Engineering Data Compendium was the product of a research and development program (Integrated Perceptual Information for Designers project) aimed at facilitating the application of basic research findings in human performance to the design and military crew systems. The principal objective was to develop a workable strategy for: (1) identifying and distilling information of potential value to system design from the existing research literature, and (2) presenting this technical information in a way that would aid its accessibility, interpretability, and applicability by systems designers. The present four volumes of the Engineering Data Compendium represent the first implementation of this strategy. This is the first volume, the User's Guide, containing a description of the program and instructions for its use

Transits and Occultations of an Earth-Sized Planet in an 8.5-Hour Orbit

We report the discovery of an Earth-sized planet ($1.16\pm 0.19 R_\oplus$) in an 8.5-hour orbit around a late G-type star (KIC 8435766, Kepler-78). The object was identified in a search for short-period planets in the {\it Kepler} database and confirmed to be a transiting planet (as opposed to an eclipsing stellar system) through the absence of ellipsoidal light variations or substantial radial-velocity variations. The unusually short orbital period and the relative brightness of the host star ($m_{\rm Kep}$ = 11.5) enable robust detections of the changing illumination of the visible hemisphere of the planet, as well as the occultations of the planet by the star. We interpret these signals as representing a combination of reflected and reprocessed light, with the highest planet dayside temperature in the range of 2300 K to 3100 K. Follow-up spectroscopy combined with finer sampling photometric observations will further pin down the system parameters and may even yield the mass of the planet.Comment: Accepted for publication, ApJ, 10 pages and 6 figure

A Low-Dimensional Model of Coordinated Eye and Head Movements

Eye and head movement data, were recorded under head fixed and head-free conditions, and compared with theoretical results obtained using a nonlinear model of eye-head coordination. The model explains slow, or pursuit movement correlated closely to target movement, and saccades, or quick phases of eye movement. Eye movement under head-fixed conditions was modeled by an externally forced Duffing equation, whilst properties of head movement are described by a second externally forced Duffing equation with lower eigen frequency. In the more natural, head-free conditions where both eye and head movements are used synergetically to pursue a visual target, the vestibulocular reflex (VOR) is represented by coefficients defining the mutual coupling between these two oscillatory systems. In the present model, the oscillator that models eye movement has an inhibitory influence on head movement; head to eye coupling coefficients are included to model the influence of the VOR mechanism. Individual eye and head movement patterns in different subjects can be adequately modeled by altering the coupling coefficients. In order to adequatcly simulate those changes introduced by microgravity conditions, the coefficients defining eye-head coordination (mutual coupling) must be changed. It may be hypothesized tlmt such changes in the neurovestibular system could introduce the instability in eye-head coordination, which is known to lead to space sickness.Whitehall Foundation (S93-24

Optimizations and applications in head-mounted video-based eye tracking

Video-based eye tracking techniques have become increasingly attractive in many research fields, such as visual perception and human-computer interface design. The technique primarily relies on the positional difference between the center of the eye\u27s pupil and the first-surface reflection at the cornea, the corneal reflection (CR). This difference vector is mapped to determine an observer\u27s point of regard (POR). In current head-mounted video-based eye trackers, the systems are limited in several aspects, such as inadequate measurement range and misdetection of eye features (pupil and CR). This research first proposes a new `structured illumination\u27 configuration, using multiple IREDs to illuminate the eye, to ensure that eye positions can still be tracked even during extreme eye movements (up to ±45° horizontally and ±25° vertically). Then eye features are detected by a two-stage processing approach. First, potential CRs and the pupil are isolated based on statistical information in an eye image. Second, genuine CRs are distinguished by a novel CR location prediction technique based on the well-correlated relationship between the offset of the pupil and that of the CR. The optical relationship of the pupil and CR offsets derived in this thesis can be applied to two typical illumination configurations - collimated and near-source ones- in the video-based eye tracking system. The relationships from the optical derivation and that from an experimental measurement match well. Two application studies, smooth pursuit dynamics in controlled static (laboratory) and unconstrained vibrating (car) environments were conducted. In the first study, the extended stimuli (color photographs subtending 2° and 17°, respectively) were found to enhance smooth pursuit movements induced by realistic images, and the eye velocity for tracking a small dot (subtending \u3c0.1°) was saturated at about 64 deg/sec while the saturation velocity occurred at higher velocities for the extended images. The difference in gain due to target size was significant between dot and the two extended stimuli, while no statistical difference existed between the two extended stimuli. In the second study, twovisual stimuli same as in the first study were used. The visual performance was impaired dramatically due to the whole body motion in the car, even in the tracking of a slowly moving target (2 deg/sec); the eye was found not able to perform a pursuit task as smooth as in the static environment though the unconstrained head motion in the unstable condition was supposed to enhance the visual performance

Active User Authentication for Smartphones: A Challenge Data Set and Benchmark Results

In this paper, automated user verification techniques for smartphones are investigated. A unique non-commercial dataset, the University of Maryland Active Authentication Dataset 02 (UMDAA-02) for multi-modal user authentication research is introduced. This paper focuses on three sensors - front camera, touch sensor and location service while providing a general description for other modalities. Benchmark results for face detection, face verification, touch-based user identification and location-based next-place prediction are presented, which indicate that more robust methods fine-tuned to the mobile platform are needed to achieve satisfactory verification accuracy. The dataset will be made available to the research community for promoting additional research.Comment: 8 pages, 12 figures, 6 tables. Best poster award at BTAS 201