Memory pattern identification for feedback tracking control in human-machine systems

Objective: The aim of this paper was to identify the characteristics of memory patterns with respect to a visual input, perceived by the human operator during a manual control task, which consisted in following a moving target on a display with a cursor.Background: Manual control tasks involve nondeclarative memory. The memory encodings of different motor skills have been referred to as procedural memories. The procedural memories have a pattern, which this paper sought to identify for the particular case of a onedimensional tracking task. Specifically, data recorded from human subjects controlling dynamical systems with different fractional order were investigated.Method: A Finite Impulse Response (FIR) controller was fitted to the data, and pattern analysis was performed to the fitted parameters. Then, the FIR model was further reduced to a lower order controller; from the simplified model, the stability analysis of the human-machine system in closedloop was conducted.Results: It is shown that the FIR model can be employed to identify and represent patterns in human procedural memories during manual control tasks. The obtained procedural memory pattern presents a time scale of about 650 ms before decay. Furthermore, the fitted controller is stable for systems with fractional order less or equal to 1.Conclusion: For systems of different fractional order, the proposed control scheme – based on a FIR model – can effectively characterize the linear properties of manual control in humans.Application: This research supports a biofidelic approach to human manual control modeling over feedback visual perceptions. Relevant applications of this research are: the development of shared-control systems, where a virtual human model assists the human during a control task, and human operator state monitoring.</div

Characterization of Two-Dimensional Oculomotor Control During Goal-Directed Eye Movements in Humans

Oculomotor control is a subset of sensorimotor control that allows humans to make extremely accurate eye movements for ADL. Impairments to oculomotor control can increase the impact of sensorimotor control deficits, especially in neurodegenerative diseases such as MS. Here, a two-dimensional computational control system of saccades and smooth-pursuit eye movements was compiled from literature to systematically characterize oculomotor control in eight visually-healthy humans as a precursor to studying the relationship between oculomotor and sensorimotor control in patient populations. Subjects visually tracked a single dot on a 41 x 30.5 cm monitor in a dark room while eye positions were recorded at 60 Hz by a video based eye tracker. Data from visual tasks separately consisting of saccades and smooth-pursuit along the horizontal and/or vertical midlines were inputs to an error minimization algorithm that identified individually for each subject the parameters characterizing motor command generation and two-dimensional interactions within ocular dynamics, with bootstrap analysis quantifying the certainty of parameter estimates. Cross-correlation between target and subject gaze positions was used to identify neuronal conduction speeds for saccades and smooth-pursuit processing. A task consisting of small saccades identified the minimum position error required for saccade initiation. A final task combining saccade and smooth-pursuit movements was used to evaluate model performances. The model accounted for 96% and 98% of variability for subject saccade and smooth-pursuit eye movements, respectively. The 2-D model analysis of saccades and smooth-pursuit identified interactions between horizontal and vertical oculomotor control indicative of component stretching but did not verify the increased speed of vertical versus horizontal eye movements reported in literature. A novel interaction associated with centrifugal curvature was also identified, but the functional effects the interactions were small. Estimated latencies of saccade and smooth-pursuit processing of 242 and 107 ms, respectively, were within ranges provided by literature, while dead zone values for saccade initiation had a 97% error from values provided by literature. The quantitative framework presented in this study may be used in future studies that include MS patients, in which oculomotor control characterization may reveal differences in control strategies for goal-directed ocular movements relative to healthy individuals

Extended crossover model for human-control of fractional order plants

A data-driven generalization of the crossover model is proposed, characterizing the human control of systems with both integer and fractional-order plant dynamics. The model is developed and validated using data obtained from human subjects operating in compensatory and pursuit tracking tasks. From the model, it is inferred that humans possess a limited but consistent capability to compensate for fractional-order plant dynamics. Further, a review of potential sources of fractionality within such man–machine systems suggests that visual perception, based on visual cues that contain memory, and muscular dynamics are likely sources of fractional-order dynamics within humans themselves. Accordingly, a possible mechanism for fractional-order compensation, operating between visual and muscular sub-systems, is proposed. Deeper analysis of the data shows that human response is more highly correlated to fractional-order representations of visual cues, rather than directly to objective engineering variables, as is commonly proposed in human control models in the literature. These results are expected to underpin future design developments in human-in-the-loop cyber-physical systems, for example, in semi-autonomous highway driving

Change blindness: eradication of gestalt strategies

Arrays of eight, texture-defined rectangles were used as stimuli in a one-shot change blindness (CB) task where there was a 50% chance that one rectangle would change orientation between two successive presentations separated by an interval. CB was eliminated by cueing the target rectangle in the first stimulus, reduced by cueing in the interval and unaffected by cueing in the second presentation. This supports the idea that a representation was formed that persisted through the interval before being 'overwritten' by the second presentation (Landman et al, 2003 Vision Research 43149–164]. Another possibility is that participants used some kind of grouping or Gestalt strategy. To test this we changed the spatial position of the rectangles in the second presentation by shifting them along imaginary spokes (by ±1 degree) emanating from the central fixation point. There was no significant difference seen in performance between this and the standard task [F(1,4)=2.565, p=0.185]. This may suggest two things: (i) Gestalt grouping is not used as a strategy in these tasks, and (ii) it gives further weight to the argument that objects may be stored and retrieved from a pre-attentional store during this task

Identification of low order models for large scale processes

Many industrial chemical processes are complex, multi-phase and large scale in nature. These processes are characterized by various nonlinear physiochemical effects and fluid flows. Such processes often show coexistence of fast and slow dynamics during their time evolutions. The increasing demand for a flexible operation of a complex process, a pressing need to improve the product quality, an increasing energy cost and tightening environmental regulations make it rewarding to automate a large scale manufacturing process. Mathematical tools used for process modeling, simulation and control are useful to meet these challenges. Towards this purpose, development of process models, either from the first principles (conservation laws) i.e. the rigorous models or the input-output data based models constitute an important step. Both types of models have their own advantages and pitfalls. Rigorous process models can approximate the process behavior reasonably well. The ability to extrapolate the rigorous process models and the physical interpretation of their states make them more attractive for the automation purpose over the input-output data based identified models. Therefore, the use of rigorous process models and rigorous model based predictive control (R-MPC) for the purpose of online control and optimization of a process is very promising. However, due to several limitations e.g. slow computation speed and the high modeling efforts, it becomes difficult to employ the rigorous models in practise. This thesis work aims to develop a methodology which will result in smaller, less complex and computationally efficient process models from the rigorous process models which can be used in real time for online control and dynamic optimization of the industrial processes. Such methodology is commonly referred to as a methodology of Model (order) Reduction. Model order reduction aims at removing the model redundancy from the rigorous process models. The model order reduction methods that are investigated in this thesis, are applied to two benchmark examples, an industrial glass manufacturing process and a tubular reactor. The complex, nonlinear, multi-phase fluid flow that is observed in a glass manufacturing process offers multiple challenges to any model reduction technique. Often, the rigorous first principle models of these benchmark examples are implemented in a discretized form of partial differential equations and their solutions are computed using the Computational Fluid Dynamics (CFD) numerical tools. Although these models are reliable representations of the underlying process, computation of their dynamic solutions require a significant computation efforts in the form of CPU power and simulation time. The glass manufacturing process involves a large furnace whose walls wear out due to the high process temperature and aggressive nature of the molten glass. It is shown here that the wearing of a glass furnace walls result in change of flow patterns of the molten glass inside the furnace. Therefore it is also desired from the reduced order model to approximate the process behavior under the influence of changes in the process parameters. In this thesis the problem of change in flow patterns as result of changes in the geometric parameter is treated as a bifurcation phenomenon. Such bifurcations exhibited by the full order model are detected using a novel framework of reduced order models and hybrid detection mechanisms. The reduced order models are obtained using the methods explained in the subsequent paragraphs. The model reduction techniques investigated in this thesis are based on the concept of Proper Orthogonal Decompositions (POD) of the process measurements or the simulation data. The POD method of model reduction involves spectral decomposition of system solutions and results into arranging the spatio-temporal data in an order of increasing importance. The spectral decomposition results into spatial and temporal patterns. Spatial patterns are often known as POD basis while the temporal patterns are known as the POD modal coefficients. Dominant spatio-temporal patterns are then chosen to construct the most relevant lower dimensional subspace. The subsequent step involves a Galerkin projection of the governing equations of a full order first principle model on the resulting lower dimensional subspace. This thesis can be viewed as a contribution towards developing the databased nonlinear model reduction technique for large scale processes. The major contribution of this thesis is presented in the form of two novel identification based approaches to model order reduction. The methods proposed here are based on the state information of a full order model and result into linear and nonlinear reduced order models. Similar to the POD method explained in the previous paragraph, the first step of the proposed identification based methods involve spectral decomposition. The second step is different and does not involve the Galerkin projection of the equation residuals. Instead, the second step involves identification of reduced order models to approximate the evolution of POD modal coefficients. Towards this purpose, two different methods are presented. The first method involves identification of locally valid linear models to represent the dynamic behavior of the modal coefficients. Global behavior is then represented by ‘blending’ the local models. The second method involves direct identification of the nonlinear models to represent dynamic evolution of the model coefficients. In the first proposed model reduction method, the POD modal coefficients, are treated as outputs of an unknown reduced order model that is to be identified. Using the tools from the field of system identification, a blackbox reduced order model is then identified as a linear map between the plant inputs and the modal coefficients. Using this method, multiple local reduced LTI models corresponding to various working points of the process are identified. The working points cover the nonlinear operation range of the process which describes the global process behavior. These reduced LTI models are then blended into a single Reduced Order-Linear Parameter Varying (ROLPV) model. The weighted blending is based on nonlinear splines whose coefficients are estimated using the state information of the full order model. Along with the process nonlinearity, the nonlinearity arising due to the wear of the furnace wall is also approximated using the RO-LPV modeling framework. The second model reduction method that is proposed in this thesis allows approximation of a full order nonlinear model by various (linear or nonlinear) model structures. It is observed in this thesis, that, for certain class of full order models, the POD modal coefficients can be viewed as the states of the reduced order model. This knowledge is further used to approximate the dynamic behavior of the POD modal coefficients. In particular, reduced order nonlinear models in the form of tensorial (multi-variable polynomial) systems are identified. In the view of these nonlinear tensorial models, the stability and dissipativity of these models is investigated. During the identification of the reduced order models, the physical interpretation of the states of the full order rigorous model is preserved. Due to the smaller dimension and the reduced complexity, the reduced order models are computationally very efficient. The smaller computation time allows them to be used for online control and optimization of the process plant. The possibility of inferring reduced order models from the state information of a full order model alone i.e. the possibility to infer the reduced order models in the absence of access to the governing equations of a full order model (as observed for many commercial software packages) make the methods presented here attractive. The resulting reduced order models need further system theoretic analysis in order to estimate the model quality with respect to their usage in an online controller setting

Deep into the Eyes: Applying Machine Learning to improve Eye-Tracking

Eye-tracking has been an active research area with applications in personal and behav- ioral studies, medical diagnosis, virtual reality, and mixed reality applications. Improving the robustness, generalizability, accuracy, and precision of eye-trackers while maintaining privacy is crucial. Unfortunately, many existing low-cost portable commercial eye trackers suffer from signal artifacts and a low signal-to-noise ratio. These trackers are highly depen- dent on low-level features such as pupil edges or diffused bright spots in order to precisely localize the pupil and corneal reflection. As a result, they are not reliable for studying eye movements that require high precision, such as microsaccades, smooth pursuit, and ver- gence. Additionally, these methods suffer from reflective artifacts, occlusion of the pupil boundary by the eyelid and often require a manual update of person-dependent parame- ters to identify the pupil region. In this dissertation, I demonstrate (I) a new method to improve precision while maintaining the accuracy of head-fixed eye trackers by combin- ing velocity information from iris textures across frames with position information, (II) a generalized semantic segmentation framework for identifying eye regions with a further extension to identify ellipse fits on the pupil and iris, (III) a data-driven rendering pipeline to generate a temporally contiguous synthetic dataset for use in many eye-tracking ap- plications, and (IV) a novel strategy to preserve privacy in eye videos captured as part of the eye-tracking process. My work also provides the foundation for future research by addressing critical questions like the suitability of using synthetic datasets to improve eye-tracking performance in real-world applications, and ways to improve the precision of future commercial eye trackers with improved camera specifications

Sparse Modeling for Image and Vision Processing

In recent years, a large amount of multi-disciplinary research has been conducted on sparse models and their applications. In statistics and machine learning, the sparsity principle is used to perform model selection---that is, automatically selecting a simple model among a large collection of them. In signal processing, sparse coding consists of representing data with linear combinations of a few dictionary elements. Subsequently, the corresponding tools have been widely adopted by several scientific communities such as neuroscience, bioinformatics, or computer vision. The goal of this monograph is to offer a self-contained view of sparse modeling for visual recognition and image processing. More specifically, we focus on applications where the dictionary is learned and adapted to data, yielding a compact representation that has been successful in various contexts.Comment: 205 pages, to appear in Foundations and Trends in Computer Graphics and Visio

A Unified Cognitive Model of Visual Filling-In Based on an Emergic Network Architecture

The Emergic Cognitive Model (ECM) is a unified computational model of visual filling-in based on the Emergic Network architecture. The Emergic Network was designed to help realize systems undergoing continuous change. In this thesis, eight different filling-in phenomena are demonstrated under a regime of continuous eye movement (and under static eye conditions as well). ECM indirectly demonstrates the power of unification inherent with Emergic Networks when cognition is decomposed according to finer-grained functions supporting change. These can interact to raise additional emergent behaviours via cognitive re-use, hence the Emergic prefix throughout. Nevertheless, the model is robust and parameter free. Differential re-use occurs in the nature of model interaction with a particular testing paradigm. ECM has a novel decomposition due to the requirements of handling motion and of supporting unified modelling via finer functional grains. The breadth of phenomenal behaviour covered is largely to lend credence to our novel decomposition. The Emergic Network architecture is a hybrid between classical connectionism and classical computationalism that facilitates the construction of unified cognitive models. It helps cutting up of functionalism into finer-grains distributed over space (by harnessing massive recurrence) and over time (by harnessing continuous change), yet simplifies by using standard computer code to focus on the interaction of information flows. Thus while the structure of the network looks neurocentric, the dynamics are best understood in flowcentric terms. Surprisingly, dynamic system analysis (as usually understood) is not involved. An Emergic Network is engineered much like straightforward software or hardware systems that deal with continuously varying inputs. Ultimately, this thesis addresses the problem of reduction and induction over complex systems, and the Emergic Network architecture is merely a tool to assist in this epistemic endeavour. ECM is strictly a sensory model and apart from perception, yet it is informed by phenomenology. It addresses the attribution problem of how much of a phenomenon is best explained at a sensory level of analysis, rather than at a perceptual one. As the causal information flows are stable under eye movement, we hypothesize that they are the locus of consciousness, howsoever it is ultimately realized