Vision systems with the human in the loop

The emerging cognitive vision paradigm deals with vision systems that apply machine learning and automatic reasoning in order to learn from what they perceive. Cognitive vision systems can rate the relevance and consistency of newly acquired knowledge, they can adapt to their environment and thus will exhibit high robustness. This contribution presents vision systems that aim at flexibility and robustness. One is tailored for content-based image retrieval, the others are cognitive vision systems that constitute prototypes of visual active memories which evaluate, gather, and integrate contextual knowledge for visual analysis. All three systems are designed to interact with human users. After we will have discussed adaptive content-based image retrieval and object and action recognition in an office environment, the issue of assessing cognitive systems will be raised. Experiences from psychologically evaluated human-machine interactions will be reported and the promising potential of psychologically-based usability experiments will be stressed

U SMART ZONE – Creating highly realistic virtual environment for vehicle-in-the-loop simulations

Developing Advanced Driver Assistance Systems (ADAS) and Autonomous Vehicles (AV) based on machine learning is generally an extensive and costly process. Due to the increasing complexity of autonomous systems, the need for extensive testing and validation arises. In recent years, computer simulation has been used for these purposes. Performing realistic simulations, especially for the purpose of computer vision-based systems, requires a high-quality, almost photorealistic virtual environment. This paper introduces U SMART ZONE, a high-fidelity virtual model of the Severní Terasa district in Ústí nad Labem, Czech Republic comprising of more than 7.5 km of drivable roads with a total area of approximately 1.4 km2 for human-in-the-loop and hardware-in-the-loop (HiL) simulations

Guest Editorial: Social and human aspects of cyber-physical systems

open6siIn the vision of Industry 4.0, the new industrial revolution is a revolution of cyber-physical systems, of which the Internet of Things forms a key foundation that has a great impact on the way people live, and the way businesses are organised. Cyber-physical systems are often considered feedback systems that integrate computation, networking, and physical processes, and more recently with ‘human-in-the-loop’ as one of the key research topics. The advances in social computing have connected human-inthe-loop in cyber-social systems such as Facebook and Twitter, while their social-physical activities are supported by the cyberphysical systems on or near their bodies and in their interconnected environments. Cyber-physical systems become an integral part of social-cyber-physical systems (SCPS) that weave into the sociotechnical fabric of human society. These hybrid systems, exhibiting both continuous (in physical and social spaces) and discrete (in cyberspaces) dynamic behaviour, give rise to not only new opportunities but also new challenges in designing products and services where human and technical aspects are massively intertwined. This Special Issue aims to present state-of-the-art research attempts and results on the topic of SCPS.openopenHu J.; Liang R.-H.; Shih C.-S.; Catala A.; Marcenaro L.; Osawa H.Hu, J.; Liang, R. -H.; Shih, C. -S.; CATALA MALLOFRE, Andreu; Marcenaro, L.; Osawa, H

A cognitive ego-vision system for interactive assistance

With increasing computational power and decreasing size, computers nowadays are already wearable and mobile. They become attendant of peoples' everyday life. Personal digital assistants and mobile phones equipped with adequate software gain a lot of interest in public, although the functionality they provide in terms of assistance is little more than a mobile databases for appointments, addresses, to-do lists and photos. Compared to the assistance a human can provide, such systems are hardly to call real assistants. The motivation to construct more human-like assistance systems that develop a certain level of cognitive capabilities leads to the exploration of two central paradigms in this work. The first paradigm is termed cognitive vision systems. Such systems take human cognition as a design principle of underlying concepts and develop learning and adaptation capabilities to be more flexible in their application. They are embodied, active, and situated. Second, the ego-vision paradigm is introduced as a very tight interaction scheme between a user and a computer system that especially eases close collaboration and assistance between these two. Ego-vision systems (EVS) take a user's (visual) perspective and integrate the human in the system's processing loop by means of a shared perception and augmented reality. EVSs adopt techniques of cognitive vision to identify objects, interpret actions, and understand the user's visual perception. And they articulate their knowledge and interpretation by means of augmentations of the user's own view. These two paradigms are studied as rather general concepts, but always with the goal in mind to realize more flexible assistance systems that closely collaborate with its users. This work provides three major contributions. First, a definition and explanation of ego-vision as a novel paradigm is given. Benefits and challenges of this paradigm are discussed as well. Second, a configuration of different approaches that permit an ego-vision system to perceive its environment and its user is presented in terms of object and action recognition, head gesture recognition, and mosaicing. These account for the specific challenges identified for ego-vision systems, whose perception capabilities are based on wearable sensors only. Finally, a visual active memory (VAM) is introduced as a flexible conceptual architecture for cognitive vision systems in general, and for assistance systems in particular. It adopts principles of human cognition to develop a representation for information stored in this memory. So-called memory processes continuously analyze, modify, and extend the content of this VAM. The functionality of the integrated system emerges from their coordinated interplay of these memory processes. An integrated assistance system applying the approaches and concepts outlined before is implemented on the basis of the visual active memory. The system architecture is discussed and some exemplary processing paths in this system are presented and discussed. It assists users in object manipulation tasks and has reached a maturity level that allows to conduct user studies. Quantitative results of different integrated memory processes are as well presented as an assessment of the interactive system by means of these user studies

Active Vision-Based Guidance with a Mobile Device for People with Visual Impairments

The aim of this research is to determine whether an active-vision system with a human-in-the-loop can be implemented to guide a user with visual impairments in finding a target object. Active vision techniques have successfully been applied to various electro-mechanical object search and exploration systems to boost their effectiveness at a given task. However, despite the potential of intelligent visual sensor arrays to enhance a user’s vision capabilities and alleviate some of the impacts that visual deficiencies have on their day-to-day lives, active vision techniques with human-in-the-loop remains an open research topic. In this thesis, an active guidance system is presented, which uses visual input from an object detector and an initial understanding of a typical room layout to generate navigation cues that assist a user with visual impairments in finding a target object. A complete guidance system prototype is implemented, along with a new audio-based interface and a state-of-the-art object detector, onto a mobile device and evaluated with a set of users in real environments. The results show that an active guidance approach performs well compared to other unguided solutions. This research highlights the potential benefits of the proposed active guidance controller and audio interface, which could enhance current vision-based guidance systems and travel aids for people with visual impairments

Learn to automate GUI tasks from demonstration

This thesis explores and extends Computer Vision applications in the context of Graphical User Interface (GUI) environments to address the challenges of Programming by Demonstration (PbD). The challenges are explored in PbD which could be addressed through innovations in Computer Vision, when GUIs are treated as an application domain, analogous to automotive or factory settings. Existing PbD systems were restricted by domain applications or special application interfaces. Although they use the term Demonstration, the systems did not actually see what the user performs. Rather they listen to the demonstrations through internal communications via operating system. Machine Vision and Human in the Loop Machine Learning are used to circumvent many restrictions, allowing the PbD system to watch the demonstration like another human observer would. This thesis will demonstrate that our prototype PbD systems allow non-programmer users to easily create their own automation scripts for their repetitive and looping tasks. Our PbD systems take their input from sequences of screenshots, and sometimes from easily available keyboard and mouse sniffer software. It will also be shown that the problem of inconsistent human demonstration can be remedied with our proposed Human in the Loop Computer Vision techniques. Lastly, the problem is extended to learn from demonstration videos. Due to the sheer complexity of computer desktop GUI manipulation videos, attention is focused on the domain of video game environments. The initial studies illustrate that it is possible to teach a computer to watch gameplay videos and to estimate what buttons the user pressed

Towards Flexible and Cognitive Production—Addressing the Production Challenges

Globalization in the field of industry is fostering the need for cognitive production systems. To implement modern concepts that enable tools and systems for such a cognitive production system, several challenges on the shop floor level must first be resolved. This paper discusses the implementation of selected cognitive technologies on a real industrial case-study of a construction machine manufacturer. The partner company works on the concept of mass customization but utilizes manual labour for the high-variety assembly stations or lines. Sensing and guidance devices are used to provide information to the worker and also retrieve and monitor the working, with respecting data privacy policies. Next, a specified process of data contextualization, visual analytics, and causal discovery is used to extract useful information from the retrieved data via sensors. Communications and safety systems are explained further to complete the loop of implementation of cognitive entities on a manual assembly line. This deepened involvement of cognitive technologies are human-centered, rather than automated systems. The explained cognitive technologies enhance human interaction with the processes and ease the production methods. These concepts form a quintessential vision for an effective assembly line. This paper revolutionizes the existing industry 4.0 with an even-intensified human–machine interaction and moving towards cognitivity

Instance Selection Mechanisms for Human-in-the-Loop Systems in Few-Shot Learning

Business analytics and machine learning have become essential success factors for various industries - with the downside of cost-intensive gathering and labeling of data. Few-shot learning addresses this challenge and reduces data gathering and labeling costs by learning novel classes with very few labeled data. In this paper, we design a human-in-the-loop (HITL) system for few-shot learning and analyze an extensive range of mechanisms that can be used to acquire human expert knowledge for instances that have an uncertain prediction outcome. We show that the acquisition of human expert knowledge significantly accelerates the few-shot model performance given a negligible labeling effort. We validate our findings in various experiments on a benchmark dataset in computer vision and real-world datasets. We further demonstrate the cost-effectiveness of HITL systems for few-shot learning. Overall, our work aims at supporting researchers and practitioners in effectively adapting machine learning models to novel classes at reduced costs

Instance Selection Mechanisms for Human-in-the-Loop Systems in Few-Shot Learning

Business analytics and machine learning have become essential success factors for various industries - with the downside of cost-intensive gathering and labeling of data. Few-shot learning addresses this challenge and reduces data gathering and labeling costs by learning novel classes with very few labeled data. In this paper, we design a human-in-the-loop (HITL) system for few-shot learning and analyze an extensive range of mechanisms that can be used to acquire human expert knowledge for instances that have an uncertain prediction outcome. We show that the acquisition of human expert knowledge significantly accelerates the few-shot model performance given a negligible labeling effort. We validate our findings in various experiments on a benchmark dataset in computer vision and real-world datasets. We further demonstrate the cost-effectiveness of HITL systems for few-shot learning. Overall, our work aims at supporting researchers and practitioners in effectively adapting machine learning models to novel classes at reduced costs.Comment: International Conference on Wirtschaftsinformatik, 14 page

Quantifying Pilot Visual Attention in Low Visibility Terminal Operations

Quantifying pilot visual behavior allows researchers to determine not only where a pilot is looking and when, but holds implications for specific behavioral tracking when these data are coupled with flight technical performance. Remote eye tracking systems have been integrated into simulators at NASA Langley with effectively no impact on the pilot environment. This paper discusses the installation and use of a remote eye tracking system. The data collection techniques from a complex human-in-the-loop (HITL) research experiment are discussed; especially, the data reduction algorithms and logic to transform raw eye tracking data into quantified visual behavior metrics, and analysis methods to interpret visual behavior. The findings suggest superior performance for Head-Up Display (HUD) and improved attentional behavior for Head-Down Display (HDD) implementations of Synthetic Vision System (SVS) technologies for low visibility terminal area operations. Keywords: eye tracking, flight deck, NextGen, human machine interface, aviatio