Visual Recognition and Synthesis of Human-Object Interactions

The ability to perceive and understand people's actions enables humans to efficiently communicate and collaborate in society. Endowing machines with such ability is an important step for building assistive and socially-aware robots. Despite such significance, the problem poses a great challenge and the current state of the art is still nowhere close to human-level performance. This dissertation drives progress on visual action understanding in the scope of human-object interactions (HOI), a major branch of human actions that dominates our everyday life. Specifically, we address the challenges of two important tasks: visual recognition and visual synthesis. The first part of this dissertation considers the recognition task. The main bottleneck of current research is a lack of proper benchmark, since existing action datasets contain only a small number of categories with limited diversity. To this end, we set out to construct a large-scale benchmark for HOI recognition. We first tackle the problem of establishing the vocabulary for human-object interactions, by investigating a variety of automatic approaches as well as a crowdsourcing approach that collects human labeled categories. Given the vocabulary, we then construct a large-scale image dataset of human-object interactions by annotating web images through online crowdsourcing. The new "HICO" dataset surpasses prior datasets in term of both the number of images and action categories by one order of magnitude. The introduction of HICO enables us to benchmark state-of-the-art recognition approaches and also shed light on new challenges in the realm of large-scale HOI recognition. We further discover that visual features of humans, objects, as well as their spatial relations play a central role in the representation of interaction, and the combination of three can improve the recognition outcome. The second part of this dissertation considers the synthesis task, and focuses particularly on the synthesis of body motion. The central goal is: given an image of a scene, synthesize the course of an action conditioned on the observed scene. Such capability can predict possible actions afforded by the scene, and will facilitate efficient reactions in human-robot interactions. We investigate two types of synthesis tasks: semantic-driven synthesis and goal-driven synthesis. For semantic-driven synthesis, we study the forecasting of human dynamics from a static image. We propose a novel deep neural network architecture that extracts semantic information from the image and use it to predict future body movement. For goal-directed synthesis, we study the synthesis of motion defined by human-object interactions. We focus on one particular class of interactions—a person sitting onto a chair. To ensure realistic motion from physical interactions, we leverage a physics simulated environment that contains a humanoid and chair model. We propose a novel reinforcement learning framework, and show that the synthesized motion can generalize to different initial human-chair configurations. At the end of this dissertation, we also contribute a new approach to temporal action localization, an essential task in video action understanding. We address the shortcomings of prior Faster R-CNN based approaches, and show state-of-the-art performance on standard benchmarks.PHDComputer Science & EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/150045/1/ywchao_1.pd

15-01 Effect of Cycling Skills on Bicycle Safety and Comfort Associated with Bicycle Infrastructure and Environment

This study seeks to improve the methodology for determining the relationship between cycling dynamic performance and roadway environment characteristics across different bicyclists’ skill levels. To achieve the goal of this study, an Instrumented Probe Bicycle (IPB) equipped with various sensors was built. A naturalistic field experiment, including intersections, roundabout, alignment changes, and different road surface conditions, was conducted. Two self-reported questionnaires were used in order to obtain each participant’s skill level as well as perception on the level of cycling comfortability. The Cycling Comfortability Index (CCI) was derived from the probabilistic outcome of an Ordered Probit Model, which describes the relationship between bicycle dynamics and level of comfortability. Fault Tree Analysis (FTA), a technique widely used to measure the risk of a fault event occurrence in a system, was employed to integrate mobility and comfortability. The estimation results showed that the probability of a fault event occurrence is related to the bicyclist’s experience level, incline of the roadway, and quality of the road surface. It was also found that cycling comfort level is significantly affected by the average y-axis acceleration and the mean absolute deviation of the z-axis velocity. The results of this study have practical implications for improving bicyclist perceptions on comfortability and for increasing safety for cyclists

A computer vision system for detecting and analysing critical events in cities

Whether for commuting or leisure, cycling is a growing transport mode in many cities worldwide. However, it is still perceived as a dangerous activity. Although serious incidents related to cycling leading to major injuries are rare, the fear of getting hit or falling hinders the expansion of cycling as a major transport mode. Indeed, it has been shown that focusing on serious injuries only touches the tip of the iceberg. Near miss data can provide much more information about potential problems and how to avoid risky situations that may lead to serious incidents. Unfortunately, there is a gap in the knowledge in identifying and analysing near misses. This hinders drawing statistically significant conclusions to provide measures for the built-environment that ensure a safer environment for people on bikes. In this research, we develop a method to detect and analyse near misses and their risk factors using artificial intelligence. This is accomplished by analysing video streams linked to near miss incidents within a novel framework relying on deep learning and computer vision. This framework automatically detects near misses and extracts their risk factors from video streams before analysing their statistical significance. It also provides practical solutions implemented in a camera with embedded AI (URBAN-i Box) and a cloud-based service (URBAN-i Cloud) to tackle the stated issue in the real-world settings for use by researchers, policy-makers, or citizens. The research aims to provide human-centred evidence that may enable policy-makers and planners to provide a safer built environment for cycling in London, or elsewhere. More broadly, this research aims to contribute to the scientific literature with the theoretical and empirical foundations of a computer vision system that can be utilised for detecting and analysing other critical events in a complex environment. Such a system can be applied to a wide range of events, such as traffic incidents, crime or overcrowding

Computational interaction models for automated vehicles and cyclists

Cyclists’ safety is crucial for a sustainable transport system. Cyclists are considered vulnerableroad users because they are not protected by a physical compartment around them. In recentyears, passenger car occupants’ share of fatalities has been decreasing, but that of cyclists hasactually increased. Most of the conflicts between cyclists and motorized vehicles occur atcrossings where they cross each other’s path. Automated vehicles (AVs) are being developedto increase traffic safety and reduce human errors in driving tasks, including when theyencounter cyclists at intersections. AVs use behavioral models to predict other road user’sbehaviors and then plan their path accordingly. Thus, there is a need to investigate how cyclistsinteract and communicate with motorized vehicles at conflicting scenarios like unsignalizedintersections. This understanding will be used to develop accurate computational models ofcyclists’ behavior when they interact with motorized vehicles in conflict scenarios.The overall goal of this thesis is to investigate how cyclists communicate and interact withmotorized vehicles in the specific conflict scenario of an unsignalized intersection. In the firstof two studies, naturalistic data was used to model the cyclists’ decision whether to yield to apassenger car at an unsignalized intersection. Interaction events were extracted from thetrajectory dataset, and cyclists’ behavioral cues were added from the sensory data. Bothcyclists’ kinematics and visual cues were found to be significant in predicting who crossed theintersection first. The second study used a cycling simulator to acquire in-depth knowledgeabout cyclists’ behavioral patterns as they interacted with an approaching vehicle at theunsignalized intersection. Two independent variables were manipulated across the trials:difference in time to arrival at the intersection (DTA) and visibility condition (field of viewdistance). Results from the mixed effect logistic model showed that only DTA affected thecyclist’s decision to cross before the vehicle. However, increasing the visibility at theintersection reduced the severity of the cyclists’ braking profiles. Both studies contributed tothe development of computational models of cyclist behavior that may be used to support safeautomated driving.Future work aims to find differences in cyclists’ interactions with different vehicle types, suchas passenger cars, taxis, and trucks. In addition, the interaction process may also be evaluatedfrom the driver’s perspective by using a driving simulator instead of a riding simulator. Thissetup would allow us to investigate how drivers respond to cyclists at the same intersection.The resulting data will contribute to the development of accurate predictive models for AVs

spinfortec2022 : Tagungsband zum 14. Symposium der Sektion Sportinformatik und Sporttechnologie der Deutschen Vereinigung für Sportwissenschaft (dvs), Chemnitz 29. - 30. September 2022

Dieser Tagungsband enthält die Beiträge aller Vorträge und Posterpräsentationen des 14. Symposiums der Sektion Sportinformatik und Sporttechnologie der Deutschen Vereinigung für Sportwissenschaft (dvs) an der Technischen Universität Chemnitz (29.-30. September 2022). Mit dem Ziel, das Forschungsfeld der Sportinformatik und Sporttechnologie voranzubringen, wurden knapp 20 vierseitige Beiträge eingereicht und in den Sessions Informations- und Feedbacksysteme im Sport, Digitale Bewegung: Datenerfassung, Analyse und Algorithmen sowie Sportgeräteentwicklung: Materialien, Konstruktion, Tests vorgestellt.This conference volume contains the contributions of all oral and poster presentations of the 14th Symposium of the Section Sport Informatics and Engineering of the German Association for Sport Science (dvs) at Chemnitz University of Technology (September 29-30, 2022). With the goal of advancing the research field of sports informatics and sports technology, nearly 20 four-page papers were submitted and presented in the sessions Information and Feedback Systems in Sport, Digital Movement: Data Acquisition, Analysis and Algorithms, and Sports Equipment Development: Materials, Construction, Testing

Fast object detection

The ultimate goal of computer vision is to understand images. We describe methods to understand images at two levels. One is at the level of description of images which we produce using sentences. These sentences talk about the things that are present in the image and about where they are and what they are doing. Then we ask in what ways should we describe images. We introduce visual phrases that are composite chunks of meaning. We show that object detectors could be better at detecting some visual phrases than detecting single objects. This process of image understanding needs to use a lot of detectors. Running conventional object detectors at the rate required for image understanding could be very slow. We study fast object detection from an engineering perspective. We argue that a desirable object detector must: (1) be able to work with legacy templates; (2) be random access; (3) be able to trade accuracy versus speed; (4) have any-time property. We describe a method to have all of these features together for a fast detector. We apply these techniques to deformable parts model object detectors and show two orders of magnitude speed-up while adding their desirable features. We finally investigate the consequences of this architecture with a view of improving convolutional neural networks