370 research outputs found
Human Motion Trajectory Prediction: A Survey
With growing numbers of intelligent autonomous systems in human environments,
the ability of such systems to perceive, understand and anticipate human
behavior becomes increasingly important. Specifically, predicting future
positions of dynamic agents and planning considering such predictions are key
tasks for self-driving vehicles, service robots and advanced surveillance
systems. This paper provides a survey of human motion trajectory prediction. We
review, analyze and structure a large selection of work from different
communities and propose a taxonomy that categorizes existing methods based on
the motion modeling approach and level of contextual information used. We
provide an overview of the existing datasets and performance metrics. We
discuss limitations of the state of the art and outline directions for further
research.Comment: Submitted to the International Journal of Robotics Research (IJRR),
37 page
Focus for Free in Density-Based Counting
This work considers supervised learning to count from images and their
corresponding point annotations. Where density-based counting methods typically
use the point annotations only to create Gaussian-density maps, which act as
the supervision signal, the starting point of this work is that point
annotations have counting potential beyond density map generation. We introduce
two methods that repurpose the available point annotations to enhance counting
performance. The first is a counting-specific augmentation that leverages point
annotations to simulate occluded objects in both input and density images to
enhance the network's robustness to occlusions. The second method, foreground
distillation, generates foreground masks from the point annotations, from which
we train an auxiliary network on images with blacked-out backgrounds. By doing
so, it learns to extract foreground counting knowledge without interference
from the background. These methods can be seamlessly integrated with existing
counting advances and are adaptable to different loss functions. We demonstrate
complementary effects of the approaches, allowing us to achieve robust counting
results even in challenging scenarios such as background clutter, occlusion,
and varying crowd densities. Our proposed approach achieves strong counting
results on multiple datasets, including ShanghaiTech Part\_A and Part\_B,
UCF\_QNRF, JHU-Crowd++, and NWPU-Crowd.Comment: 18 page
Redesigning Multi-Scale Neural Network for Crowd Counting
Perspective distortions and crowd variations make crowd counting a
challenging task in computer vision. To tackle it, many previous works have
used multi-scale architecture in deep neural networks (DNNs). Multi-scale
branches can be either directly merged (e.g. by concatenation) or merged
through the guidance of proxies (e.g. attentions) in the DNNs. Despite their
prevalence, these combination methods are not sophisticated enough to deal with
the per-pixel performance discrepancy over multi-scale density maps. In this
work, we redesign the multi-scale neural network by introducing a hierarchical
mixture of density experts, which hierarchically merges multi-scale density
maps for crowd counting. Within the hierarchical structure, an expert
competition and collaboration scheme is presented to encourage contributions
from all scales; pixel-wise soft gating nets are introduced to provide
pixel-wise soft weights for scale combinations in different hierarchies. The
network is optimized using both the crowd density map and the local counting
map, where the latter is obtained by local integration on the former.
Optimizing both can be problematic because of their potential conflicts. We
introduce a new relative local counting loss based on relative count
differences among hard-predicted local regions in an image, which proves to be
complementary to the conventional absolute error loss on the density map.
Experiments show that our method achieves the state-of-the-art performance on
five public datasets, i.e. ShanghaiTech, UCF_CC_50, JHU-CROWD++, NWPU-Crowd and
Trancos.Comment: IEEE Transactions on Image Processin
A Survey on Human-aware Robot Navigation
Intelligent systems are increasingly part of our everyday lives and have been
integrated seamlessly to the point where it is difficult to imagine a world
without them. Physical manifestations of those systems on the other hand, in
the form of embodied agents or robots, have so far been used only for specific
applications and are often limited to functional roles (e.g. in the industry,
entertainment and military fields). Given the current growth and innovation in
the research communities concerned with the topics of robot navigation,
human-robot-interaction and human activity recognition, it seems like this
might soon change. Robots are increasingly easy to obtain and use and the
acceptance of them in general is growing. However, the design of a socially
compliant robot that can function as a companion needs to take various areas of
research into account. This paper is concerned with the navigation aspect of a
socially-compliant robot and provides a survey of existing solutions for the
relevant areas of research as well as an outlook on possible future directions.Comment: Robotics and Autonomous Systems, 202
From pixels to people : recovering location, shape and pose of humans in images
Humans are at the centre of a significant amount of research in computer vision. Endowing machines with the ability to perceive people from visual data is an immense scientific challenge with a high degree of direct practical relevance. Success in automatic perception can be measured at different levels of abstraction, and this will depend on which intelligent behaviour we are trying to replicate: the ability to localise persons in an image or in the environment, understanding how persons are moving at the skeleton and at the surface level, interpreting their interactions with the environment including with other people, and perhaps even anticipating future actions. In this thesis we tackle different sub-problems of the broad research area referred to as "looking at people", aiming to perceive humans in images at different levels of granularity. We start with bounding box-level pedestrian detection: We present a retrospective analysis of methods published in the decade preceding our work, identifying various strands of research that have advanced the state of the art. With quantitative exper- iments, we demonstrate the critical role of developing better feature representations and having the right training distribution. We then contribute two methods based on the insights derived from our analysis: one that combines the strongest aspects of past detectors and another that focuses purely on learning representations. The latter method outperforms more complicated approaches, especially those based on hand- crafted features. We conclude our work on pedestrian detection with a forward-looking analysis that maps out potential avenues for future research. We then turn to pixel-level methods: Perceiving humans requires us to both separate them precisely from the background and identify their surroundings. To this end, we introduce Cityscapes, a large-scale dataset for street scene understanding. This has since established itself as a go-to benchmark for segmentation and detection. We additionally develop methods that relax the requirement for expensive pixel-level annotations, focusing on the task of boundary detection, i.e. identifying the outlines of relevant objects and surfaces. Next, we make the jump from pixels to 3D surfaces, from localising and labelling to fine-grained spatial understanding. We contribute a method for recovering 3D human shape and pose, which marries the advantages of learning-based and model- based approaches. We conclude the thesis with a detailed discussion of benchmarking practices in computer vision. Among other things, we argue that the design of future datasets should be driven by the general goal of combinatorial robustness besides task-specific considerations.Der Mensch steht im Zentrum vieler Forschungsanstrengungen im Bereich des maschinellen Sehens. Es ist eine immense wissenschaftliche Herausforderung mit hohem unmittelbarem Praxisbezug, Maschinen mit der Fähigkeit auszustatten, Menschen auf der Grundlage von visuellen Daten wahrzunehmen. Die automatische Wahrnehmung kann auf verschiedenen Abstraktionsebenen erfolgen. Dies hängt davon ab, welches intelligente Verhalten wir nachbilden wollen: die Fähigkeit, Personen auf der Bildfläche oder im 3D-Raum zu lokalisieren, die Bewegungen von Körperteilen und Körperoberflächen zu erfassen, Interaktionen einer Person mit ihrer Umgebung einschließlich mit anderen Menschen zu deuten, und vielleicht sogar zukünftige Handlungen zu antizipieren. In dieser Arbeit beschäftigen wir uns mit verschiedenen Teilproblemen die dem breiten Forschungsgebiet "Betrachten von Menschen" gehören. Beginnend mit der Fußgängererkennung präsentieren wir eine Analyse von Methoden, die im Jahrzehnt vor unserem Ausgangspunkt veröffentlicht wurden, und identifizieren dabei verschiedene Forschungsstränge, die den Stand der Technik vorangetrieben haben. Unsere quantitativen Experimente zeigen die entscheidende Rolle sowohl der Entwicklung besserer Bildmerkmale als auch der Trainingsdatenverteilung. Anschließend tragen wir zwei Methoden bei, die auf den Erkenntnissen unserer Analyse basieren: eine Methode, die die stärksten Aspekte vergangener Detektoren kombiniert, eine andere, die sich im Wesentlichen auf das Lernen von Bildmerkmalen konzentriert. Letztere übertrifft kompliziertere Methoden, insbesondere solche, die auf handgefertigten Bildmerkmalen basieren. Wir schließen unsere Arbeit zur Fußgängererkennung mit einer vorausschauenden Analyse ab, die mögliche Wege für die zukünftige Forschung aufzeigt. Anschließend wenden wir uns Methoden zu, die Entscheidungen auf Pixelebene betreffen. Um Menschen wahrzunehmen, müssen wir diese sowohl praezise vom Hintergrund trennen als auch ihre Umgebung verstehen. Zu diesem Zweck führen wir Cityscapes ein, einen umfangreichen Datensatz zum Verständnis von Straßenszenen. Dieser hat sich seitdem als Standardbenchmark für Segmentierung und Erkennung etabliert. Darüber hinaus entwickeln wir Methoden, die die Notwendigkeit teurer Annotationen auf Pixelebene reduzieren. Wir konzentrieren uns hierbei auf die Aufgabe der Umgrenzungserkennung, d. h. das Erkennen der Umrisse relevanter Objekte und Oberflächen. Als nächstes machen wir den Sprung von Pixeln zu 3D-Oberflächen, vom Lokalisieren und Beschriften zum präzisen räumlichen Verständnis. Wir tragen eine Methode zur Schätzung der 3D-Körperoberfläche sowie der 3D-Körperpose bei, die die Vorteile von lernbasierten und modellbasierten Ansätzen vereint. Wir schließen die Arbeit mit einer ausführlichen Diskussion von Evaluationspraktiken im maschinellen Sehen ab. Unter anderem argumentieren wir, dass der Entwurf zukünftiger Datensätze neben aufgabenspezifischen Überlegungen vom allgemeinen Ziel der kombinatorischen Robustheit bestimmt werden sollte
Understanding Video Transformers for Segmentation: A Survey of Application and Interpretability
Video segmentation encompasses a wide range of categories of problem
formulation, e.g., object, scene, actor-action and multimodal video
segmentation, for delineating task-specific scene components with pixel-level
masks. Recently, approaches in this research area shifted from concentrating on
ConvNet-based to transformer-based models. In addition, various
interpretability approaches have appeared for transformer models and video
temporal dynamics, motivated by the growing interest in basic scientific
understanding, model diagnostics and societal implications of real-world
deployment. Previous surveys mainly focused on ConvNet models on a subset of
video segmentation tasks or transformers for classification tasks. Moreover,
component-wise discussion of transformer-based video segmentation models has
not yet received due focus. In addition, previous reviews of interpretability
methods focused on transformers for classification, while analysis of video
temporal dynamics modelling capabilities of video models received less
attention. In this survey, we address the above with a thorough discussion of
various categories of video segmentation, a component-wise discussion of the
state-of-the-art transformer-based models, and a review of related
interpretability methods. We first present an introduction to the different
video segmentation task categories, their objectives, specific challenges and
benchmark datasets. Next, we provide a component-wise review of recent
transformer-based models and document the state of the art on different video
segmentation tasks. Subsequently, we discuss post-hoc and ante-hoc
interpretability methods for transformer models and interpretability methods
for understanding the role of the temporal dimension in video models. Finally,
we conclude our discussion with future research directions
Multi-Scale Architectures for Human Pose Estimation
In this dissertation we present multiple state-of-the-art deep learning methods for computer vision tasks using multi-scale approaches for two main tasks: pose estimation and semantic segmentation. For pose estimation, we introduce a complete framework expanding the fields-of-view of the network through a multi-scale approach, resulting in a significant increasing the effectiveness of conventional backbone architectures, for several pose estimation tasks without requiring a larger network or postprocessing. Our multi-scale pose estimation framework contributes to research on methods for single-person pose estimation in both 2D and 3D scenarios, pose estimation in videos, and the estimation of multiple people’s pose in a single image for both top-down and bottom-up approaches. In addition to the enhanced capability of multi-person pose estimation generated by our multi-scale approach, our framework also demonstrates a superior capacity to expanded the more detailed and heavier task of full-body pose estimation, including up to 133 joints per person. For segmentation, we present a new efficient architecture for semantic segmentation, based on a “Waterfall” Atrous Spatial Pooling architecture, that achieves a considerable accuracy increase while decreasing the number of network parameters and memory footprint. The proposed Waterfall architecture leverages the efficiency of progressive filtering in the cascade architecture while maintaining multi-scale fields-of-view comparable to spatial pyramid configurations. Additionally, our method does not rely on a postprocessing stage with conditional random fields, which further reduces complexity and required training time
Making the Most of Crowd Information: Learning and Evaluation in AI tasks with Disagreements.
PhD ThesesThere is plenty of evidence that humans disagree on the interpretation of many
tasks in Natural Language Processing (nlp) and Computer Vision (cv), from objective
tasks rooted in linguistics such as part-of-speech tagging to more subjective (observerdependent)
tasks such as classifying an image or deciding whether a proposition follows
from a certain premise. While most learning in Artificial Intelligence (ai) still relies
on the assumption that a single interpretation, captured by the gold label, exists for
each item, a growing research body in recent years has focused on learning methods
that do not rely on this assumption. Rather, they aim to learn ranges of truth amidst
disagreement. This PhD research makes a contribution to this field of study.
Firstly, we analytically review the evidence for disagreement on nlp and cv tasks,
focusing on tasks where substantial datasets with such information have been created.
As part of this review, we also discuss the most popular approaches to training
models from datasets containing multiple judgments and group these methods
together according to their handling of disagreement. Secondly, we make three proposals
for learning with disagreement; soft-loss, multi-task learning from gold and
crowds, and automatic temperature-scaled soft-loss. Thirdly, we address one gap in
this field of study – the prevalence of hard metrics for model evaluation even when
the gold assumption is shown to be an idealization – by proposing several previously
existing metrics and novel soft metrics that do not make this assumption and analyzing
the merits and assumptions of all the metrics, hard and soft. Finally, we carry
out a systematic investigation of the key proposals in learning with disagreement by
training them across several tasks, considering several ways to evaluate the resulting
models and assessing the conditions under which each approach is effective. This is
a key contribution of this research as research in learning with disagreement do not
often test proposals across tasks, compare proposals with a variety of approaches, or
evaluate using both soft metrics and hard metrics.
The results obtained suggest, first of all, that it is essential to reach a consensus
on how to evaluate models. This is because the relative performance of the various
training methods is critically affected by the chosen form of evaluation. Secondly,
we observed a strong dataset effect. With substantial datasets, providing many judgments
by high-quality coders for each item, training directly with soft labels achieved
better results than training from aggregated or even gold labels. This result holds for
both hard and soft evaluation. But when the above conditions do not hold, leveraging
both gold and soft labels generally achieved the best results in the hard evaluation.
All datasets and models employed in this paper are freely available as supplementary
materials
Building Towards Automated Cyberbullying Detection: A Comparative Analysis
The increased use of social media between digitally anonymous users, sharing their thoughts and opinions, can facilitate participation and collaboration. However, it’s this anonymity feature which gives users freedom of speech and allows them to conduct activities without being judged by others can also encourage cyberbullying and hate speech. Predators can hide their identity and reach a wide range of audience anytime and anywhere. According to the detrimental effect of cyberbullying, there is a growing need for cyberbullying detection approaches. In this survey paper, a comparative analysis of the automated cyberbullying techniques from different perspectives is discussed including data annotation, data pre-processing and feature engineering. In addition, the importance of emojis in expressing emotions as well as their influence on sentiment classification and text comprehension lead us to discuss the role of incorporating emojis in the process of cyberbullying detection and their influence on the detection performance. Furthermore, the different domains for using Self-Supervised Learning (SSL) as an annotation technique for cyberbullying detection is explored
- …