Semantic Trajectories:Computing and Understanding Mobility Data

Thanks to the rapid development of mobile sensing technologies (like GPS, GSM, RFID, accelerometer, gyroscope, sound and other sensors in smartphones), the large-scale capture of evolving positioning data (called mobility data or trajectories) generated by moving objects with embedded sensors has become easily feasible, both technically and economically. We have already entered a world full of trajectories. The state-of-the-art on trajectory, either from the moving object database area or in the statistical analysis viewpoint, has built a bunch of sophisticated techniques for trajectory data ad-hoc storage, indexing, querying and mining etc. However, most of these existing methods mainly focus on a spatio-temporal viewpoint of mobility data, which means they analyze only the geometric movement of trajectories (e.g., the raw ‹x, y, t› sequential data) without enough consideration on the high-level semantics that can better understand the underlying meaningful movement behaviors. Addressing this challenging issue for better understanding movement behaviors from the raw mobility data, this doctoral work aims at providing a high-level modeling and computing methodology for semantically abstracting the rapidly increasing mobility data. Therefore, we bring top-down semantic modeling and bottom-up data computing together and establish a new concept called "semantic trajectories" for mobility data representation and understanding. As the main novelty contribution, this thesis provides a rich, holistic, heterogeneous and application-independent methodology for computing semantic trajectories to better understand mobility data at different levels. In details, this methodology is composed of five main parts with dedicated contributions. Semantic Trajectory Modeling. By investigating trajectory modeling requirements to better understand mobility data, this thesis first designs a hybrid spatio-semantic trajectory model that represents mobility with rich data abstraction at different levels, i.e., from the low-level spatio-temporal trajectory to the intermediate-level structured trajectory, and finally to the high-level semantic trajectory. In addition, a semantic based ontological framework has also been designed and applied for querying and reasoning on trajectories. Offline Trajectory Computing. To utilize the hybrid model, the thesis complementarily designs a holistic trajectory computing platform with dedicated algorithms for reconstructing trajectories at different levels. The platform can preprocess collected mobility data (i.e., raw movement tracks like GPS feeds) in terms of data cleaning/compression etc., identify individual trajectories, and segment them into structurally meaningful trajectory episodes. Therefore, this trajectory computing platform can construct spatio-temporal trajectories and structured trajectories from the raw mobility data. Such computing platform is initially designed as an offline solution which is supposed to analyze past trajectories via a batch procedure. Trajectory Semantic Annotation. To achieve the final semantic level for better understanding mobility data, this thesis additionally designs a semantic annotation platform that can enrich trajectories with third party sources that are composed of geographic background information and application domain knowledge, to further infer more meaningful semantic trajectories. Such annotation platform is application-independent that can annotate various trajectories (e.g., mobility data of people, vehicle and animals) with heterogeneous data sources of semantic knowledge (e.g., third party sources in any kind of geometric shapes like point, line and region) that can help trajectory enrichment. Online Trajectory Computing. In addition to the offline trajectory computing for analyzing past trajectories, this thesis also contributes to dealing with ongoing trajectories in terms of real-time trajectory computing from movement data streams. The online trajectory computing platform is capable of providing real-life trajectory data cleaning, compression, and segmentation over streaming movement data. In addition, the online platform explores the functionality of online tagging to achieve fully semantic-aware trajectories and further evaluate trajectory computing in a real-time setting. Mining Trajectories from Multi-Sensors. Previously, the focus is on computing semantic trajectories using single-sensory data (i.e., GPS feeds), where most datasets are from moving objects with wearable GPS-embedded sensors (e.g., mobility data of animal, vehicle and people tracking). In addition, we explore the problem of mining people trajectories using multi-sensory feeds from smartphones (GPS, gyroscope, accelerometer etc). The research results reveal that the combination of two sensors (GPS+accelerometer) can significantly infer a complete life-cycle semantic trajectories of people's daily behaviors, both outdoor movement via GPS and indoor activities via accelerometer

Intelligent Transportation Related Complex Systems and Sensors

Building around innovative services related to different modes of transport and traffic management, intelligent transport systems (ITS) are being widely adopted worldwide to improve the efficiency and safety of the transportation system. They enable users to be better informed and make safer, more coordinated, and smarter decisions on the use of transport networks. Current ITSs are complex systems, made up of several components/sub-systems characterized by time-dependent interactions among themselves. Some examples of these transportation-related complex systems include: road traffic sensors, autonomous/automated cars, smart cities, smart sensors, virtual sensors, traffic control systems, smart roads, logistics systems, smart mobility systems, and many others that are emerging from niche areas. The efficient operation of these complex systems requires: i) efficient solutions to the issues of sensors/actuators used to capture and control the physical parameters of these systems, as well as the quality of data collected from these systems; ii) tackling complexities using simulations and analytical modelling techniques; and iii) applying optimization techniques to improve the performance of these systems. It includes twenty-four papers, which cover scientific concepts, frameworks, architectures and various other ideas on analytics, trends and applications of transportation-related data

Spatiotemporal visual analysis of human actions

In this dissertation we propose four methods for the recognition of human activities. In all four of them, the representation of the activities is based on spatiotemporal features that are automatically detected at areas where there is a significant amount of independent motion, that is, motion that is due to ongoing activities in the scene. We propose the use of spatiotemporal salient points as features throughout this dissertation. The algorithms presented, however, can be used with any kind of features, as long as the latter are well localized and have a well-defined area of support in space and time. We introduce the utilized spatiotemporal salient points in the first method presented in this dissertation. By extending previous work on spatial saliency, we measure the variations in the information content of pixel neighborhoods both in space and time, and detect the points at the locations and scales for which this information content is locally maximized. In this way, an activity is represented as a collection of spatiotemporal salient points. We propose an iterative linear space-time warping technique in order to align the representations in space and time and propose to use Relevance Vector Machines (RVM) in order to classify each example into an action category. In the second method proposed in this dissertation we propose to enhance the acquired representations of the first method. More specifically, we propose to track each detected point in time, and create representations based on sets of trajectories, where each trajectory expresses how the information engulfed by each salient point evolves over time. In order to deal with imperfect localization of the detected points, we augment the observation model of the tracker with background information, acquired using a fully automatic background estimation algorithm. In this way, the tracker favors solutions that contain a large number of foreground pixels. In addition, we perform experiments where the tracked templates are localized on specific parts of the body, like the hands and the head, and we further augment the tracker’s observation model using a human skin color model. Finally, we use a variant of the Longest Common Subsequence algorithm (LCSS) in order to acquire a similarity measure between the resulting trajectory representations, and RVMs for classification. In the third method that we propose, we assume that neighboring salient points follow a similar motion. This is in contrast to the previous method, where each salient point was tracked independently of its neighbors. More specifically, we propose to extract a novel set of visual descriptors that are based on geometrical properties of three-dimensional piece-wise polynomials. The latter are fitted on the spatiotemporal locations of salient points that fall within local spatiotemporal neighborhoods, and are assumed to follow a similar motion. The extracted descriptors are invariant in translation and scaling in space-time. Coupling the neighborhood dimensions to the scale at which the corresponding spatiotemporal salient points are detected ensures the latter. The descriptors that are extracted across the whole dataset are subsequently clustered in order to create a codebook, which is used in order to represent the overall motion of the subjects within small temporal windows.Finally,we use boosting in order to select the most discriminative of these windows for each class, and RVMs for classification. The fourth and last method addresses the joint problem of localization and recognition of human activities depicted in unsegmented image sequences. Its main contribution is the use of an implicit representation of the spatiotemporal shape of the activity, which relies on the spatiotemporal localization of characteristic ensembles of spatiotemporal features. The latter are localized around automatically detected salient points. Evidence for the spatiotemporal localization of the activity is accumulated in a probabilistic spatiotemporal voting scheme. During training, we use boosting in order to create codebooks of characteristic feature ensembles for each class. Subsequently, we construct class-specific spatiotemporal models, which encode where in space and time each codeword ensemble appears in the training set. During testing, each activated codeword ensemble casts probabilistic votes concerning the spatiotemporal localization of the activity, according to the information stored during training. We use a Mean Shift Mode estimation algorithm in order to extract the most probable hypotheses from each resulting voting space. Each hypothesis corresponds to a spatiotemporal volume which potentially engulfs the activity, and is verified by performing action category classification with an RVM classifier

Towards Efficient and Secure Intelligent Transportation Services: AI-driven Traffic Light Controller and Privacy-Preserving Mobility Data Generation

The widespread adoption of artificial intelligence (AI) and Intelligent Transportation Systems (ITS) technologies has led to the increasing application of AI-based ITS controllers, with the Traffic Signal Controller (TSC) being a prominent example. Reinforcement learning (RL) models have shown promising results for adaptively adjusting traffic light schedules in urban environments through RL-based TSCs (RL-TSCs). The real-world deployment of RL-TSCs involves three key aspects: performance, security, and data privacy. In terms of performance, RL-TSC models need to be designed with consideration for various metrics, such as fair traffic scheduling and air quality impact. To address this, our approach takes into account a multi-objective constrained learning formulation to optimize performance. However, the use of RL-TSCs for automation, by leveraging external inputs, introduces security concerns that require active research to mitigate. We address these security challenges by introducing an innovative defense mechanism. Additionally, the training of RL-TSCs relies on real-world mobility datasets, necessitating the protection of data privacy at different levels of granularity. To minimize the constraints associated with limited real data availability or privacy concerns, we introduce two distinct directions: synthetic trajectory data generation using recent generative AI methods, and location privacy models for raw mobility datasets based on differential privacy, which safeguard individual trajectories and aggregated mobility datasets.This research provides a valuable tool for evaluating the practical deployment of RL-TSCs, particularly in real-world settings where the last mile of implementation and security is paramount. By addressing the key challenges of performance, security, and data privacy, this work aims to facilitate the successful real-world deployment of AI-powered ITS controllers

Multiple-Aspect Analysis of Semantic Trajectories

This open access book constitutes the refereed post-conference proceedings of the First International Workshop on Multiple-Aspect Analysis of Semantic Trajectories, MASTER 2019, held in conjunction with the 19th European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2019, in Würzburg, Germany, in September 2019. The 8 full papers presented were carefully reviewed and selected from 12 submissions. They represent an interesting mix of techniques to solve recurrent as well as new problems in the semantic trajectory domain, such as data representation models, data management systems, machine learning approaches for anomaly detection, and common pathways identification

Action in Mind: Neural Models for Action and Intention Perception

To notice, recognize, and ultimately perceive the others’ actions and to discern the intention behind those observed actions is an essential skill for social communications and improves markedly the chances of survival. Encountering dangerous behavior, for instance, from a person or an animal requires an immediate and suitable reaction. In addition, as social creatures, we need to perceive, interpret, and judge correctly the other individual’s actions as a fundamental skill for our social life. In other words, our survival and success in adaptive social behavior and nonverbal communication depends heavily on our ability to thrive in complex social situations. However, it has been shown that humans spontaneously can decode animacy and social interactions even from strongly impoverished stimuli and this is a fundamental part of human experience that develops early in infancy and is shared with other primates. In addition, it is well established that perceptual and motor representations of actions are tightly coupled and both share common mechanisms. This coupling between action perception and action execution plays a critical role in action understanding as postulated in various studies and they are potentially important for our social cognition. This interaction likely is mediated by action-selective neurons in the superior temporal sulcus (STS), premotor and parietal cortex. STS and TPJ have been identified also as coarse neural substrate for the processing of social interactions stimuli. Despite this localization, the underlying exact neural circuits of this processing remain unclear. The aim of this thesis is to understand the neural mechanisms behind the action perception coupling and to investigate further how human brain perceive different classes of social interactions. To achieve this goal, first we introduce a neural model that provides a unifying account for multiple experiments on the interaction between action execution and action perception. The model reproduces correctly the interactions between action observation and execution in several experiments and provides a link towards electrophysiological detailed models of relevant circuits. This model might thus provide a starting point for the detailed quantitative investigation how motor plans interact with perceptual action representations at the level of single-cell mechanisms. Second we present a simple neural model that reproduces some of the key observations in psychophysical experiments about the perception of animacy and social interactions from stimuli. Even in its simple form the model proves that animacy and social interaction judgments partly might be derived by very elementary operations in hierarchical neural vision systems, without a need of sophisticated or accurate probabilistic inference

Proceedings of the 2021 Joint Workshop of Fraunhofer IOSB and Institute for Anthropomatics, Vision and Fusion Laboratory

2021, the annual joint workshop of the Fraunhofer IOSB and KIT IES was hosted at the IOSB in Karlsruhe. For a week from the 2nd to the 6th July the doctoral students extensive reports on the status of their research. The results and ideas presented at the workshop are collected in this book in the form of detailed technical reports

Proceedings of the 2021 Joint Workshop of Fraunhofer IOSB and Institute for Anthropomatics, Vision and Fusion Laboratory

2021, the annual joint workshop of the Fraunhofer IOSB and KIT IES was hosted at the IOSB in Karlsruhe. For a week from the 2nd to the 6th July the doctoral students extensive reports on the status of their research. The results and ideas presented at the workshop are collected in this book in the form of detailed technical reports

Effective demand response gathering and deployment in smart grids for intensive renewable integration using aggregation and machine learning

Tesis por compendio de publicaciones.[EN] Distributed generation, namely renewables-based technologies, have emerged as a crucial component in the transition to mitigate the effects of climate change, providing a decentralized approach to electricity production. However, the volatile behavior of distributed generation has created new challenges in maintaining system balance and reliability. In this context, the demand response concept and corresponding programs arise giving the local energy communities prominence. In demand response concept, it is expected an empowerment of the consumer in the electricity sector. This has a significant impact on grid operations and brings complex interactions due to the volatile behavior, privacy concerns, and lack of consumer knowledge in the energy market context. For this, aggregators play a crucial role addressing these challenges. It is crucial to develop tools that allow the aggregators helping consumers to make informed decisions, maximize the benefits of their flexibility resources, and contribute to the overall success of grid operations. This thesis, through innovative solutions and resorting to artificial intelligence models, addresses the integration of renewables, promoting fair participation among all demand response providers. The thesis ultimately results in an innovative decision support system - MAESTRO, the Machine learning Assisted Energy System management Tool for Renewable integration using demand respOnse. MAESTRO is composed by a set of diversified models that together contribute for handling the complexity of managing energy communities with distributed generation resources, demand response providers, energy storage systems and electric vehicles. This PhD thesis comprises a comprehensive analysis of state-of-the-art techniques, system design and development, experimental results, and key findings. In this research were published twenty-six scientific papers, in both international journals and conference proceedings. Contributions to international projects and Portuguese projects was accomplished. [ES] La generación distribuida, en particular las tecnologías basadas en energías renovables, se ha convertido en un componente crucial en la transición para mitigar los efectos del cambio climático, al proporcionar un enfoque descentralizado para la producción de electricidad. Sin embargo, el comportamiento volátil de la generación distribuida ha generado nuevos desafíos para mantener el equilibrio y la confiabilidad del sistema. En este contexto, surge el concepto de respuesta de la demanda y los programas correspondientes, otorgando prominencia a las comunidades energéticas locales. En el concepto de "respuesta a la demanda" (DR por sus siglas en inglés), se espera un empoderamiento del consumidor en el sector eléctrico. Esto tiene un impacto significativo en la operación de la red y genera interacciones complejas debido al comportamiento volátil, las preocupaciones de privacidad y la falta de conocimiento del consumidor en el contexto del mercado energético. Para esto, los agregadores desempeñan un papel crucial al abordar estos desafíos. Es fundamental desarrollar herramientas que permitan a los agregadores ayudar a los consumidores a tomar decisiones informadas, maximizar los beneficios de sus recursos de flexibilidad y contribuir al éxito general de las operaciones de la red. Esta tesis, a través de soluciones innovadoras y utilizando modelos de inteligencia artificial, aborda la integración de energías renovables, promoviendo una participación justa entre todos los proveedores de respuesta de la demanda. La tesis resulta en última instancia en un sistema de apoyo a la toma de decisiones innovador: MAESTRO, Machine learning Assisted Energy System management Tool for Renewable integration using demand respOnse. MAESTRO está compuesto por un conjunto de modelos diversificados que contribuyen juntos para manejar la complejidad de la gestión de comunidades energéticas con recursos de generación distribuida, proveedores de respuesta de la demanda, sistemas de almacenamiento de energía y vehículos eléctricos. Esta tesis de doctorado comprende un análisis exhaustivo de las técnicas de vanguardia, el diseño y desarrollo del sistema, los resultados experimentales y los hallazgos clave. En esta investigación se publicaron veintiséis artículos científicos, tanto en revistas internacionales como en actas de conferencias. Se lograron contribuciones a proyectos internacionales y proyectos portugueses. [POR] A produção distribuída, nomeadamente as tecnologias baseadas em energias renováveis, emergiram como um componente crucial na transição para mitigar os efeitos das alterações climáticas, proporcionando uma abordagem descentralizada à produção de eletricidade. No entanto, o comportamento volátil da geração distribuída criou desafios na manutenção do equilíbrio e da fiabilidade do sistema. Nesse contexto, surge o conceito de resposta à procura e os programas correspondentes, conferindo proeminência às comunidades energéticas locais. No conceito de resposta à procura, espera-se um empoderamento do consumidor no setor elétrico. Isso tem um impacto significativo nas operações da rede e gera interações complexas devido ao comportamento volátil, preocupações com a privacidade e falta de conhecimento dos consumidores no contexto do mercado energético. Para isso, os agregadores desempenham um papel crucial ao lidar com esses desafios. É fundamental desenvolver ferramentas que permitam aos agregadores ajudar os consumidores a tomar decisões informadas, maximizar os benefícios de seus recursos de flexibilidade e contribuir para o sucesso global das operações da rede. Esta tese de doutoramento, através de soluções inovadoras e recorrendo a modelos de inteligência artificial, aborda a integração de energias renováveis, promovendo uma participação justa entre todos os fornecedores de resposta à procura. A tese resulta, em última instância, num sistema inovador de apoio à tomada de decisões - MAESTRO, Machine learning Assisted Energy System management Tool for Renewable integration using demand respOnse. A ferramenta MAESTRO é composta por um conjunto de modelos diversificados que, em conjunto, contribuem para lidar com a complexidade da gestão de comunidades energéticas com recursos de geração distribuída, fornecedores de resposta à procura, sistemas de armazenamento de energia e veículos elétricos. Esta tese de doutoramento abrange uma análise abrangente de técnicas de ponta, design e desenvolvimento do sistema, resultados experimentais e descobertas-chave. Nesta pesquisa, foram publicados vinte e seis artigos científicos, tanto em revistas internacionais como em atas de conferências. Foram realizadas contribuições para projetos internacionais e projetos portugueses