Spatial representation for planning and executing robot behaviors in complex environments

Robots are already improving our well-being and productivity in different applications such as industry, health-care and indoor service applications. However, we are still far from developing (and releasing) a fully functional robotic agent that can autonomously survive in tasks that require human-level cognitive capabilities. Robotic systems on the market, in fact, are designed to address specific applications, and can only run pre-defined behaviors to robustly repeat few tasks (e.g., assembling objects parts, vacuum cleaning). They internal representation of the world is usually constrained to the task they are performing, and does not allows for generalization to other scenarios. Unfortunately, such a paradigm only apply to a very limited set of domains, where the environment can be assumed to be static, and its dynamics can be handled before deployment. Additionally, robots configured in this way will eventually fail if their "handcrafted'' representation of the environment does not match the external world. Hence, to enable more sophisticated cognitive skills, we investigate how to design robots to properly represent the environment and behave accordingly. To this end, we formalize a representation of the environment that enhances the robot spatial knowledge to explicitly include a representation of its own actions. Spatial knowledge constitutes the core of the robot understanding of the environment, however it is not sufficient to represent what the robot is capable to do in it. To overcome such a limitation, we formalize SK4R, a spatial knowledge representation for robots which enhances spatial knowledge with a novel and "functional" point of view that explicitly models robot actions. To this end, we exploit the concept of affordances, introduced to express opportunities (actions) that objects offer to an agent. To encode affordances within SK4R, we define the "affordance semantics" of actions that is used to annotate an environment, and to represent to which extent robot actions support goal-oriented behaviors. We demonstrate the benefits of a functional representation of the environment in multiple robotic scenarios that traverse and contribute different research topics relating to: robot knowledge representations, social robotics, multi-robot systems and robot learning and planning. We show how a domain-specific representation, that explicitly encodes affordance semantics, provides the robot with a more concrete understanding of the environment and of the effects that its actions have on it. The goal of our work is to design an agent that will no longer execute an action, because of mere pre-defined routine, rather, it will execute an actions because it "knows'' that the resulting state leads one step closer to success in its task

Linguistic variation across Twitter and Twitter trolling

Trolling is used to label a variety of behaviours, from the spread of misinformation and hyperbole to targeted abuse and malicious attacks. Despite this, little is known about how trolling varies linguistically and what its major linguistic repertoires and communicative functions are in comparison to general social media posts. Consequently, this dissertation collects two corpora of tweets – a general English Twitter corpus and a Twitter trolling corpus using other Twitter users’ accusations – and introduces and applies a new short-text version of Multi-Dimensional Analysis to each corpus, which is designed to identify aggregated dimensions of linguistic variation across them. The analysis finds that trolling tweets and general tweets only differ on the final dimension of linguistic variation, but share the following linguistic repertoires: “Informational versus Interactive”, “Personal versus Other Description”, and “Promotional versus Oppositional”. Moreover, the analysis compares trolling tweets to general Twitter’s dimensions and finds that trolling tweets and general tweets are remarkably more similar than they are different in their distribution along all dimensions. These findings counter various theories on trolling and problematise the notion that trolling can be detected automatically using grammatical variation. Overall, this dissertation provides empirical evidence on how trolling and general tweets vary linguistically

Selected Inductive Biases in Neural Networks To Generalize Beyond the Training Domain

Die künstlichen neuronalen Netze des computergesteuerten Sehens können mit den vielf\"altigen Fähigkeiten des menschlichen Sehens noch lange nicht mithalten. Im Gegensatz zum Menschen können künstliche neuronale Netze durch kaum wahrnehmbare Störungen durcheinandergebracht werden, es mangelt ihnen an Generalisierungsfähigkeiten über ihre Trainingsdaten hinaus und sie benötigen meist noch enorme Datenmengen für das Erlernen neuer Aufgaben. Somit sind auf neuronalen Netzen basierende Anwendungen häufig auf kleine Bereiche oder kontrollierte Umgebungen beschränkt und lassen sich schlecht auf andere Aufgaben übertragen. In dieser Dissertation, werden vier Veröffentlichungen besprochen, die sich mit diesen Einschränkungen auseinandersetzen und Algorithmen im Bereich des visuellen Repräsentationslernens weiterentwickeln. In der ersten Veröffentlichung befassen wir uns mit dem Erlernen der unabhängigen Faktoren, die zum Beispiel eine Szenerie beschreiben. Im Gegensatz zu vorherigen Arbeiten in diesem Forschungsfeld verwenden wir hierbei jedoch weniger künstliche, sondern natürlichere Datensätze. Dabei beobachten wir, dass die zeitlichen Änderungen von Szenerien beschreibenden, natürlichen Faktoren (z.B. die Positionen von Personen in einer Fußgängerzone) einer verallgemeinerten Laplace-Verteilung folgen. Wir nutzen die verallgemeinerte Laplace-Verteilung als schwaches Lernsignal, um neuronale Netze für mathematisch beweisbares Repräsentationslernen unabhängiger Faktoren zu trainieren. Wir erzielen in den disentanglement_lib Wettbewerbsdatensätzen vergleichbare oder bessere Ergebnisse als vorherige Arbeiten – dies gilt auch für die von uns beigesteuerten Datensätze, welche natürliche Faktoren beinhalten. Die zweite Veröffentlichung untersucht, ob verschiedene neuronale Netze bereits beobachtete, eine Szenerie beschreibende Faktoren generalisieren können. In den meisten bisherigen Generalisierungswettbewerben werden erst während der Testphase neue Störungsfaktoren hinzugefügt - wir hingegen garantieren, dass die für die Testphase relevanten Variationsfaktoren bereits während der Trainingsphase teilweise vorkommen. Wir stellen fest, dass die getesteten neuronalen Netze meist Schwierigkeiten haben, die beschreibenden Faktoren zu generalisieren. Anstatt die richtigen Werte der Faktoren zu bestimmen, neigen die Netze dazu, Werte in zuvor beobachteten Bereichen vorherzusagen. Dieses Verhalten ist bei allen untersuchten neuronalen Netzen recht ähnlich. Trotz ihrer begrenzten Generalisierungsfähigkeiten, können die Modelle jedoch modular sein: Obwohl sich einige Faktoren während der Trainingsphase in einem zuvor ungesehenen Wertebereich befinden, können andere Faktoren aus einem bereits bekannten Wertebereich größtenteils dennoch korrekt bestimmt werden. Die dritte Veröffentlichung präsentiert ein adversielles Trainingsverfahren für neuronale Netze. Das Verfahren ist inspiriert durch lokale Korrelationsstrukturen häufiger Bildartefakte, die z.B. durch Regen, Unschärfe oder Rauschen entstehen können. Im Klassifizierungswettbewerb ImageNet-C zeigen wir, dass mit unserer Methode trainierte Netzwerke weniger anfällig für häufige Störungen sind als einige, die mit bestehenden Methoden trainiert wurden. Schließlich stellt die vierte Veröffentlichung einen generativen Ansatz vor, der bestehende Ansätze gemäß mehrerer Robustheitsmetriken beim MNIST Ziffernklassifizierungswettbewerb übertrifft. Perzeptiv scheint unser generatives Modell im Vergleich zu früheren Ansätzen stärker auf das menschliche Sehen abgestimmt zu sein, da Bilder von Ziffern, die für unser generatives Modell mehrdeutig sind, auch für den Menschen mehrdeutig erscheinen können. Diese Arbeit liefert also Möglichkeiten zur Verbesserung der adversiellen Robustheit und der Störungstoleranz sowie Erweiterungen im Bereich des visuellen Repräsentationslernens. Somit nähern wir uns im Bereich des maschinellen Lernens weiter der Vielfalt menschlicher Fähigkeiten an.Artificial neural networks in computer vision have yet to approach the broad performance of human vision. Unlike humans, artificial networks can be derailed by almost imperceptible perturbations, lack strong generalization capabilities beyond the training data and still mostly require enormous amounts of data to learn novel tasks. Thus, current applications based on neural networks are often limited to a narrow range of controlled environments and do not transfer well across tasks. This thesis presents four publications that address these limitations and advance visual representation learning algorithms. In the first publication, we aim to push the field of disentangled representation learning towards more realistic settings. We observe that natural factors of variation describing scenes, e.g., the position of pedestrians, have temporally sparse transitions in videos. We leverage this sparseness as a weak form of learning signal to train neural networks for provable disentangled visual representation learning. We achieve competitive results on the disentanglement_lib benchmark datasets and our own contributed datasets, which include natural transitions. The second publication investigates whether various visual representation learning approaches generalize along partially observed factors of variation. In contrast to prior robustness benchmarks that add unseen types of perturbations during test time, we compose, interpolate, or extrapolate the factors observed during training. We find that the tested models mostly struggle to generalize to our proposed benchmark. Instead of predicting the correct factors, models tend to predict values in previously observed ranges. This behavior is quite common across models. Despite their limited out-of-distribution performances, the models can be fairly modular as, even though some factors are out-of-distribution, other in-distribution factors are still mostly inferred correctly. The third publication presents an adversarial noise training method for neural networks inspired by the local correlation structure of common corruptions caused by rain, blur, or noise. On the ImageNet-C classification benchmark, we show that networks trained with our method are less susceptible to common corruptions than those trained with existing methods. Finally, the fourth publication introduces a generative approach that outperforms existing approaches according to multiple robustness metrics on the MNIST digit classification benchmark. Perceptually, our generative model is more aligned with human vision compared to previous approaches, as images of digits at our model's decision boundary can also appear ambiguous to humans. In a nutshell, this work investigates ways of improving adversarial and corruption robustness, and disentanglement in visual representation learning algorithms. Thus, we alleviate some limitations in machine learning and narrow the gap towards human capabilities

The Big Five:Addressing Recurrent Multimodal Learning Data Challenges

The analysis of multimodal data in learning is a growing field of research, which has led to the development of different analytics solutions. However, there is no standardised approach to handle multimodal data. In this paper, we describe and outline a solution for five recurrent challenges in the analysis of multimodal data: the data collection, storing, annotation, processing and exploitation. For each of these challenges, we envision possible solutions. The prototypes for some of the proposed solutions will be discussed during the Multimodal Challenge of the fourth Learning Analytics & Knowledge Hackathon, a two-day hands-on workshop in which the authors will open up the prototypes for trials, validation and feedback

Multimodal Challenge: Analytics Beyond User-computer Interaction Data

This contribution describes one the challenges explored in the Fourth LAK Hackathon. This challenge aims at shifting the focus from learning situations which can be easily traced through user-computer interactions data and concentrate more on user-world interactions events, typical of co-located and practice-based learning experiences. This mission, pursued by the multimodal learning analytics (MMLA) community, seeks to bridge the gap between digital and physical learning spaces. The “multimodal” approach consists in combining learners’ motoric actions with physiological responses and data about the learning contexts. These data can be collected through multiple wearable sensors and Internet of Things (IoT) devices. This Hackathon table will confront with three main challenges arising from the analysis and valorisation of multimodal datasets: 1) the data collection and storing, 2) the data annotation, 3) the data processing and exploitation. Some research questions which will be considered in this Hackathon challenge are the following: how to process the raw sensor data streams and extract relevant features? which data mining and machine learning techniques can be applied? how can we compare two action recordings? How to combine sensor data with Experience API (xAPI)? what are meaningful visualisations for these data

A Data-driven Methodology Towards Mobility- and Traffic-related Big Spatiotemporal Data Frameworks

Human population is increasing at unprecedented rates, particularly in urban areas. This increase, along with the rise of a more economically empowered middle class, brings new and complex challenges to the mobility of people within urban areas. To tackle such challenges, transportation and mobility authorities and operators are trying to adopt innovative Big Data-driven Mobility- and Traffic-related solutions. Such solutions will help decision-making processes that aim to ease the load on an already overloaded transport infrastructure. The information collected from day-to-day mobility and traffic can help to mitigate some of such mobility challenges in urban areas. Road infrastructure and traffic management operators (RITMOs) face several limitations to effectively extract value from the exponentially growing volumes of mobility- and traffic-related Big Spatiotemporal Data (MobiTrafficBD) that are being acquired and gathered. Research about the topics of Big Data, Spatiotemporal Data and specially MobiTrafficBD is scattered, and existing literature does not offer a concrete, common methodological approach to setup, configure, deploy and use a complete Big Data-based framework to manage the lifecycle of mobility-related spatiotemporal data, mainly focused on geo-referenced time series (GRTS) and spatiotemporal events (ST Events), extract value from it and support decision-making processes of RITMOs. This doctoral thesis proposes a data-driven, prescriptive methodological approach towards the design, development and deployment of MobiTrafficBD Frameworks focused on GRTS and ST Events. Besides a thorough literature review on Spatiotemporal Data, Big Data and the merging of these two fields through MobiTraffiBD, the methodological approach comprises a set of general characteristics, technical requirements, logical components, data flows and technological infrastructure models, as well as guidelines and best practices that aim to guide researchers, practitioners and stakeholders, such as RITMOs, throughout the design, development and deployment phases of any MobiTrafficBD Framework. This work is intended to be a supporting methodological guide, based on widely used Reference Architectures and guidelines for Big Data, but enriched with inherent characteristics and concerns brought about by Big Spatiotemporal Data, such as in the case of GRTS and ST Events. The proposed methodology was evaluated and demonstrated in various real-world use cases that deployed MobiTrafficBD-based Data Management, Processing, Analytics and Visualisation methods, tools and technologies, under the umbrella of several research projects funded by the European Commission and the Portuguese Government.A população humana cresce a um ritmo sem precedentes, particularmente nas áreas urbanas. Este aumento, aliado ao robustecimento de uma classe média com maior poder económico, introduzem novos e complexos desafios na mobilidade de pessoas em áreas urbanas. Para abordar estes desafios, autoridades e operadores de transportes e mobilidade estão a adotar soluções inovadoras no domínio dos sistemas de Dados em Larga Escala nos domínios da Mobilidade e Tráfego. Estas soluções irão apoiar os processos de decisão com o intuito de libertar uma infraestrutura de estradas e transportes já sobrecarregada. A informação colecionada da mobilidade diária e da utilização da infraestrutura de estradas pode ajudar na mitigação de alguns dos desafios da mobilidade urbana. Os operadores de gestão de trânsito e de infraestruturas de estradas (em inglês, road infrastructure and traffic management operators — RITMOs) estão limitados no que toca a extrair valor de um sempre crescente volume de Dados Espaciotemporais em Larga Escala no domínio da Mobilidade e Tráfego (em inglês, Mobility- and Traffic-related Big Spatiotemporal Data —MobiTrafficBD) que estão a ser colecionados e recolhidos. Os trabalhos de investigação sobre os tópicos de Big Data, Dados Espaciotemporais e, especialmente, de MobiTrafficBD, estão dispersos, e a literatura existente não oferece uma metodologia comum e concreta para preparar, configurar, implementar e usar uma plataforma (framework) baseada em tecnologias Big Data para gerir o ciclo de vida de dados espaciotemporais em larga escala, com ênfase nas série temporais georreferenciadas (em inglês, geo-referenced time series — GRTS) e eventos espacio- temporais (em inglês, spatiotemporal events — ST Events), extrair valor destes dados e apoiar os RITMOs nos seus processos de decisão. Esta dissertação doutoral propõe uma metodologia prescritiva orientada a dados, para o design, desenvolvimento e implementação de plataformas de MobiTrafficBD, focadas em GRTS e ST Events. Além de uma revisão de literatura completa nas áreas de Dados Espaciotemporais, Big Data e na junção destas áreas através do conceito de MobiTrafficBD, a metodologia proposta contem um conjunto de características gerais, requisitos técnicos, componentes lógicos, fluxos de dados e modelos de infraestrutura tecnológica, bem como diretrizes e boas práticas para investigadores, profissionais e outras partes interessadas, como RITMOs, com o objetivo de guiá-los pelas fases de design, desenvolvimento e implementação de qualquer pla- taforma MobiTrafficBD. Este trabalho deve ser visto como um guia metodológico de suporte, baseado em Arqui- teturas de Referência e diretrizes amplamente utilizadas, mas enriquecido com as característi- cas e assuntos implícitos relacionados com Dados Espaciotemporais em Larga Escala, como no caso de GRTS e ST Events. A metodologia proposta foi avaliada e demonstrada em vários cenários reais no âmbito de projetos de investigação financiados pela Comissão Europeia e pelo Governo português, nos quais foram implementados métodos, ferramentas e tecnologias nas áreas de Gestão de Dados, Processamento de Dados e Ciência e Visualização de Dados em plataformas MobiTrafficB

Symmetry-Adapted Machine Learning for Information Security

Symmetry-adapted machine learning has shown encouraging ability to mitigate the security risks in information and communication technology (ICT) systems. It is a subset of artificial intelligence (AI) that relies on the principles of processing future events by learning past events or historical data. The autonomous nature of symmetry-adapted machine learning supports effective data processing and analysis for security detection in ICT systems without the interference of human authorities. Many industries are developing machine-learning-adapted solutions to support security for smart hardware, distributed computing, and the cloud. In our Special Issue book, we focus on the deployment of symmetry-adapted machine learning for information security in various application areas. This security approach can support effective methods to handle the dynamic nature of security attacks by extraction and analysis of data to identify hidden patterns of data. The main topics of this Issue include malware classification, an intrusion detection system, image watermarking, color image watermarking, battlefield target aggregation behavior recognition model, IP camera, Internet of Things (IoT) security, service function chain, indoor positioning system, and crypto-analysis

How to tell stories using visualization: strategies towards narrative visualization

Os benefícios da utilização das narrativas são desde há muito conhecidos e o seu potencial para simplificar conceitos, transmitir valores culturais e experiências, criar ligações emocionais e capacidade para ajudar a reter a informação tem sido explorado em diferentes áreas. As narrativas não são só a principal forma como as pessoas obtêm o sentido do mundo, mas também a forma mais fácil que encontrámos para partilhar informações complexas. Devido ao seu potencial, as narrativas foram recentemente abordadas na área da Visualização de Informação e do Conhecimento, muitas vezes apelidada de Visualização Narrativa. Esta questão é particularmente importante para os media, uma das áreas que tem impulsionado a investigação em Visualização Narrativa. A necessidade de incorporar histórias nas visualizações surge da necessidade de partilhar dados complexos de um modo envolvente. Hoje em dia somos confrontados com a elevada quantidade de informação disponível, um desafio difícil de resolver. Os avanços da tecnologia permitiram ir além das formas tradicionais de narrativa e de representação de dados, dando-nos meios mais atraentes e sofisticados para contar histórias. Nesta tese, exploro os benefícios da introdução de narrativas nas visualizações. Adicionalmente também exploro formas de combinar histórias com a visualizações e métodos eficientes para representar e dar sentido aos dados de uma forma que permite que as pessoas se relacionem com a informação. Esta investigação está bastante próxima da área do jornalismo, no entanto estas técnicas podem ser aplicadas em diferente áreas (educação, visualização científica, etc.). Para explorar ainda mais este tema foi adotada um avaliação que utiliza diferentes metodologias como a tipologia, vários casos de estudo, um estudo com grupos de foco, e ainda estudos de design e análise de técnicas.The benefits of storytelling are long-known and its potential to simplify concepts, convey cultural values and experiences, create emotional connection, and capacity to help retain information has been explored in di erent areas, such as journalism, education, marketing, and others. Narratives not only have been the main way people make sense of the world, but also the easiest way humans found out to share complex information. Due to its potential narratives have also recently been approached in the area of Information and Knowledge Visualization, several times being referred to as Narrative Visualization. This matter is also particularly important for news media, one of the areas that has been pushing the research on Narrative Visualization. The necessity to incorporate storytelling in visualizations arises from the need to share complex data in a way that is engaging. Nowadays we also have the challenge of the high amount of information available, which can be hard to cope with. Advances in technology have enabled us to go beyond the traditional forms of storytelling and representing data, giving us more attractive and sophisticated means to tell stories. In this dissertation, I explore the benefits of infusing visualizations with narratives. In addition I also present ways of combining storytelling with visualization and e cient methods to represent and make sense of data in a way that allows people to relate with the information. This research is closely related to journalism, but these techniques can be applied to completely di erent areas (education, scientific visualization, etc.). To further explore this topic a mixedmethod evaluation that consists of a typology, several case studies and a focus group study was chosen, as well as design studies and techniques review. This dissertation is intended to contribute to the evolving understanding of the field of narrative visualization