Augmenting Deep Learning Performance in an Evidential Multiple Classifier System

International audienceThe main objective of this work is to study the applicability of ensemble methods in the context of deep learning with limited amounts of labeled data. We exploit an ensemble of neural networks derived using Monte Carlo dropout, along with an ensemble of SVM classifiers which owes its effectiveness to the hand-crafted features used as inputs and to an active learning procedure. In order to leverage each classifier's respective strengths, we combine them in an evidential framework, which models specifically their imprecision and uncertainty. The application we consider in order to illustrate the interest of our Multiple Classifier System is pedestrian detection in high-density crowds, which is ideally suited for its difficulty, cost of labeling and intrinsic imprecision of annotation data. We show that the fusion resulting from the effective modeling of uncertainty allows for performance improvement, and at the same time, for a deeper interpretation of the result in terms of commitment of the decision

Semantic Decision Support for Information Fusion Applications

La thèse s'inscrit dans le domaine de la représentation des connaissances et la modélisation de l'incertitude dans un contexte de fusion d'informations. L'idée majeure est d'utiliser les outils sémantiques que sont les ontologies, non seulement pour représenter les connaissances générales du domaine et les observations, mais aussi pour représenter les incertitudes que les sources introduisent dans leurs observations. Nous proposons de représenter ces incertitudes au travers d'une méta-ontologie (DS-ontology) fondée sur la théorie des fonctions de croyance. La contribution de ce travail porte sur la définition d'opérateurs d'inclusion et d'intersection sémantique et sur lesquels s'appuie la mise en œuvre de la théorie des fonctions de croyance, et sur le développement d'un outil appelé FusionLab permettant la fusion d'informations sémantiques à partir du développement théorique précédent. Une application de ces travaux a été réalisée dans le cadre d'un projet de surveillance maritime.This thesis is part of the knowledge representation domain and modeling of uncertainty in a context of information fusion. The main idea is to use semantic tools and more specifically ontologies, not only to represent the general domain knowledge and observations, but also to represent the uncertainty that sources may introduce in their own observations. We propose to represent these uncertainties and semantic imprecision trough a metaontology (called DS-Ontology) based on the theory of belief functions. The contribution of this work focuses first on the definition of semantic inclusion and intersection operators for ontologies and on which relies the implementation of the theory of belief functions, and secondly on the development of a tool called FusionLab for merging semantic information within ontologies from the previous theorical development. These works have been applied within a European maritime surveillance project.ROUEN-INSA Madrillet (765752301) / SudocSudocFranceF

A general cognitive framework for context-aware systems: extensions and applications for high level information fusion approaches

Mención Internacional en el título de doctorContext-aware systems aims at the development of computational systems that process data acquired from different datasources and adapt their behaviour in order to provide the 'right' information, at the 'right' time, in the 'right' place, in the 'right' way to the 'right' person (Fischer, 2012). Traditionally computational research has tried to answer these needs by means of low-level algorithms. In the last years the combination of numeric and symbolic approaches has offered the opportunity to create systems to deal with these issues. However, although the performance of algorithms and the quality of the data directly provided by computers and devices has quickly improved, symbolic models used to represent the resulting knowledge have not yet been adapted to smart environments. This lack of representation does not allow to take advantage of the semantic quality of the information provided by new sensors. This dissertation proposes a set of extensions and applications focused on a cognitive framework for the implementation of context-aware systems based on a general model inspired by the Information Fusion paradigm. This model is stepped in several abstraction levels from low-level raw data to high level scene interpretation whose structure is determined by a set of ontologies. Each ontology level provides a skeleton that includes general concepts and relations to describe entities and their connections. This structure has been designed to promote extensibility and modularity, and might be refined to apply this model in specific domains. This framework combines a priori context knowledge represented with ontologies with real data coming from sensors to support logic-based high-level interpretation of the current situation and to automatically generate feedback recommendations to adjust data acquisition procedures. This work advocates for the introduction of general purpose cognitive layers in order to obtain a closer representation to the human cognition, generate additional knowledge and improve the high-level interpretation. Extensibility and adaptability of the basic ontology levels is demonstrated with the introduction of these traverse semantic layers which are able to be present and represent information at several granularity levels of knowledge using a common formalism. Context-based system must be able to reason about uncertainty. However the reasoning associated to ontologies has been limited to classical description logic mechanisms. This research also tackle the problem of reasoning under uncertainty circumstances through a logic-based paradigm for abductive reasoning: the Belief-Argumentation System. The main contribution of this dissertation is the adaptation of the general architecture and the theoretical proposals to several context-aware application areas such as Ambient Intelligence, Social Signal Processing and surveillance systems. The implementation of prototypes and examples for these areas are explained along this dissertation to progressively illustrate the improvements and extensions in the framework. To initially depict the general model, its components and the basic reasoning mechanisms a video-based Ambient Intelligence application is presented. The advantages and features of the framework extensions through traverse cognitive layers are demonstrated in a Social Signal Processing case for the elaboration of automatic market researches. Finally, the functioning of the system under uncertainty circumstances is illustrated with several examples to support decision makers in the detection of potential threats in common harbor scenarios.Programa Oficial de Doctorado en Ciencia y Tecnología InformáticaPresidente: José Manuel Molina López.- Secretario: Ángel Arroyo.- Vocal: Nayat Sánchez P

Data Fusion for Materials Location Estimation in Construction

Effective automated tracking and locating of the thousands of materials on construction sites improves material distribution and project performance and thus has a significant positive impact on construction productivity. Many locating technologies and data sources have therefore been developed, and the deployment of a cost-effective, scalable, and easy-to-implement materials location sensing system at actual construction sites has very recently become both technically and economically feasible. However, considerable opportunity still exists to improve the accuracy, precision, and robustness of such systems. The quest for fundamental methods that can take advantage of the relative strengths of each individual technology and data source motivated this research, which has led to the development of new data fusion methods for improving materials location estimation. In this study a data fusion model is used to generate an integrated solution for the automated identification, location estimation, and relocation detection of construction materials. The developed model is a modified functional data fusion model. Particular attention is paid to noisy environments where low-cost RFID tags are attached to all materials, which are sometimes moved repeatedly around the site. A portion of the work focuses partly on relocation detection because it is closely coupled with location estimation and because it can be used to detect the multi-handling of materials, which is a key indicator of inefficiency. This research has successfully addressed the challenges of fusing data from multiple sources of information in a very noisy and dynamic environment. The results indicate potential for the proposed model to improve location estimation and movement detection as well as to automate the calculation of the incidence of multi-handling

A new framework for deep learning video based Human Action Recognition on the edge

Nowadays, video surveillance systems are commonly found in most public and private spaces. These systems typically consist of a network of cameras that feed into a central node. However, the processing aspect is evolving towards distributed approaches, leveraging edge-computing. These distributed systems are capable of effectively addressing the detection of people or events at each individual node. Most of these systems, rely on the use of deep-learning and segmentation algorithms which enable them to achieve high performance, but usually with a significant computational cost, hindering real-time execution. This paper presents an approach for people detection and action recognition in the wild, optimized for running on the edge, and that is able to work in real-time, in an embedded platform. Human Action Recognition (HAR) is performed by using a Recurrent Neural Network (RNN), specifically a Long Short-Term Memory (LSTM). The input to the LSTM is an ad-hoc, lightweight feature vector obtained from the bounding box of each detected person in the video surveillance image. The resulting system is highly portable and easily scalable, providing a powerful tool for real-world video surveillance applications (in the wild and real-time action recognition). The proposal has been exhaustively evaluated and compared against other state-of-the-art (SOTA) proposals in five datasets, including four widely used (KTH, WEIZMAN, WVU, IXMAX) and a novel one (GBA) recorded in the wild, that includes several people performing different actions simultaneously. The obtained results validate the proposal, since it achieves SOTA accuracy within a much more complicated video surveillance real scenario, and using a lightweight embedded hardware.European CommissionAgencia Estatal de InvestigaciónUniversidad de Alcal

Inferring Complex Activities for Context-aware Systems within Smart Environments

The rising ageing population worldwide and the prevalence of age-related conditions such as physical fragility, mental impairments and chronic diseases have significantly impacted the quality of life and caused a shortage of health and care services. Over-stretched healthcare providers are leading to a paradigm shift in public healthcare provisioning. Thus, Ambient Assisted Living (AAL) using Smart Homes (SH) technologies has been rigorously investigated to help address the aforementioned problems. Human Activity Recognition (HAR) is a critical component in AAL systems which enables applications such as just-in-time assistance, behaviour analysis, anomalies detection and emergency notifications. This thesis is aimed at investigating challenges faced in accurately recognising Activities of Daily Living (ADLs) performed by single or multiple inhabitants within smart environments. Specifically, this thesis explores five complementary research challenges in HAR. The first study contributes to knowledge by developing a semantic-enabled data segmentation approach with user-preferences. The second study takes the segmented set of sensor data to investigate and recognise human ADLs at multi-granular action level; coarse- and fine-grained action level. At the coarse-grained actions level, semantic relationships between the sensor, object and ADLs are deduced, whereas, at fine-grained action level, object usage at the satisfactory threshold with the evidence fused from multimodal sensor data is leveraged to verify the intended actions. Moreover, due to imprecise/vague interpretations of multimodal sensors and data fusion challenges, fuzzy set theory and fuzzy web ontology language (fuzzy-OWL) are leveraged. The third study focuses on incorporating uncertainties caused in HAR due to factors such as technological failure, object malfunction, and human errors. Hence, existing studies uncertainty theories and approaches are analysed and based on the findings, probabilistic ontology (PR-OWL) based HAR approach is proposed. The fourth study extends the first three studies to distinguish activities conducted by more than one inhabitant in a shared smart environment with the use of discriminative sensor-based techniques and time-series pattern analysis. The final study investigates in a suitable system architecture with a real-time smart environment tailored to AAL system and proposes microservices architecture with sensor-based off-the-shelf and bespoke sensing methods. The initial semantic-enabled data segmentation study was evaluated with 100% and 97.8% accuracy to segment sensor events under single and mixed activities scenarios. However, the average classification time taken to segment each sensor events have suffered from 3971ms and 62183ms for single and mixed activities scenarios, respectively. The second study to detect fine-grained-level user actions was evaluated with 30 and 153 fuzzy rules to detect two fine-grained movements with a pre-collected dataset from the real-time smart environment. The result of the second study indicate good average accuracy of 83.33% and 100% but with the high average duration of 24648ms and 105318ms, and posing further challenges for the scalability of fusion rule creations. The third study was evaluated by incorporating PR-OWL ontology with ADL ontologies and Semantic-Sensor-Network (SSN) ontology to define four types of uncertainties presented in the kitchen-based activity. The fourth study illustrated a case study to extended single-user AR to multi-user AR by combining RFID tags and fingerprint sensors discriminative sensors to identify and associate user actions with the aid of time-series analysis. The last study responds to the computations and performance requirements for the four studies by analysing and proposing microservices-based system architecture for AAL system. A future research investigation towards adopting fog/edge computing paradigms from cloud computing is discussed for higher availability, reduced network traffic/energy, cost, and creating a decentralised system. As a result of the five studies, this thesis develops a knowledge-driven framework to estimate and recognise multi-user activities at fine-grained level user actions. This framework integrates three complementary ontologies to conceptualise factual, fuzzy and uncertainties in the environment/ADLs, time-series analysis and discriminative sensing environment. Moreover, a distributed software architecture, multimodal sensor-based hardware prototypes, and other supportive utility tools such as simulator and synthetic ADL data generator for the experimentation were developed to support the evaluation of the proposed approaches. The distributed system is platform-independent and currently supported by an Android mobile application and web-browser based client interfaces for retrieving information such as live sensor events and HAR results

Multisensor data fusion

Thesis (M.Eng.Sc.)--University of Adelaide, Dept. of Electrical and Electronic Engineering, 199

Panopticism and Complicity: The State of Surveillance and Everyday Oppression in Libraries, Archives, and Museums

Historically, libraries, archives, and museums—or LAM institutions—have been complicit in enacting state power by surveilling and policing communities. This article broadens previous scholars’ critiques about individual institutions to LAM institutions writ large, drawing connections between these sites and ongoing racist, classist, and oppressive designs. We do so by dialing in on the ethical premise that justifies panoptic systems, utilitarianism, and how the glorification of pragmatism reifies systems of control and oppression. First, we revisit LIS applications of Benthamian and Foucauldian ideas of panoptic power to examine the role of LAM institutions as sites of social enmity. We then describe examples of surveillance and state power as they manifest in contemporary data infrastructure and information practices, showing how LAM institutional fixations with utilitarianism reify the U.S. carceral state through norms such as the aggregation and weaponization of user data and the overreliance on metrics. We argue that such practices are akin to widespread systems of surveillance and criminalization. Finally, we reflect on how LAM workers can combat structures that rely on oppressive assumptions and claims to information authority. Pre-print first published online February 10, 202