A novel approach for multimodal graph dimensionality reduction

This thesis deals with the problem of multimodal dimensionality reduction (DR), which arises when the input objects, to be mapped on a low-dimensional space, consist of multiple vectorial representations, instead of a single one. Herein, the problem is addressed in two alternative manners. One is based on the traditional notion of modality fusion, but using a novel approach to determine the fusion weights. In order to optimally fuse the modalities, the known graph embedding DR framework is extended to multiple modalities by considering a weighted sum of the involved affinity matrices. The weights of the sum are automatically calculated by minimizing an introduced notion of inconsistency of the resulting multimodal affinity matrix. The other manner for dealing with the problem is an approach to consider all modalities simultaneously, without fusing them, which has the advantage of minimal information loss due to fusion. In order to avoid fusion, the problem is viewed as a multi-objective optimization problem. The multiple objective functions are defined based on graph representations of the data, so that their individual minimization leads to dimensionality reduction for each modality separately. The aim is to combine the multiple modalities without the need to assign importance weights to them, or at least postpone such an assignment as a last step. The proposed approaches were experimentally tested in mapping multimedia data on low-dimensional spaces for purposes of visualization, classification and clustering. The no-fusion approach, namely Multi-objective DR, was able to discover mappings revealing the structure of all modalities simultaneously, which cannot be discovered by weight-based fusion methods. However, it results in a set of optimal trade-offs, from which one needs to be selected, which is not trivial. The optimal-fusion approach, namely Multimodal Graph Embedding DR, is able to easily extend unimodal DR methods to multiple modalities, but depends on the limitations of the unimodal DR method used. Both the no-fusion and the optimal-fusion approaches were compared to state-of-the-art multimodal dimensionality reduction methods and the comparison showed performance improvement in visualization, classification and clustering tasks. The proposed approaches were also evaluated for different types of problems and data, in two diverse application fields, a visual-accessibility-enhanced search engine and a visualization tool for mobile network security data. The results verified their applicability in different domains and suggested promising directions for future advancements.Open Acces

A novel framework for retrieval and interactive visualization of multimodal data

With the abundance of multimedia in web databases and the increasing user need for content of many modalities, such as images, sounds, etc. , new methods for retrieval and visualization of multimodal media are required. In this paper, novel techniques for retrieval and visualization of multimodal data, i. e. documents consisting of many modalities, are proposed. A novel cross-modal retrieval framework is presented, in which the results of several unimodal retrieval systems are fused into a single multimodal list by the introduction of a cross-modal distance. For the presentation of the retrieved results, a multimodal visualization framework is also proposed, which extends existing unimodal similarity-based visualization methods for multimodal data. The similarity measure between two multimodal objects is defined as the weighted sum of unimodal similarities, with the weights determined via an interactive user feedback scheme. Experimental results show that the cross-modal framework outperforms unimodal and other multimodal approaches while the visualization framework enhances existing visualization methods by efficiently exploiting multimodality and user feedback

Accessibility-based reranking in multimedia search engines

Traditional multimedia search engines retrieve results based mostly on the query submitted by the user, or using a log of previous searches to provide personalized results, while not considering the accessibility of the results for users with vision or other types of impairments. In this paper, a novel approach is presented which incorporates the accessibility of images for users with various vision impairments, such as color blindness, cataract and glaucoma, in order to rerank the results of an image search engine. The accessibility of individual images is measured through the use of vision simulation filters. Multi-objective optimization techniques utilizing the image accessibility scores are used to handle users with multiple vision impairments, while the impairment profile of a specific user is used to select one from the Pareto-optimal solutions. The proposed approach has been tested with two image datasets, using both simulated and real impaired users, and the results verify its applicability. Although the proposed method has been used for vision accessibility-based reranking, it can also be extended for other types of personalization context

A novel framework for retrieval and interactive visualization of multimodal data

With the abundance of multimedia in web databases and the increasing user need for content of many modalities, such as images, sounds, etc., new methods for retrieval and visualization of multimodal media are required. In this paper, novel techniques for retrieval and visualization of multimodal data, i.e. documents consisting of many modalities, are proposed. A novel cross-modal retrieval framework is presented, in which the results of several unimodal retrieval systems are fused into a single multimodal list by the introduction of a cross-modal distance. For the presentation of the retrieved results, a multimodal visualization framework is also proposed, which extends existing unimodal similarity-based visualization methods for multimodal data. The similarity measure between two multimodal objects is defined as the weighted sum of unimodal similarities, with the weights determined via an interactive user feedback scheme. Experimental results show that the cross-modal framework outperforms unimodal and other multimodal approaches while the visualization framework enhances existing visualization methods by efficiently exploiting multimodality and user feedback

MoVA: A Visual Analytics Tool Providing Insight in the Big Mobile Network Data

Part 7: New Methods and Tools for Big DataInternational audienceMobile networks have numerous exploitable vulnerabilities that enable malicious individuals to launch Denial of Service (DoS) attacks and affect network security and performance. The efficient detection and attribution of these anomalies are of major importance to the mobile network operators, especially since there is a vast amount of information collected, which renders the problem as a Big Data problem. Previous approaches focus on either anomaly detection methods, or visualization methods separately. In addition, they utilize solely either the signaling or the Call Detail Record (CDR) activity in the network. This paper presents MoVA (Mobile network Visual Analytics), a visual analytics tool for the detection and attribution of anomalies in mobile cellular networks which combines anomaly detection and visualization, and is applied on both signaling and CDR activity in the network. In order to address the large volume of the data, the proposed application starts with an aggregated overview of the whole network and allows the operator to gradually focus on smaller sets of data, using different levels of abstraction. The proposed visualization methods are able to differentiate between different user behaviors, and enable the analyst to have an insight in the mobile network operation and easily spot the anomalous mobile devices. Hypothesis formulation and validation methods are also provided, in order to enable the analyst to formulate network security-related hypotheses, and validate or reject them based on the results of the analysis

A novel framework for retrieval and interactive visualization of multimodal data

With the abundance of multimedia in web databases and the increasing user need for content of many modalities, such as images, sounds, etc., new methods for retrieval and visualization of multimodal media are required. In this paper, novel techniques for retrieval and visualization of multimodal data, i.e. documents consisting of many modalities, are proposed. A novel cross-modal retrieval framework is presented, in which the results of several unimodal retrieval systems are fused into a single multimodal list by the introduction of a cross-modal distance. For the presentation of the retrieved results, a multimodal visualization framework is also proposed, which extends existing unimodal similarity-based visualization methods for multimodal data. The similarity measure between two multimodal objects is defined as the weighted sum of unimodal similarities, with the weights determined via an interactive user feedback scheme. Experimental results show that the cross-modal framework outperforms unimodal and other multimodal approaches while the visualization framework enhances existing visualization methods by efficiently exploiting multimodality and user feedback

On the Use of Air Quality Microsensors for Supporting Decision Makers

In this poster we present how a network of Internet-of-things (IoT) devices facilitated through machine learning can improve decision making. Our application domain is air quality in the municipality of Trondheim. Ambient air pollution poses a major threat to both health and climate with millions of premature deaths occurring every year. To enable solutions to this problem, accurate measurements of the phenomenon are required and tools for decision makers need to be in place to quickly understand situations as well as suggest actions that lead to the best possible outcome

Towards prediction of Quality of Life aspects using wearable data with limited ground truth

Today people live with a variety of diseases and health conditions that can impact their daily lives, from chronic illnesses such as cancer to mental health issues such as anxiety and depression. With the advancement of technology and healthcare, quality of life (QoL) prediction and improvement is becoming increasingly important to help individuals live better. In addition to data collected by questionnaires, machine learning can be employed to predict aspects of QoL, such as fatigue and anxiety, based on data from wearable devices. However, this requires ground truth information, which, in real-world cases is not always present, or is scarce. This paper addresses the problem of fatigue prediction, which is part of the overall QoL prediction. We extend previous literature by viewing the problem as regression, with the ultimate goal of transferring a model trained with a complete dataset of the literature to a real-world dataset with no ground truth, facing the challenge of only a few common features. We compare several regression and classification techniques to select the best-performing ones. We then explore the effect of using only limited important features on accuracy, which allow us to assess the applicability of transferring the models to a real-world dataset. We apply the trained model on data from the ONCORELIEF project and discuss the identified challenges

A Low-Cost Indoor Activity Monitoring System for Detecting Frailty in Older Adults

Indoor localization systems have already wide applications mainly for providing localized information and directions. The majority of them focus on commercial applications providing information such us advertisements, guidance and asset tracking. Medical oriented localization systems are uncommon. Given the fact that an individual’s indoor movements can be indicative of his/her clinical status, in this paper we present a low-cost indoor localization system with room-level accuracy used to assess the frailty of older people. We focused on designing a system with easy installation and low cost to be used by non technical staff. The system was installed in older people houses in order to collect data about their indoor localization habits. The collected data were examined in combination with their frailty status, showing a correlation between them. The indoor localization system is based on the processing of Received Signal Strength Indicator (RSSI) measurements by a tracking device, from Bluetooth Beacons, using a fingerprint-based procedure. The system has been tested in realistic settings achieving accuracy above 93% in room estimation. The proposed system was used in 271 houses collecting data for 1–7-day sessions. The evaluation of the collected data using ten-fold cross-validation showed an accuracy of 83% in the classification of a monitored person regarding his/her frailty status (Frail, Pre-frail, Non-frail)