A novel framework for retrieval and interactive visualization of multimodal data

With the abundance of multimedia in web databases and the increasing user need for content of many modalities, such as images, sounds, etc. , new methods for retrieval and visualization of multimodal media are required. In this paper, novel techniques for retrieval and visualization of multimodal data, i. e. documents consisting of many modalities, are proposed. A novel cross-modal retrieval framework is presented, in which the results of several unimodal retrieval systems are fused into a single multimodal list by the introduction of a cross-modal distance. For the presentation of the retrieved results, a multimodal visualization framework is also proposed, which extends existing unimodal similarity-based visualization methods for multimodal data. The similarity measure between two multimodal objects is defined as the weighted sum of unimodal similarities, with the weights determined via an interactive user feedback scheme. Experimental results show that the cross-modal framework outperforms unimodal and other multimodal approaches while the visualization framework enhances existing visualization methods by efficiently exploiting multimodality and user feedback

A novel approach for multimodal graph dimensionality reduction

This thesis deals with the problem of multimodal dimensionality reduction (DR), which arises when the input objects, to be mapped on a low-dimensional space, consist of multiple vectorial representations, instead of a single one. Herein, the problem is addressed in two alternative manners. One is based on the traditional notion of modality fusion, but using a novel approach to determine the fusion weights. In order to optimally fuse the modalities, the known graph embedding DR framework is extended to multiple modalities by considering a weighted sum of the involved affinity matrices. The weights of the sum are automatically calculated by minimizing an introduced notion of inconsistency of the resulting multimodal affinity matrix. The other manner for dealing with the problem is an approach to consider all modalities simultaneously, without fusing them, which has the advantage of minimal information loss due to fusion. In order to avoid fusion, the problem is viewed as a multi-objective optimization problem. The multiple objective functions are defined based on graph representations of the data, so that their individual minimization leads to dimensionality reduction for each modality separately. The aim is to combine the multiple modalities without the need to assign importance weights to them, or at least postpone such an assignment as a last step. The proposed approaches were experimentally tested in mapping multimedia data on low-dimensional spaces for purposes of visualization, classification and clustering. The no-fusion approach, namely Multi-objective DR, was able to discover mappings revealing the structure of all modalities simultaneously, which cannot be discovered by weight-based fusion methods. However, it results in a set of optimal trade-offs, from which one needs to be selected, which is not trivial. The optimal-fusion approach, namely Multimodal Graph Embedding DR, is able to easily extend unimodal DR methods to multiple modalities, but depends on the limitations of the unimodal DR method used. Both the no-fusion and the optimal-fusion approaches were compared to state-of-the-art multimodal dimensionality reduction methods and the comparison showed performance improvement in visualization, classification and clustering tasks. The proposed approaches were also evaluated for different types of problems and data, in two diverse application fields, a visual-accessibility-enhanced search engine and a visualization tool for mobile network security data. The results verified their applicability in different domains and suggested promising directions for future advancements.Open Acces

Accessibility-based reranking in multimedia search engines

Traditional multimedia search engines retrieve results based mostly on the query submitted by the user, or using a log of previous searches to provide personalized results, while not considering the accessibility of the results for users with vision or other types of impairments. In this paper, a novel approach is presented which incorporates the accessibility of images for users with various vision impairments, such as color blindness, cataract and glaucoma, in order to rerank the results of an image search engine. The accessibility of individual images is measured through the use of vision simulation filters. Multi-objective optimization techniques utilizing the image accessibility scores are used to handle users with multiple vision impairments, while the impairment profile of a specific user is used to select one from the Pareto-optimal solutions. The proposed approach has been tested with two image datasets, using both simulated and real impaired users, and the results verify its applicability. Although the proposed method has been used for vision accessibility-based reranking, it can also be extended for other types of personalization context

Indoor Localization Dataset

The dataset contains information concerning the older people’s movement inside their homes regarding their indoor location in the home setting. The dataset is recorded daily with the use of smart beacon devices installed in each older person's home and monitored through the system. Each record of the dataset has the following fields: - part_id: The user ID, which should be a 4-digit number - ts_date: The recording date, which follows the “YYYYMMDD” format, e.g. 14 September 2017, is formatted as 20170914 - ts_time: The recording time, which follows the “hh:mm:ss” format - room: The room which the person entered on the specific date and time (It is assumed that the person remained in the room till the next recording of the same day)</p

An enhanced Graph Analytics Platform (GAP) providing insight in Big Network Data

AbstractBeing a widely adapted and acknowledged practice for the representation of inter- and intra-dependent information streams, network graphs are nowadays growing vast in both size and complexity, due to the rapid expansion of sources, types, and amounts of produced data. In this context, the efficient processing of the big amounts of information, also known as Big Data forms a major challenge for both the research community and a wide variety of industrial sectors, involving security, health and financial applications. Serving these emerging needs, the current paper presents a Graph Analytics based Platform (GAP) that implements a top-down approach for the facilitation of Data Mining processes through the incorporation of state-of-the-art techniques, like behavioural clustering, interactive visualizations, multi-objective optimization, etc. The applicability of this platform is validated on 2 istinct real-world use cases, which can be considered as characteristic examples of modern Big Data problems, due to the vast amount of information they deal with. In particular, (i) the root cause analysis of a Denial of Service attack in the network of a mobile operator and (ii) the early detection of an emerging event or a hot topic in social media communities. In order to address the large volume of the data, the proposed application starts with an aggregated overview of the whole network and allows the operator to gradually focus on smaller sets of data, using different levels of abstraction. The proposed platform offers differentiation between different user behaviors that enable the analyst to obtain insight on the network’s operation and to extract the meaningful information in an effortless manner. Dynamic hypothesis formulation techniques exploited by graph traversing and pattern mining, enable the analyst to set concrete network-related hypotheses, and validate or reject them accordingly

MoVA: A Visual Analytics Tool Providing Insight in the Big Mobile Network Data

Part 7: New Methods and Tools for Big DataInternational audienceMobile networks have numerous exploitable vulnerabilities that enable malicious individuals to launch Denial of Service (DoS) attacks and affect network security and performance. The efficient detection and attribution of these anomalies are of major importance to the mobile network operators, especially since there is a vast amount of information collected, which renders the problem as a Big Data problem. Previous approaches focus on either anomaly detection methods, or visualization methods separately. In addition, they utilize solely either the signaling or the Call Detail Record (CDR) activity in the network. This paper presents MoVA (Mobile network Visual Analytics), a visual analytics tool for the detection and attribution of anomalies in mobile cellular networks which combines anomaly detection and visualization, and is applied on both signaling and CDR activity in the network. In order to address the large volume of the data, the proposed application starts with an aggregated overview of the whole network and allows the operator to gradually focus on smaller sets of data, using different levels of abstraction. The proposed visualization methods are able to differentiate between different user behaviors, and enable the analyst to have an insight in the mobile network operation and easily spot the anomalous mobile devices. Hypothesis formulation and validation methods are also provided, in order to enable the analyst to formulate network security-related hypotheses, and validate or reject them based on the results of the analysis