Multimodal music information processing and retrieval: survey and future challenges

Towards improving the performance in various music information processing tasks, recent studies exploit different modalities able to capture diverse aspects of music. Such modalities include audio recordings, symbolic music scores, mid-level representations, motion, and gestural data, video recordings, editorial or cultural tags, lyrics and album cover arts. This paper critically reviews the various approaches adopted in Music Information Processing and Retrieval and highlights how multimodal algorithms can help Music Computing applications. First, we categorize the related literature based on the application they address. Subsequently, we analyze existing information fusion approaches, and we conclude with the set of challenges that Music Information Retrieval and Sound and Music Computing research communities should focus in the next years

An investigation into weighted data fusion for content-based multimedia information retrieval

Content Based Multimedia Information Retrieval (CBMIR) is characterised by the combination of noisy sources of information which, in unison, are able to achieve strong performance. In this thesis we focus on the combination of ranked results from the independent retrieval experts which comprise a CBMIR system through linearly weighted data fusion. The independent retrieval experts are low-level multimedia features, each of which contains an indexing function and ranking algorithm. This thesis is comprised of two halves. In the first half, we perform a rigorous empirical investigation into the factors which impact upon performance in linearly weighted data fusion. In the second half, we leverage these finding to create a new class of weight generation algorithms for data fusion which are capable of determining weights at query-time, such that the weights are topic dependent

Resource selection and data fusion for multimedia international digital libraries: an overview of the MIND project

The inspiration for MIND grew out of the problems which users face when they have remote access to thousands of heterogeneous and distributed multimedia digital libraries. A user must know where to search, how to query different media, and how to combine information from diverse resources. As digital libraries continue to proliferate, in a variety of media and from a variety of sources, the problems of resource selection, query formulation and data fusion become major obstacles to effective search and retrieval. The key goal of MIND is to develop a common system for identifying, searching and combining results from multiple digital libraries. MIND, therefore, is investigating methods for resource description and selection (i.e., gathering and updating information about digital libraries to assist in selecting those which are most likely to contain the information sought), query processing (i.e. modifying the terms contained in a query and transforming the query into the local command language), data fusion (i.e., the merging of different data retrieved from different digital libraries) and information visualisation (in particular, the automatic generation of surrogates and presentation of fused retrieved data)

Fusion of Text and Image in Multimedia Information Retrieval System

Multimedia Information Retrieval is very useful for any application in our daily work. Most of the applications consist of Multimedia data that are images, text, audio and video. Multimedia information retrieval system is used to search an image. There are same meanings for different data which is also known as semantic gap. This problem is solved by fusion of text based image retrieval and content based image retrieval. Weighted Mean, OWA and WOWA are aggregation operators used in this system for the fusion of text and image numeric values. The Scale invariant feature transforms and speeded up robust feature are two algorithms for feature extraction. To increase the speed of system, the speeded up robust feature algorithm is used. Bag of Words and Bag of Visual Word approaches are used in this system for retrieving images. DOI: 10.17762/ijritcc2321-8169.15066

Key Information Retrieval in Hyperspectral Imagery through Spatial-Spectral Data Fusion

Hyperspectral (HS) imaging is measuring the radiance of materials within each pixel area at a large number of contiguous spectral wavelength bands. The key spatial information such as small targets and border lines are hard to be precisely detected from HS data due to the technological constraints. Therefore, the need for image processing techniques is an important field of research in HS remote sensing. A novel semisupervised spatial-spectral data fusion method for resolution enhancement of HS images through maximizing the spatial correlation of the endmembers (signature of pure or purest materials in the scene) using a superresolution mapping (SRM) technique is proposed in this paper. The method adopts a linear mixture model and a fully constrained least squares spectral unmixing algorithm to obtain the endmember abundances (fractional images) of HS images. Then, the extracted endmember distribution maps are fused with the spatial information using a spatial-spectral correlation maximizing model and a learning-based SRM technique to exploit the subpixel level data. The obtained results validate the reliability of the technique for key information retrieval. The proposed method is very efficient and is low in terms of computational cost which makes it favorable for real-time applications

Online forum thread retrieval using data fusion

Online forums empower people to seek and share information via discussion threads. However, finding threads satisfying a user information need is a daunting task due to information overload. In addition, traditional retrieval techniques do not suit the unique structure of threads because thread retrieval returns threads, whereas traditional retrieval techniques return text messages. A few representations have been proposed to address this problem; and, in some representations aggregating query relevance evidence is an essential step. This thesis proposes several data fusion techniques to aggregate evidence of relevance within and across thread representations. In that regard, this thesis has three contributions. Firstly, this work adapts the Voting Model from the expert finding task to thread retrieval. The adapted Voting Model approaches thread retrieval as a voting process. It ranks a list of messages, then it groups messages based on their parent threads; also, it treats each ranked message as a vote supporting the relevance of its parent thread. To rank parent threads, a data fusion technique aggregates evidence from threads’ ranked messages. Secondly, this study proposes two extensions of the voting model: Top K and Balanced Top K voting models. The Top K model aggregates evidence from only the top K ranked messages from each thread. The Balanced Top K model adds a number of artificial ranked messages to compensate the difference if a thread has less than K ranked messages (a padding step). Experiments with these voting models and thirteen data fusion methods reveal that summing relevance scores of the top K ranked messages from each thread with the padding step outperforms the state of the art on all measures on two datasets. The third contribution of this thesis is a multi-representation thread retrieval using data fusion techniques. In contrast to the Voting Model, data fusion methods were used to fuse several ranked lists of threads instead of a single ranked list of messages. The thread lists were generated by five retrieval methods based on various thread representations; the Voting Model is one of them. The first three methods assume a message to be the unit of indexing, while the latter two assume the title and the concatenation of the thread message texts to be the units of indexing respectively. A thorough evaluation of the performance of data fusion techniques in fusing various combinations of thread representations was conducted. The experimental results show that using the sum of relevance scores or the sum of relevance scores multiplied by the number of retrieving methods to develop multi-representation thread retrieval improves performance and outperforms all individual representation