Deep Cross-Modal Correlation Learning for Audio and Lyrics in Music Retrieval

Deep cross-modal learning has successfully demonstrated excellent performance in cross-modal multimedia retrieval, with the aim of learning joint representations between different data modalities. Unfortunately, little research focuses on cross-modal correlation learning where temporal structures of different data modalities such as audio and lyrics should be taken into account. Stemming from the characteristic of temporal structures of music in nature, we are motivated to learn the deep sequential correlation between audio and lyrics. In this work, we propose a deep cross-modal correlation learning architecture involving two-branch deep neural networks for audio modality and text modality (lyrics). Data in different modalities are converted to the same canonical space where inter modal canonical correlation analysis is utilized as an objective function to calculate the similarity of temporal structures. This is the first study that uses deep architectures for learning the temporal correlation between audio and lyrics. A pre-trained Doc2Vec model followed by fully-connected layers is used to represent lyrics. Two significant contributions are made in the audio branch, as follows: i) We propose an end-to-end network to learn cross-modal correlation between audio and lyrics, where feature extraction and correlation learning are simultaneously performed and joint representation is learned by considering temporal structures. ii) As for feature extraction, we further represent an audio signal by a short sequence of local summaries (VGG16 features) and apply a recurrent neural network to compute a compact feature that better learns temporal structures of music audio. Experimental results, using audio to retrieve lyrics or using lyrics to retrieve audio, verify the effectiveness of the proposed deep correlation learning architectures in cross-modal music retrieval

Development Of Information Visualization Methods For Use In Multimedia Applications

The aim of the article is development of a technique for visualizing information for use in multimedia applications. In this study, to visualize information, it is proposed first to compile a list of key terms of the subject area and create data tables. Based on the structuring of fragments of the subject area, a visual display of key terms in the form of pictograms, a visual display of key terms in the form of images, and a visual display of data tables are performed. The types of visual structures that should be used to visualize information for further use in multimedia applications are considered. The analysis of existing visual structures in desktop publishing systems and word processors is performed.To build a mechanism for visualizing information about the task as a presentation, a multimedia application is developed using Microsoft Visual Studio software, the C# programming language by using the Windows Forms application programming interface. An algorithm is proposed for separating pieces of information text that have key terms. Tabular data was visualized using the “parametric ruler” metaphorical visualization method, based on the metaphor of a slide rule.The use of the parametric ruler method on the example of data visualization for the font design of children's publications is proposed. Interaction of using the method is ensured due to the fact that the user will enter the size of the size that interests for it and will see the ratio of the values of other parameters. The practical result of the work is the creation of a multimedia application “Visualization of Publishing Standards” for the visualization of information for the font design of publications for children. The result of the software implementation is the finished multimedia applications, which, according to the standardization visualization technique in terms of prepress preparation of publications, is the final product of the third stage of the presentation of the visual for