Semantic Federation of Musical and Music-Related Information for Establishing a Personal Music Knowledge Base

Music is perceived and described very subjectively by every individual. Nowadays, people often get lost in their steadily growing, multi-placed, digital music collection. Existing music player and management applications get in trouble when dealing with poor metadata that is predominant in personal music collections. There are several music information services available that assist users by providing tools for precisely organising their music collection, or for presenting them new insights into their own music library and listening habits. However, it is still not the case that music consumers can seamlessly interact with all these auxiliary services directly from the place where they access their music individually. To profit from the manifold music and music-related knowledge that is or can be available via various information services, this information has to be gathered up, semantically federated, and integrated into a uniform knowledge base that can personalised represent this data in an appropriate visualisation to the users. This personalised semantic aggregation of music metadata from several sources is the gist of this thesis. The outlined solution particularly concentrates on users’ needs regarding music collection management which can strongly alternate between single human beings. The author’s proposal, the personal music knowledge base (PMKB), consists of a client-server architecture with uniform communication endpoints and an ontological knowledge representation model format that is able to represent the versatile information of its use cases. The PMKB concept is appropriate to cover the complete information flow life cycle, including the processes of user account initialisation, information service choice, individual information extraction, and proactive update notification. The PMKB implementation makes use of SemanticWeb technologies. Particularly the knowledge representation part of the PMKB vision is explained in this work. Several new Semantic Web ontologies are defined or existing ones are massively modified to meet the requirements of a personalised semantic federation of music and music-related data for managing personal music collections. The outcome is, amongst others, • a new vocabulary for describing the play back domain, • another one for representing information service categorisations and quality ratings, and • one that unites the beneficial parts of the existing advanced user modelling ontologies. The introduced vocabularies can be perfectly utilised in conjunction with the existing Music Ontology framework. Some RDFizers that also make use of the outlined ontologies in their mapping definitions, illustrate the fitness in practise of these specifications. A social evaluation method is applied to carry out an examination dealing with the reutilisation, application and feedback of the vocabularies that are explained in this work. This analysis shows that it is a good practise to properly publish Semantic Web ontologies with the help of some Linked Data principles and further basic SEO techniques to easily reach the searching audience, to avoid duplicates of such KR specifications, and, last but not least, to directly establish a \"shared understanding\". Due to their project-independence, the proposed vocabularies can be deployed in every knowledge representation model that needs their knowledge representation capacities. This thesis added its value to make the vision of a personal music knowledge base come true.:1 Introduction and Background 11 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.2 Personal Music Collection Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . 14 1.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2 Music Information Management 17 2.1 Knowledge Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.1.1 Knowledge Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.1.1.1 Knowledge Representation Models . . . . . . . . . . . . . . . . . 18 2.1.1.2 Semantic Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.1.1.3 Ontologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.1.1.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.1.2 Knowledge Management Systems . . . . . . . . . . . . . . . . . . . . . . . 19 2.1.2.1 Information Services . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.1.2.2 Ontology-based Distributed Knowledge Management Systems . . 20 2.1.2.3 Knowledge Management System Design Guideline . . . . . . . . 21 2.1.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.2 Semantic Web Technologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.2.1 The Evolution of the World Wide Web . . . . . . . . . . . . . . . . . . . . . 22 Personal Music Knowledge Base Contents 2.2.1.1 The Hypertext Web . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.2.1.2 The Normative Principles of Web Architecture . . . . . . . . . . . 23 2.2.1.3 The Semantic Web . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.2.2 Common Semantic Web Knowledge Representation Languages . . . . . . 25 2.2.3 Resource Description Levels and their Relations . . . . . . . . . . . . . . . 26 2.2.4 Semantic Web Knowledge Representation Models . . . . . . . . . . . . . . 29 2.2.4.1 Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 2.2.4.2 Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 2.2.4.3 Context Modelling . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 2.2.4.4 Storing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 2.2.4.5 Providing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 2.2.4.6 Consuming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 2.2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 2.3 Music Content and Context Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 2.3.1 Categories of Musical Characteristics . . . . . . . . . . . . . . . . . . . . . 37 2.3.2 Music Metadata Formats . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 2.3.3 Music Metadata Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 2.3.3.1 Audio Signal Carrier Indexing Services . . . . . . . . . . . . . . . . 41 2.3.3.2 Music Recommendation and Discovery Services . . . . . . . . . . 42 2.3.3.3 Music Content and Context Analysis Services . . . . . . . . . . . 43 2.3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 2.4 Personalisation and Environmental Context . . . . . . . . . . . . . . . . . . . . . . 44 2.4.1 User Modelling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 2.4.2 Context Modelling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 2.4.3 Stereotype Modelling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3 The Personal Music Knowledge Base 48 3.1 Foundations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 3.1.1 Knowledge Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 3.1.2 Knowledge Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 3.2 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 3.3 Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 3.3.1 User Account Initialisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 3.3.2 Individual Information Extraction . . . . . . . . . . . . . . . . . . . . . . . . 53 3.3.3 Information Service Choice . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 3.3.4 Proactive Update Notification . . . . . . . . . . . . . . . . . . . . . . . . . . 55 3.3.5 Information Exploration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 3.3.6 Personal Associations and Context . . . . . . . . . . . . . . . . . . . . . . . 56 3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 4 A Personal Music Knowledge Base 57 4.1 Knowledge Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 4.1.1 The Info Service Ontology . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 4.1.2 The Play Back Ontology and related Ontologies . . . . . . . . . . . . . . . . 61 4.1.2.1 The Ordered List Ontology . . . . . . . . . . . . . . . . . . . . . . 61 4.1.2.2 The Counter Ontology . . . . . . . . . . . . . . . . . . . . . . . . . 62 4.1.2.3 The Association Ontology . . . . . . . . . . . . . . . . . . . . . . . 64 4.1.2.4 The Play Back Ontology . . . . . . . . . . . . . . . . . . . . . . . . 65 4.1.3 The Recommendation Ontology . . . . . . . . . . . . . . . . . . . . . . . . 69 4.1.4 The Cognitive Characteristics Ontology and related Vocabularies . . . . . . 72 4.1.4.1 The Weighting Ontology . . . . . . . . . . . . . . . . . . . . . . . 72 4.1.4.2 The Cognitive Characteristics Ontology . . . . . . . . . . . . . . . 73 4.1.4.3 The Property Reification Vocabulary . . . . . . . . . . . . . . . . . 78 4.1.5 The Media Types Taxonomy . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 4.1.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 4.2 Knowledge Management System . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 4.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 5 Personal Music Knowledge Base in Practice 87 5.1 Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 5.1.1 AudioScrobbler RDF Service . . . . . . . . . . . . . . . . . . . . . . . . . . 87 5.1.2 PMKB ID3 Tag Extractor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 5.2 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 5.2.1 Reutilisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 5.2.2 Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 5.2.3 Reviews and Mentions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 5.2.4 Indexing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 5.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 6 Conclusion and Future Work 93 6.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 6.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

CHORUS Deliverable 2.1: State of the Art on Multimedia Search Engines

Based on the information provided by European projects and national initiatives related to multimedia search as well as domains experts that participated in the CHORUS Think-thanks and workshops, this document reports on the state of the art related to multimedia content search from, a technical, and socio-economic perspective. The technical perspective includes an up to date view on content based indexing and retrieval technologies, multimedia search in the context of mobile devices and peer-to-peer networks, and an overview of current evaluation and benchmark inititiatives to measure the performance of multimedia search engines. From a socio-economic perspective we inventorize the impact and legal consequences of these technical advances and point out future directions of research

Intelligent Tools for Drum Loop Retrieval and Generation

Large libraries of musical data are an increasingly common feature of contemporary computer-based music production practice, with producers often relying heavily on large, curated libraries of data such as loops and samples when making tracks. Drum loop libraries are a particularly common type of library in this context. However, their typically large size, coupled with often poor user interfaces means navigating and exploring them in a fast, easy and enjoyable way is not always possible. Additionally, writing a drum part for a whole track out of many drum loops can be a laborious process, requiring manually editing of many drum loops. The aim of this thesis is to contribute novel techniques based on Music Information Retrieval (MIR) and machine learning that make the process of writing drum tracks using drum loops faster, easier and more enjoyable. We primarily focus on tools for drum loop library navigation and exploration, with additional work on assistive generation of drum loops. We contribute proof-of-concept and prototype tools, Groove Explorer and Groove Explorer 2, for drum loop library exploration based on an interface applying similarity-based visual arrangement of drum loops. Work on Groove Explorer suggested that there were limitations in the existing state-of-the-art approaches to drum loop similarity modelling that must be addressed for tools such as ours to be successful. This was verified via a perceptual study, which identified possible areas of improvement in similarity modelling. Following this, we develop and evaluate a set of novel models for drum loop analysis that capture rhythmic structure and the perceptually relevant qualities of microtiming. Drawing from this, a new approach to drum loop similarity modelling was verified in context as part of Groove Explorer 2, which we evaluated via a user study. The results indicated that our approach could make drum loop library exploration faster, easier and more enjoyable. We finally present an automatic drum loop generation system, jaki, that uses a novel approach for drum loop generation according to user constraints, that could extend Groove Explorer 2 as a drum loop editing and composition tool. Combined, these two systems could offer an end-to-end solution to improved writing of drum tracks

Application of Common Sense Computing for the Development of a Novel Knowledge-Based Opinion Mining Engine

The ways people express their opinions and sentiments have radically changed in the past few years thanks to the advent of social networks, web communities, blogs, wikis and other online collaborative media. The distillation of knowledge from this huge amount of unstructured information can be a key factor for marketers who want to create an image or identity in the minds of their customers for their product, brand, or organisation. These online social data, however, remain hardly accessible to computers, as they are specifically meant for human consumption. The automatic analysis of online opinions, in fact, involves a deep understanding of natural language text by machines, from which we are still very far. Hitherto, online information retrieval has been mainly based on algorithms relying on the textual representation of web-pages. Such algorithms are very good at retrieving texts, splitting them into parts, checking the spelling and counting their words. But when it comes to interpreting sentences and extracting meaningful information, their capabilities are known to be very limited. Existing approaches to opinion mining and sentiment analysis, in particular, can be grouped into three main categories: keyword spotting, in which text is classified into categories based on the presence of fairly unambiguous affect words; lexical affinity, which assigns arbitrary words a probabilistic affinity for a particular emotion; statistical methods, which calculate the valence of affective keywords and word co-occurrence frequencies on the base of a large training corpus. Early works aimed to classify entire documents as containing overall positive or negative polarity, or rating scores of reviews. Such systems were mainly based on supervised approaches relying on manually labelled samples, such as movie or product reviews where the opinionist’s overall positive or negative attitude was explicitly indicated. However, opinions and sentiments do not occur only at document level, nor they are limited to a single valence or target. Contrary or complementary attitudes toward the same topic or multiple topics can be present across the span of a document. In more recent works, text analysis granularity has been taken down to segment and sentence level, e.g., by using presence of opinion-bearing lexical items (single words or n-grams) to detect subjective sentences, or by exploiting association rule mining for a feature-based analysis of product reviews. These approaches, however, are still far from being able to infer the cognitive and affective information associated with natural language as they mainly rely on knowledge bases that are still too limited to efficiently process text at sentence level. In this thesis, common sense computing techniques are further developed and applied to bridge the semantic gap between word-level natural language data and the concept-level opinions conveyed by these. In particular, the ensemble application of graph mining and multi-dimensionality reduction techniques on two common sense knowledge bases was exploited to develop a novel intelligent engine for open-domain opinion mining and sentiment analysis. The proposed approach, termed sentic computing, performs a clause-level semantic analysis of text, which allows the inference of both the conceptual and emotional information associated with natural language opinions and, hence, a more efficient passage from (unstructured) textual information to (structured) machine-processable data. The engine was tested on three different resources, namely a Twitter hashtag repository, a LiveJournal database and a PatientOpinion dataset, and its performance compared both with results obtained using standard sentiment analysis techniques and using different state-of-the-art knowledge bases such as Princeton’s WordNet, MIT’s ConceptNet and Microsoft’s Probase. Differently from most currently available opinion mining services, the developed engine does not base its analysis on a limited set of affect words and their co-occurrence frequencies, but rather on common sense concepts and the cognitive and affective valence conveyed by these. This allows the engine to be domain-independent and, hence, to be embedded in any opinion mining system for the development of intelligent applications in multiple fields such as Social Web, HCI and e-health. Looking ahead, the combined novel use of different knowledge bases and of common sense reasoning techniques for opinion mining proposed in this work, will, eventually, pave the way for development of more bio-inspired approaches to the design of natural language processing systems capable of handling knowledge, retrieving it when necessary, making analogies and learning from experience

Application of Common Sense Computing for the Development of a Novel Knowledge-Based Opinion Mining Engine

The ways people express their opinions and sentiments have radically changed in the past few years thanks to the advent of social networks, web communities, blogs, wikis and other online collaborative media. The distillation of knowledge from this huge amount of unstructured information can be a key factor for marketers who want to create an image or identity in the minds of their customers for their product, brand, or organisation. These online social data, however, remain hardly accessible to computers, as they are specifically meant for human consumption. The automatic analysis of online opinions, in fact, involves a deep understanding of natural language text by machines, from which we are still very far. Hitherto, online information retrieval has been mainly based on algorithms relying on the textual representation of web-pages. Such algorithms are very good at retrieving texts, splitting them into parts, checking the spelling and counting their words. But when it comes to interpreting sentences and extracting meaningful information, their capabilities are known to be very limited. Existing approaches to opinion mining and sentiment analysis, in particular, can be grouped into three main categories: keyword spotting, in which text is classified into categories based on the presence of fairly unambiguous affect words; lexical affinity, which assigns arbitrary words a probabilistic affinity for a particular emotion; statistical methods, which calculate the valence of affective keywords and word co-occurrence frequencies on the base of a large training corpus. Early works aimed to classify entire documents as containing overall positive or negative polarity, or rating scores of reviews. Such systems were mainly based on supervised approaches relying on manually labelled samples, such as movie or product reviews where the opinionist’s overall positive or negative attitude was explicitly indicated. However, opinions and sentiments do not occur only at document level, nor they are limited to a single valence or target. Contrary or complementary attitudes toward the same topic or multiple topics can be present across the span of a document. In more recent works, text analysis granularity has been taken down to segment and sentence level, e.g., by using presence of opinion-bearing lexical items (single words or n-grams) to detect subjective sentences, or by exploiting association rule mining for a feature-based analysis of product reviews. These approaches, however, are still far from being able to infer the cognitive and affective information associated with natural language as they mainly rely on knowledge bases that are still too limited to efficiently process text at sentence level. In this thesis, common sense computing techniques are further developed and applied to bridge the semantic gap between word-level natural language data and the concept-level opinions conveyed by these. In particular, the ensemble application of graph mining and multi-dimensionality reduction techniques on two common sense knowledge bases was exploited to develop a novel intelligent engine for open-domain opinion mining and sentiment analysis. The proposed approach, termed sentic computing, performs a clause-level semantic analysis of text, which allows the inference of both the conceptual and emotional information associated with natural language opinions and, hence, a more efficient passage from (unstructured) textual information to (structured) machine-processable data. The engine was tested on three different resources, namely a Twitter hashtag repository, a LiveJournal database and a PatientOpinion dataset, and its performance compared both with results obtained using standard sentiment analysis techniques and using different state-of-the-art knowledge bases such as Princeton’s WordNet, MIT’s ConceptNet and Microsoft’s Probase. Differently from most currently available opinion mining services, the developed engine does not base its analysis on a limited set of affect words and their co-occurrence frequencies, but rather on common sense concepts and the cognitive and affective valence conveyed by these. This allows the engine to be domain-independent and, hence, to be embedded in any opinion mining system for the development of intelligent applications in multiple fields such as Social Web, HCI and e-health. Looking ahead, the combined novel use of different knowledge bases and of common sense reasoning techniques for opinion mining proposed in this work, will, eventually, pave the way for development of more bio-inspired approaches to the design of natural language processing systems capable of handling knowledge, retrieving it when necessary, making analogies and learning from experience

Mapping Acoustic and Semantic Dimensions of Auditory Perception

Auditory categorisation is a function of sensory perception which allows humans to generalise across many different sounds present in the environment and classify them into behaviourally relevant categories. These categories cover not only the variance of acoustic properties of the signal but also a wide variety of sound sources. However, it is unclear to what extent the acoustic structure of sound is associated with, and conveys, different facets of semantic category information. Whether people use such data and what drives their decisions when both acoustic and semantic information about the sound is available, also remains unknown. To answer these questions, we used the existing methods broadly practised in linguistics, acoustics and cognitive science, and bridged these domains by delineating their shared space. Firstly, we took a model-free exploratory approach to examine the underlying structure and inherent patterns in our dataset. To this end, we ran principal components, clustering and multidimensional scaling analyses. At the same time, we drew sound labels’ semantic space topography based on corpus-based word embeddings vectors. We then built an LDA model predicting class membership and compared the model-free approach and model predictions with the actual taxonomy. Finally, by conducting a series of web-based behavioural experiments, we investigated whether acoustic and semantic topographies relate to perceptual judgements. This analysis pipeline showed that natural sound categories could be successfully predicted based on the acoustic information alone and that perception of natural sound categories has some acoustic grounding. Results from our studies help to recognise the role of physical sound characteristics and their meaning in the process of sound perception and give an invaluable insight into the mechanisms governing the machine-based and human classifications

Automatic execution of expressive music performance

The definition of computer models to represent the expressiveness of a musical performance, is useful to try to understand how and what way anyone can express expressive intentions in a music performance. The CaRo 2.0 is a computer model or software system that allows automatic computation in interactive way for rendering expressive musical scores. Initially, the exclusively on Microsoft environment, which limits the interest of the product. This thesis relates to the porting and integrationope