Music and the scholarship around it can be challenging to present in the forms associated with books and articles – primarily linear and with an emphasis on the static and visual over the sonic and interactive. We introduce the Lohengrin TimeMachine, a multiple-path multimedia app, optimised for a touch-screen tablet. The app offers two essays about motifs in the opera – one in textual form, and one a 30-minute video. These linear narratives are supported by audio examples, along with dynamic links that take the reader into a fully-interactive exploration of the occurrence of motifs across the opera. The reader’s journey is supported with recorded music and scores, as well as novel visualisations of the orchestration and the timeline of the opera. The app is designed to operate over standards-based web documents published online, with extension and reuse in mind. In this paper, we describe the app, its underlying technology, and the journeys it supports

David Lewis

David Rizo

Kevin R. Page

Laurence Dreyfus

Stefan Münnich

Humanities Commons

Music Encoding Conference Proceedings 2021 49Lohengrin TimeMachine: Musicological Multi-media Made with MELDDavid Lewis       Kevin R. Page University of Oxford e-Research Centre, UK   University of Oxford e-Research Centre, UK david.lewis@oerc.ox.ac.uk     kevin.page@oerc.ox.ac.ukLaurence Dreyfus Faculty of Music, University of Oxford, UK laurence.dreyfus@magd.ox.ac.ukAbstractMusic and the scholarship around it can be challenging to present in the forms associated with books and articles – primarily linear and with an emphasis on the static and visual over the sonic and interactive. We in-troduce the Lohengrin TimeMachine, a multiple-path multimedia app, optimised for a touch-screen tablet. The app offers two essays about motifs in the opera – one in textual form, and one a 30-minute video. These linear narratives are supported by audio examples, along with dynamic links that take the reader into a fully-interac-tive exploration of the occurrence of motifs across the opera. The reader’s journey is supported with recorded music and scores, as well as novel visualisations of the orchestration and the timeline of the opera. The app is designed to operate over standards-based web documents published online, with extension and reuse in mind. In this paper, we describe the app, its underlying technology, and the journeys it supports.IntroductionMusicology is often presented in linear, textual form, even though its materials are complex and multimodal. Scholarship must necessarily simplify and dramatically reduce complexity in order to make an ordered prose argument that others can follow. More detailed interaction with the scholar’s observations occurs almost en-tirely through other resources. We present the Lohengrin TimeMachine, a web application in which linear musicological scholarship (in the form of an essay and a 30-minute video) is augmented by user-driven interactive, multiple-path exploration of related material that can be explored with or independently from the narrative. The material, which includes musical notation, audio, analytical content, and textual quotation is accessed through a touchscreen-opti-mised web application for guided and self-directed exploration. Underlying resources are published on the web using standard file formats (including MEI and TEI) and associated with machine-readable Linked Data, supporting the creation of additional interfaces or analyses.The application is intended to be accessible to enthusiastic amateurs as well as scholars, and concerns Wagner’s use of motifs in his early opera, Lohengrin, advocating a more sophisticated understanding than ‘Leitmotif’ guides often communicate. It is optimised for use with a tablet, without excluding other means of browsing. The musicological content was conceived, written and, for the video, presented by co-author Lau-rence Dreyfus, a Wagner specialist, with the companion realised through a multi-disciplinary collaboration.The Lohengrin TimeMachine allows users to explore the compositional devices Wagner uses to vary his motifs, browsing the whole opera for motif occurrences and their musical and textual contexts. Visualisations and recordings support the analysis, making it accessible to an audience that might otherwise struggle with a Wag-nerian orchestral score. Exploration of this material can follow or be triggered by the musicological article and video, but can also be entirely reader-driven, with free browsing of the curated musical landscape.© Published in 2022 with Humanities Commons under a CC BY-NC-ND 4.0 (https://creativecommons.org/licenses/by-nc-nd/4.0/) license.50Figure 1: The Video view of the Lohengrin TimeMachine. Relevant links appear in the sidebar on the right as the video progresses. The opera timeline at the bottom is present in all views.The Application and Its DataThe application dynamically constructs views using data published openly using standard file formats and Linked Data frameworks [3]. Materials are retrieved live from a public web server. Music notation is published as MEI, and text, including the main article, as TEI. Relationships and more abstract information are made available using the Resource Description Framework (RDF),1 serialised as JSON-LD2 and N-Quads3: annotations use the Web Annotation model4 [1], while much of the musical information is represented using the Music Ontology.5The sum of all these forms an independent, repurposable, and open Research Object [5]. Interactive user views are generated directly and dynamically in the browser from this knowledge graph using novel visualisa-tions, which enable the user to navigate all possible paths through the evidential multimodal materials.Technical ImplementationThe TimeMachine is constructed as a MELD application – that is, it uses the MELD 2.0 (Music Encoding and Linked Data) framework6 [7]. MELD traverses Linked Data graphs, ‘following its nose’ by successively requesting web-accessible Linked Data – with each set of statements read potentially giving links to new URIs to explore. From this data, MELD builds up a local knowledge graph within the browser, allowing apps to discover, select and filter relevant information. The MELD framework provides reusable components for creating and retriev-1  https://www.w3.org/TR/rdf11-concepts/ (accessed January 12, 2022).2  https://www.w3.org/TR/json-ld/ (accessed January 12, 2022). 3  https://www.w3.org/TR/n-quads/ (accessed January 12, 2022). 4  https://www.w3.org/TR/annotation-vocab/ (accessed January 12, 2022). 5  http://musicontology.com/specification/ (accessed January 12, 2022). 6  https://meld.web.ox.ac.uk/ (accessed January 12, 2022). Music Encoding Conference Proceedings 2021 51Figure 2: The Essay view, with the text in the centre, the opera timeline (below) and links to other views (right). The Motifs tab on the right, when selected, allows navigation within the essay text by motif iteration number, in a similar manner to a conventional index.ing annotations, as well as for displaying and interacting with musical, textual, graphical and audio-visual ma-terials. Other uses of the framework include an interactive composition [2], a masterclass annotation app [4], and a visualiser for comparing piano performances [6].MELD is written in Javascript (as are its apps, including the Lohengrin TimeMachine). It uses the React frame-work7 and operates over resources and data structured using Web standards. Linked Data consumed by the app uses and extends that which can be processed by core MELD libraries, drawing on ontologies including the Music Ontology, Dublin Core,8 FRBR,9 and, crucially, Web Annotations. FRBR is used here not only at the level of the complete work, but also to relate the abstract concept of, for example, the Frageverbot motif to the iterations of that motif that occur in the original score and, from those, to the editions and recordings that we include.Music notation is encoded as MEI, rendered using a MELD component that calls the Verovio toolkit.10 TEI is used for textual content, including the essay, commentary and historical texts, and is rendered to screen using CETEIcean.11 Both these are Javascript libraries that preserve structures and IDs in their output where possible. Where Verovio outputs SVG images, CETEIcean creates HTML with custom tags. Both of these are placed within the Document Object Model (DOM) of their web page and can be easily manipulated with JavaScript and styled using CSS.MELD applications such as the Lohengrin TimeMachine start the MELD graph traversal based on one or more initial URLs that point to RDF that declares the area of interest – in this case the Frageverbot motif and the 7  https://reactjs.org/ (accessed January 12, 2022). 8  https://www.dublincore.org/specifications/dublin-core/ (accessed January 12, 2022). 9  https://vocab.org/frbr/ (accessed January 12, 2022). 10  https://www.verovio.org/ (accessed January 12, 2022). 11  https://teic.github.io/CETEIcean/ (accessed January 12, 2022). 52Figure 3: Diagram showing the main pages of the application and the flows between them. The TimeMachine view is a central navigation point, both for selecting single iterations of the motif and for choosing a second with which to compare.essays. A local knowledge graph is built within a certain number of traversal steps from that. The application then dynamically selects the components of the interface based on the information and relationships to media that have been discovered. Much of the interface is constructed of generic MELD components, which are then styled for the specific look of the app using CSS.Using the ApplicationA user visiting the app website will initially encounter a landing page with links to several entry points for exploring the app (a diagram of paths the app offers between views is given as Figure 3). The essay and video themselves are integrated in the app, providing important narrative grounding and intuitive starting points. Readers or viewers are presented with relevant links into various other application views as they progress through the narrative (Figures 1 and 2). Most links point to a form of the Inspector view (Figure 4). This gives the user a variety of information and vi-sualisations about one iteration of the motif being discussed. In the terms of this app, an iteration of a musical idea is one instantiation of it at a particular point in the music (thus, the Frageverbot motif has 17 iterations in the opera itself, and one that occurs in a cut scene). Two iterations can be selected for side-by-side juxtaposi-tion, using a separate Comparison view (Figure 5).Selecting motif iterations from an overview of a complete opera can be daunting, and in this case is achieved through the TimeMachine view (Figure 6). Users can also flick left or right through motif occurrences, visualised in a carousel-like rapid overview.Many types of information and visualisation recur within different page-level views. Audio and textual infor-mation, including libretto (or ‘poem’) content, are made available – this is both a crucial reflection of the inte-grated nature of Wagner’s artistic conception and a practical help to those for whom the score is not an easy way of engaging with the material. Throughout the app, we provide two different visualisations for music notational content. Vocal scores are rendered from MEI using Verovio, with structural analysis (musical sections) dynamically overlaid  (Figure 4, left pane). Activating these annotations triggers audio playback from the beginning of that section.For full orchestral scores, a new, more abstract, notational visualisation of MEI is used, simplifying the com-plexity of a Wagnerian orchestral score. In our Orchestration pane, each instrument playing at a particular time is shown as a coloured ribbon, with the instrument’s section of the orchestra providing the colour (Figure 4, right pane). This makes clear differences in orchestration that may be made invisible by the reduction to vocal score. The visualisation is generated live in the browser based on a full score in MEI, and is rendered as an SVG image. It is implemented as a reusable React component within the meld-clients-core module, and so can be Music Encoding Conference Proceedings 2021 53easily incorporated into other applications. The list of instruments or singers, and the divisions of the orchestra or choir can be customised for any specific application.For an opera thousands of bars long, overviews are crucial. In addition to the TimeMachine view itself, every view has a timeline at the bottom, which shows all occurrences of a motif, providing a visual summary and a base for navigation. The view summarises the sequence within the opera, supporting quick comparisons, and also acting as an index to detail views.DiscussionThe application is possible because of a combination of factors. With a musicological study that was conceived by its author for interactive exploration, and collaborating on design with that author, we can ensure that the visualisations we provide are relevant and add to the scholarly argument and to a user’s understanding of the material. On a technological level, the combination of web standards and formats – especially MEI – that are easily addressed using URLs supports building rich, specialised applications without the need for bespoke or closed data formats. Furthermore, open-source rendering libraries such as Verovio and CETEIcean, which preserve to a large extent the structure of their originating documents and propagate IDs through from the XML to the rendered result, provide essential support for semantics on the data level to be transformed into interactive exploration on the application level in a principled and re-usable way.The app itself is quick and responsive once loaded, and, although it takes 10–20 seconds on initial load to process and reframe the graph for use, we believe this to be within reasonable expectations for a modern Figure 4: Inspector view, showing the Vocal Score pane (Verovio) and the Orchestration summary pane.54Figure 5: Comparison view, showing the Vocal Score panes for two iterations of the Frageverbot motif. A structural analysis (x and y seg-ments) is indicated on the score. Both iterations are labelled clearly in the timeline, so their place in the opera is clear. Colour coding in the timeline of motifs in the ‘Magical’ and the ‘Grail’ realm (as magenta and turquoise lines, respectively) draws attention to and gives context for some of the differences in timbre and setting.web application. This processing time itself depends substantially on the Javascript JSON-LD library,12 and has improved by a factor of 10 as the library has matured.By relying on walking Linked Data graphs to discover information and resources, MELD has no requirement for its materials to be collocated on the same server or published by the same institution. Although individual MELD components, such as the score component, have certain expectations for supplied parameters, the framework is not strongly opinionated or exclusive about the ontologies of its knowledge graph, and applica-tions frequently transform graph data before it goes into these components. This makes the framework highly flexible for external data. There is no reason in principle why information and media could not be drawn from multiple external sources, such as Wikidata, Europeana, or YouTube, nor anything to prevent others building MELD apps using the Lohengrin TimeMachine’s resources.The MELD framework has already been used for a variety of musically-rich interfaces. We believe that the Lohengrin TimeMachine can serve as a model for many more.AcknowledgementsThe Lohengrin TimeMachine was developed by the Arts and Humanities Research Council-funded Unlocking Musicology project (AH/R004803/1) as a companion to the essay “Asking a Forbidden Question: A new view of musical themes in Wagner’s Lohengrin” by Prof. Laurence Dreyfus (University of Oxford Faculty of Music). Un-locking Musicology builds on research undertaken during the Transforming Musicology project (AH/L006820/1) with important contributions from Carolin Rindfleisch. The MELD framework was created, with David Weigl as lead developer, as part of the Fusing Audio and Semantic Technologies for Intelligent Music Production and 12  https://github.com/digitalbazaar/jsonld.js (accessed January 12, 2022). Music Encoding Conference Proceedings 2021 55Consumption project, funded by the UK Engineering and Physical Sciences Research Council (EP/L019981/1). We thank Will Elsom for his work on the Lohengrin TimeMachine design specification, Steve Pierce and Oxford Filmshed Productions for the video tutorial, and Ralph Woodward for his music engravings.Works Cited[1] Barnes, Marcus Emmanuel, Natkeeran Ledchumykanthan, Kim Pham, and Kirsta Stapelfeldt. “Annotation-Based Enrichment of Digital Objects Using Open-Source Frameworks” Code4Lib Journal 37 (2017), https://journal.code4lib.org/articles/12582.[2] Kallionpää, Maria, Chris Greenhalgh, Adrian Hazzard, David M. Weigl, Kevin R. Page, and Steve Benford. “Composing and Realising a Game-like Performance for Disklavier and Electronics” in Proceedings of the International Conference on New Interfaces for Musical Expression (NIME 2017), 464–469, https://doi.org/10.5281/zenodo.1176318.[3] Lewis, David, David M. Weigl, Joanna Bullivant, and Kevin R. Page. “Publishing Musicology Using Multimedia Digital Libraries: Creating Interactive Articles Through a Framework for Linked Data and MEI” in Proceedings of the 5th International Conference on Digital Libraries for Musicology (DLfM 2018), 21–25, https://doi.org/10.1145/3273024.3273038.[4] Lewis, David, David M. Weigl, and Kevin R. Page. “Musicological Observations During Rehearsal and Performance: a Linked Data Digital Library for Annotations” in Proceedings of the 6th International Conference on Digital Libraries for Musicology (DLfM 2019), 1–8, https://doi.org/10.1145/3358664.3358669.[5] Page, Kevin R., Ben Fields, David De Roure, Tim Crawford, and J. Stephen Downie. “Capturing the Workflows of Music Information Retrieval for Repeatability and Reuse” Journal of Intelligent Information Systems 41, no. 3 (2013), 435–459, https://doi.org/10.1007/s10844-013-0260-9.[6] Weigl, David M., Carlos Cancino-Chacón, Martin Bonev, and Werner Goebl. “Linking and Visualising Performance Data and Semantic Music Encodings in Real-Time” Late Breaking/Demo at the 20th International Society for Music Information Retrieval Conference (ISMIR 2019), https://archives.ismir.net/ismir2019/latebreaking/000049.pdf.[7] Weigl, David M., and Kevin R. Page. “A Framework for Distributed Semantic Annotation of Musical Score: ‘Take It To The Bridge!’” in Proceedings of the 18th International Society for Music Information Retrieval Conference (ISMIR 2017), 221–228, https://archives.ismir.net/ismir2017/paper/000190.pdf.Figure 6: The TimeMachine view, set to browse by vocal score. Flicking sideways scrolls through all the iterations of the motif in the opera in order of occurrence. Browsing is also possible using visualisations of the Orchestration or Libretto  (‘Poem’, as Wagner called it).

Lohengrin TimeMachine: Musicological Multimedia Made with MELD

https://hcommons.org/deposits/download/hc:45982/CONTENT/mec2021_lewis_et_al_lohengrin-timemachine.pdf/

Lohengrin TimeMachine: Musicological Multimedia Made with MELD

Abstract

Similar works

Full text

Available Versions

Humanities Commons