Visual Analytics for the Exploratory Analysis and Labeling of Cultural Data

Cultural data can come in various forms and modalities, such as text traditions, artworks, music, crafted objects, or even as intangible heritage such as biographies of people, performing arts, cultural customs and rites. The assignment of metadata to such cultural heritage objects is an important task that people working in galleries, libraries, archives, and museums (GLAM) do on a daily basis. These rich metadata collections are used to categorize, structure, and study collections, but can also be used to apply computational methods. Such computational methods are in the focus of Computational and Digital Humanities projects and research. For the longest time, the digital humanities community has focused on textual corpora, including text mining, and other natural language processing techniques. Although some disciplines of the humanities, such as art history and archaeology have a long history of using visualizations. In recent years, the digital humanities community has started to shift the focus to include other modalities, such as audio-visual data. In turn, methods in machine learning and computer vision have been proposed for the specificities of such corpora. Over the last decade, the visualization community has engaged in several collaborations with the digital humanities, often with a focus on exploratory or comparative analysis of the data at hand. This includes both methods and systems that support classical Close Reading of the material and Distant Reading methods that give an overview of larger collections, as well as methods in between, such as Meso Reading. Furthermore, a wider application of machine learning methods can be observed on cultural heritage collections. But they are rarely applied together with visualizations to allow for further perspectives on the collections in a visual analytics or human-in-the-loop setting. Visual analytics can help in the decision-making process by guiding domain experts through the collection of interest. However, state-of-the-art supervised machine learning methods are often not applicable to the collection of interest due to missing ground truth. One form of ground truth are class labels, e.g., of entities depicted in an image collection, assigned to the individual images. Labeling all objects in a collection is an arduous task when performed manually, because cultural heritage collections contain a wide variety of different objects with plenty of details. A problem that arises with these collections curated in different institutions is that not always a specific standard is followed, so the vocabulary used can drift apart from another, making it difficult to combine the data from these institutions for large-scale analysis. This thesis presents a series of projects that combine machine learning methods with interactive visualizations for the exploratory analysis and labeling of cultural data. First, we define cultural data with regard to heritage and contemporary data, then we look at the state-of-the-art of existing visualization, computer vision, and visual analytics methods and projects focusing on cultural data collections. After this, we present the problems addressed in this thesis and their solutions, starting with a series of visualizations to explore different facets of rap lyrics and rap artists with a focus on text reuse. Next, we engage in a more complex case of text reuse, the collation of medieval vernacular text editions. For this, a human-in-the-loop process is presented that applies word embeddings and interactive visualizations to perform textual alignments on under-resourced languages supported by labeling of the relations between lines and the relations between words. We then switch the focus from textual data to another modality of cultural data by presenting a Virtual Museum that combines interactive visualizations and computer vision in order to explore a collection of artworks. With the lessons learned from the previous projects, we engage in the labeling and analysis of medieval illuminated manuscripts and so combine some of the machine learning methods and visualizations that were used for textual data with computer vision methods. Finally, we give reflections on the interdisciplinary projects and the lessons learned, before we discuss existing challenges when working with cultural heritage data from the computer science perspective to outline potential research directions for machine learning and visual analytics of cultural heritage data

American Studies + Computational Humanities

While often commonly positioned at the intersection of computer science and digital humanities, computational humanities engages with other fields including data science, (computational) linguistics, and statistics.Such a transdisciplinary approach creates a digital ecology of data, algorithms, metadata, analytical and visualization tools, and new forms of scholarly expression that result from this research, as Christa Williford and Charles Henry, of the Council on Library and Information Resources, write. Text analysis, particularly the method of topic modeling, has enjoyed broad exposure within computational humanities. Given the scale of the corpus, computational methods were used to identify reprinted texts in 41,829 issues.The goal of the project is not to construct a definitive, empirical solution to the problem of nineteenth-century newspaper reprinting, Cordell writes, but to facilitate an iterative conversation between the large-scale, quantitative output generated by a corpus analysis algorithm and qualitative, literary-historical readings of the surprising texts that algorithm brings into focus. Suggesting a shift from distant reading, the project is part of a growing chorus of such work that argues that DH needs to expand beyond text to other forms such as photography and moving images, a shift that American studies has also called for. DV, therefore, focuses on how a critical use of computer vision can be used to analyze moving image culture. Since the majority of the algorithms are trained on twenty-first-century data held by companies like Google and platforms like Flickr, we need to question and adapt these algorithms using machine learning informed by our areas of inquiry

Image Segmentation methods for fine-grained OCR Document Layout Analysis

Digitization has changed history research. The materials are available, and online archives make it easier to find the correct information and speed up the search for information. The remaining challenge is how to use modern digital methods to analyze the text of historical documents in more detail. This is an active research topic in digital humanities and computer science areas. Document layout analysis is where computer vision object detection methods can be applied to historical documents to identify the document pages’ present objects (i.e., page elements). The recent development in deep learning based computer vision provides excellent tools for this purpose. However, most reviewed systems focus on coarse-grained methods, where only the high-level page elements are detected (e.g., text, figures, tables). Fine-grained detection methods are required to be able to analyze texts on a more detailed level; for example, footnotes and marginalia are distinguished from the body text to enable proper analysis. The thesis studies how image segmentation techniques can be used for fine-grained OCR document layout analysis. How to implement fine-grained page segmentation and region classification systems in practice, and what are the accuracy and the main challenges of such a system? The thesis includes implementing a layout analysis model that uses the instance segmentation method (Mask R-CNN). This implementation is compared against another existing layout analysis using the semantic segmentation method (U-net based P2PaLA implementation)

Learning from Patterns : Information Retrieval and Visualisation Issues Between Bioimage Informatics and Digital Humanities

The large amount of data generated in different fields, among which bioimage informatics and digital humanities, is increasingly requiring appropriate automatic processing techniques, such as computer vision, data mining and particular visualisation tools, to extract information out of complexity and to clearly display it. This has led, in digital humanities, to the use of pattern recognition techniques similar to those applied in biology, chemistry and medical studies, but where patterns to be analysed and segmented are extracted from texts, images, audiovisual and online media rather than from cells and tissues. Regularities can be recognised through machine learning, based on artificial neural networks that are modelled, to some extent, after the brain's structure, showing a variety of analogies between natural and artificial world. These processes can also add information to 3D models for cultural heritage: data mining technologies allow information retrieval from archives and repositories, as well as the comparison of data in order to better understand the context of-and relationships between-works of art, thus producing knowledge enhancement. Various tools to describe complexity are here analysed not only for their educational aim, but also for their heuristic value, allowing new discoveries and connections between different disciplines

Detecting Treasures in Museums with Artificial Intelligence

Museums around the world possess hundreds of thousands of priceless objects, which have stories to tell about human history. While students and scholars study them, even the general public is interested in these stories. If there is a way to automate the information delivery system about these objects it will be of immense value, e.g. it will support students to study these objects and speed up research. Adaptive blended learning options are conceivable, which can perfectly merge digital analysis and onsite viewing. Thus, the preparation and post-processing of studied objects is just as conceivable as the adequate acquisition of information for on-site studies. Examples of such solutions would be mobile apps and computer software that can be used for history and archaeology education as well. However, it is important to identify these objects correctly in order to build such solutions. Computer vision technologies in artificial intelligence (AI) can be used for this. Therefore, this paper will show how AI-algorithms can be used for digital humanities in novel ways, such as for detecting museum treasures

Blockchain, Leadership And Management: Business AS Usual Or Radical Disruption?

The Internet provided the world with interconnection. However, it did not provide it with trust. Trust is lacking everywhere in our society and is the reason for the existence of powerful intermediaries aggregating power. Trust is what prevents the digital world to take over. This has consequences for organisations: they are inefficient because time, energy, money and passion are wasted on verifying everything happens as decided. Managers play the role of intermediaries in such case: they connect experts with each others and instruct them of what to do. As a result, in our expert society, people's engagement is low because no one is there to inspire and empower them. In other words, our society faces an unprecedented lack of leadership. Provided all those shortcomings, the study imagines the potential repercussions, especially in the context of management, of implementing a blockchain infrastructure in any type of organisation. Indeed, the blockchain technology seems to be able to remedy to those issues, for this distributed and immutable ledger provides security, decentralisation and transparency. In the context of a blockchain economy, the findings show that value creation will be rearranged, with experts directly collaborating with each others, and hierarchy being eliminated. This could, in turn, render managers obsolete, as a blockchain infrastructure will automate most of the tasks. As a result, only a strong, action-oriented, leadership would maintain the organisation together. This leadership-in-action would consist in igniting people to take action; coach members of the organisations so that their contribution makes sense in the greater context of life

Combining Visual and Textual Features for Semantic Segmentation of Historical Newspapers

The massive amounts of digitized historical documents acquired over the last decades naturally lend themselves to automatic processing and exploration. Research work seeking to automatically process facsimiles and extract information thereby are multiplying with, as a first essential step, document layout analysis. If the identification and categorization of segments of interest in document images have seen significant progress over the last years thanks to deep learning techniques, many challenges remain with, among others, the use of finer-grained segmentation typologies and the consideration of complex, heterogeneous documents such as historical newspapers. Besides, most approaches consider visual features only, ignoring textual signal. In this context, we introduce a multimodal approach for the semantic segmentation of historical newspapers that combines visual and textual features. Based on a series of experiments on diachronic Swiss and Luxembourgish newspapers, we investigate, among others, the predictive power of visual and textual features and their capacity to generalize across time and sources. Results show consistent improvement of multimodal models in comparison to a strong visual baseline, as well as better robustness to high material variance