Computing Methodologies Supporting the Preservation of Electroacoustic Music from Analog Magnetic Tape

Electroacoustic music on analog magnetic tape is characterized by several specificities related to the carrier that have to be considered during the creation of a digital preservation copy of a document. The tape recorder need to be setup with the correct speed and equalization; moreover, the magnetic tape could present some intentional or unintentional alterations. During both the creation and the musicological analysis of a digital preservation copy, the quality of the work could be affected by human attention. This paper presents a methodology based on neural networks able to recognize and classify the alterations of a magnetic tape from the video of the tape itself flowing in the head of the tape recorder. Furthermore, some machine learning techniques has been tested to recognize equalization of a tape from its background noise. The encouraging results open the way to innovative tools able to unburden audio technicians and musicologists from repetitive tasks and improve the quality of their works

Multimodal Indexing of Presentation Videos

This thesis presents four novel methods to help users efficiently and effectively retrieve information from unstructured and unsourced multimedia sources, in particular the increasing amount and variety of presentation videos such as those in e-learning, conference recordings, corporate talks, and student presentations. We demonstrate a system to summarize, index and cross-reference such videos, and measure the quality of the produced indexes as perceived by the end users. We introduce four major semantic indexing cues: text, speaker faces, graphics, and mosaics, going beyond standard tag based searches and simple video playbacks. This work aims at recognizing visual content "in the wild", where the system cannot rely on any additional information besides the video itself. For text, within a scene text detection and recognition framework, we present a novel locally optimal adaptive binarization algorithm, implemented with integral histograms. It determines of an optimal threshold that maximizes the between-classes variance within a subwindow, with computational complexity independent from the size of the window itself. We obtain character recognition rates of 74%, as validated against ground truth of 8 presentation videos spanning over 1 hour and 45 minutes, which almost doubles the baseline performance of an open source OCR engine. For speaker faces, we detect, track, match, and finally select a humanly preferred face icon per speaker, based on three quality measures: resolution, amount of skin, and pose. We register a 87% accordance (51 out of 58 speakers) between the face indexes automatically generated from three unstructured presentation videos of approximately 45 minutes each, and human preferences recorded through Mechanical Turk experiments. For diagrams, we locate graphics inside frames showing a projected slide, cluster them according to an on-line algorithm based on a combination of visual and temporal information, and select and color-correct their representatives to match human preferences recorded through Mechanical Turk experiments. We register 71% accuracy (57 out of 81 unique diagrams properly identified, selected and color-corrected) on three hours of videos containing five different presentations. For mosaics, we combine two existing suturing measures, to extend video images into in-the-world coordinate system. A set of frames to be registered into a mosaic are sampled according to the PTZ camera movement, which is computed through least square estimation starting from the luminance constancy assumption. A local features based stitching algorithm is then applied to estimate the homography among a set of video frames and median blending is used to render pixels in overlapping regions of the mosaic. For two of these indexes, namely faces and diagrams, we present two novel MTurk-derived user data collections to determine viewer preferences, and show that they are matched in selection by our methods. The net result work of this thesis allows users to search, inside a video collection as well as within a single video clip, for a segment of presentation by professor X on topic Y, containing graph Z

Model-based analysis of noisy musical recordings with application to audio restoration

This thesis proposes digital signal processing algorithms for noise reduction and enhancement of audio signals. Approximately half of the work concerns signal modeling techniques for suppression of localized disturbances in audio signals, such as impulsive noise and low-frequency pulses. In this regard, novel algorithms and modifications to previous propositions are introduced with the aim of achieving a better balance between computational complexity and qualitative performance, in comparison with other schemes presented in the literature. The main contributions related to this set of articles are: an efficient algorithm for suppression of low-frequency pulses in audio signals; a scheme for impulsive noise detection that uses frequency-warped linear prediction; and two methods for reconstruction of audio signals within long gaps of missing samples. The remaining part of the work discusses applications of sound source modeling (SSM) techniques to audio restoration. It comprises application examples, such as a method for bandwidth extension of guitar tones, and discusses the challenge of model calibration based on noisy recorded sources. Regarding this matter, a frequency-selective spectral analysis technique called frequency-zooming ARMA (FZ-ARMA) modeling is proposed as an effective way to estimate the frequency and decay time of resonance modes associated with the partials of a given tone, despite the presence of corrupting noise in the observable signal.reviewe

Cultural Context-Aware Models and IT Applications for the Exploitation of Musical Heritage

Information engineering has always expanded its scope by inspiring innovation in different scientific disciplines. In particular, in the last sixty years, music and engineering have forged a strong connection in the discipline known as “Sound and Music Computing”. Musical heritage is a paradigmatic case that includes several multi-faceted cultural artefacts and traditions. Several issues arise from the analog-digital transfer of cultural objects, concerning their creation, preservation, access, analysis and experiencing. The keystone is the relationship of these digitized cultural objects with their carrier and cultural context. The terms “cultural context” and “cultural context awareness” are delineated, alongside the concepts of contextual information and metadata. Since they maintain the integrity of the object, its meaning and cultural context, their role is critical. This thesis explores three main case studies concerning historical audio recordings and ancient musical instruments, aiming to delineate models to preserve, analyze, access and experience the digital versions of these three prominent examples of musical heritage. The first case study concerns analog magnetic tapes, and, in particular, tape music, a particular experimental music born in the second half of the XX century. This case study has relevant implications from the musicology, philology and archivists’ points of view, since the carrier has a paramount role and the tight connection with its content can easily break during the digitization process or the access phase. With the aim to help musicologists and audio technicians in their work, several tools based on Artificial Intelligence are evaluated in tasks such as the discontinuity detection and equalization recognition. By considering the peculiarities of tape music, the philological problem of stemmatics in digitized audio documents is tackled: an algorithm based on phylogenetic techniques is proposed and assessed, confirming the suitability of these techniques for this task. Then, a methodology for a historically faithful access to digitized tape music recordings is introduced, by considering contextual information and its relationship with the carrier and the replay device. Based on this methodology, an Android app which virtualizes a tape recorder is presented, together with its assessment. Furthermore, two web applications are proposed to faithfully experience digitized 78 rpm discs and magnetic tape recordings, respectively. Finally, a prototype of web application for musicological analysis is presented. This aims to concentrate relevant part of the knowledge acquired in this work into a single interface. The second case study is a corpus of Arab-Andalusian music, suitable for computational research, which opens new opportunities to musicological studies by applying data-driven analysis. The description of the corpus is based on the five criteria formalized in the CompMusic project of the University Pompeu Fabra of Barcelona: purpose, coverage, completeness, quality and re-usability. Four Jupyter notebooks were developed with the aim to provide a useful tool for computational musicologists for analyzing and using data and metadata of such corpus. The third case study concerns an exceptional historical musical instrument: an ancient Pan flute exhibited at the Museum of Archaeological Sciences and Art of the University of Padova. The final objective was the creation of a multimedia installation to valorize this precious artifact and to allow visitors to interact with the archaeological find and to learn its history. The case study provided the opportunity to study a methodology suitable for the valorization of this ancient musical instrument, but also extendible to other artifacts or museum collections. Both the methodology and the resulting multimedia installation are presented, followed by the assessment carried out by a multidisciplinary group of experts

Senseable Spaces: from a theoretical perspective to the application in augmented environments

openGrazie all’ enorme diffusione di dispositivi senzienti nella vita di tutti i giorni, nell’ ultimo decennio abbiamo assistito ad un cambio definitivo nel modo in cui gli utenti interagiscono con lo spazio circostante. Viene coniato il termine Spazio Sensibile, per descrivere quegli spazi in grado di fornire servizi contestuali agli utenti, misurando e analizzando le dinamiche che in esso avvengono, e di reagire conseguentemente a questo continuo flusso di dati bidirezionale. La ricerca è stata condotta abbracciando diversi domini di applicazione, le cui singole esigenze hanno reso necessario testare il concetto di Spazi Sensibili in diverse declinazioni, mantenendo al centro della ricerca l’utente, con la duplice accezione di end-user e manager. Molteplici sono i contributi rispetto allo stato dell’ arte. Il concetto di Spazio Sensibile è stato calato nel settore dei Beni Culturali, degli Spazi Pubblici, delle Geosciences e del Retail. I casi studio nei musei e nella archeologia dimostrano come l’ utilizzo della Realtà Aumentata possa essere sfruttata di fronte a un dipinto o in outdoor per la visualizzazione di modelli complessi, In ambito urbano, il monitoraggio di dati generati dagli utenti ha consentito di capire le dinamiche di un evento di massa, durante il quale le stesse persone fruivano di servizi contestuali. Una innovativa applicazione di Realtà Aumentata è stata come servizio per facilitare l’ ispezione di fasce tampone lungo i fiumi, standardizzando flussi di dati e modelli provenienti da un Sistema Informativo Territoriale. Infine, un robusto sistema di indoor localization è stato istallato in ambiente retail, per scopi classificazione dei percorsi e per determinare le potenzialità di un punto vendita. La tesi è inoltre una dimostrazione di come Space Sensing e Geomatica siano discipline complementari: la geomatica consente di acquisire e misurare dati geo spaziali e spazio temporali a diversa scala, lo Space Sensing utilizza questi dati per fornire servizi all’ utente precisi e contestuali.Given the tremendous growth of ubiquitous services in our daily lives, during the last few decades we have witnessed a definitive change in the way users' experience their surroundings. At the current state of art, devices are able to sense the environment and users’ location, enabling them to experience improved digital services, creating synergistic loop between the use of the technology, and the use of the space itself. We coined the term Senseable Space, to define the kinds of spaces able to provide users with contextual services, to measure and analyse their dynamics and to react accordingly, in a seamless exchange of information. Following the paradigm of Senseable Spaces as the main thread, we selected a set of experiences carried out in different fields; central to this investigation there is of course the user, placed in the dual roles of end-user and manager. The main contribution of this thesis lies in the definition of this new paradigm, realized in the following domains: Cultural Heritage, Public Open Spaces, Geosciences and Retail. For the Cultural Heritage panorama, different pilot projects have been constructed from creating museum based installations to developing mobile applications for archaeological settings. Dealing with urban areas, app-based services are designed to facilitate the route finding in a urban park and to provide contextual information in a city festival. We also outlined a novel application to facilitate the on-site inspection by risk managers thanks to the use of Augmented Reality services. Finally, a robust indoor localization system has been developed, designed to ease customer profiling in the retail sector. The thesis also demonstrates how Space Sensing and Geomatics are complementary to one another, given the assumption that the branches of Geomatics cover all the different scales of data collection, whilst Space Sensing gives one the possibility to provide the services at the correct location, at the correct time.INGEGNERIA DELL'INFORMAZIONEembargoed_20181001Pierdicca, RobertoPierdicca, Robert

Senseable Spaces: from a theoretical perspective to the application in augmented environments

Grazie all’ enorme diffusione di dispositivi senzienti nella vita di tutti i giorni, nell’ ultimo decennio abbiamo assistito ad un cambio definitivo nel modo in cui gli utenti interagiscono con lo spazio circostante. Viene coniato il termine Spazio Sensibile, per descrivere quegli spazi in grado di fornire servizi contestuali agli utenti, misurando e analizzando le dinamiche che in esso avvengono, e di reagire conseguentemente a questo continuo flusso di dati bidirezionale. La ricerca è stata condotta abbracciando diversi domini di applicazione, le cui singole esigenze hanno reso necessario testare il concetto di Spazi Sensibili in diverse declinazioni, mantenendo al centro della ricerca l’utente, con la duplice accezione di end-user e manager. Molteplici sono i contributi rispetto allo stato dell’ arte. Il concetto di Spazio Sensibile è stato calato nel settore dei Beni Culturali, degli Spazi Pubblici, delle Geosciences e del Retail. I casi studio nei musei e nella archeologia dimostrano come l’ utilizzo della Realtà Aumentata possa essere sfruttata di fronte a un dipinto o in outdoor per la visualizzazione di modelli complessi, In ambito urbano, il monitoraggio di dati generati dagli utenti ha consentito di capire le dinamiche di un evento di massa, durante il quale le stesse persone fruivano di servizi contestuali. Una innovativa applicazione di Realtà Aumentata è stata come servizio per facilitare l’ ispezione di fasce tampone lungo i fiumi, standardizzando flussi di dati e modelli provenienti da un Sistema Informativo Territoriale. Infine, un robusto sistema di indoor localization è stato istallato in ambiente retail, per scopi classificazione dei percorsi e per determinare le potenzialità di un punto vendita. La tesi è inoltre una dimostrazione di come Space Sensing e Geomatica siano discipline complementari: la geomatica consente di acquisire e misurare dati geo spaziali e spazio temporali a diversa scala, lo Space Sensing utilizza questi dati per fornire servizi all’ utente precisi e contestuali.Given the tremendous growth of ubiquitous services in our daily lives, during the last few decades we have witnessed a definitive change in the way users' experience their surroundings. At the current state of art, devices are able to sense the environment and users’ location, enabling them to experience improved digital services, creating synergistic loop between the use of the technology, and the use of the space itself. We coined the term Senseable Space, to define the kinds of spaces able to provide users with contextual services, to measure and analyse their dynamics and to react accordingly, in a seamless exchange of information. Following the paradigm of Senseable Spaces as the main thread, we selected a set of experiences carried out in different fields; central to this investigation there is of course the user, placed in the dual roles of end-user and manager. The main contribution of this thesis lies in the definition of this new paradigm, realized in the following domains: Cultural Heritage, Public Open Spaces, Geosciences and Retail. For the Cultural Heritage panorama, different pilot projects have been constructed from creating museum based installations to developing mobile applications for archaeological settings. Dealing with urban areas, app-based services are designed to facilitate the route finding in a urban park and to provide contextual information in a city festival. We also outlined a novel application to facilitate the on-site inspection by risk managers thanks to the use of Augmented Reality services. Finally, a robust indoor localization system has been developed, designed to ease customer profiling in the retail sector. The thesis also demonstrates how Space Sensing and Geomatics are complementary to one another, given the assumption that the branches of Geomatics cover all the different scales of data collection, whilst Space Sensing gives one the possibility to provide the services at the correct location, at the correct time

Eye Detection and Face Recognition Across the Electromagnetic Spectrum

Biometrics, or the science of identifying individuals based on their physiological or behavioral traits, has increasingly been used to replace typical identifying markers such as passwords, PIN numbers, passports, etc. Different modalities, such as face, fingerprint, iris, gait, etc. can be used for this purpose. One of the most studied forms of biometrics is face recognition (FR). Due to a number of advantages over typical visible to visible FR, recent trends have been pushing the FR community to perform cross-spectral matching of visible images to face images from higher spectra in the electromagnetic spectrum.;In this work, the SWIR band of the EM spectrum is the primary focus. Four main contributions relating to automatic eye detection and cross-spectral FR are discussed. First, a novel eye localization algorithm for the purpose of geometrically normalizing a face across multiple SWIR bands for FR algorithms is introduced. Using a template based scheme and a novel summation range filter, an extensive experimental analysis show that this algorithm is fast, robust, and highly accurate when compared to other available eye detection methods. Also, the eye locations produced by this algorithm provides higher FR results than all other tested approaches. This algorithm is then augmented and updated to quickly and accurately detect eyes in more challenging unconstrained datasets, spanning the EM spectrum. Additionally, a novel cross-spectral matching algorithm is introduced that attempts to bridge the gap between the visible and SWIR spectra. By fusing multiple photometric normalization combinations, the proposed algorithm is not only more efficient than other visible-SWIR matching algorithms, but more accurate in multiple challenging datasets. Finally, a novel pre-processing algorithm is discussed that bridges the gap between document (passport) and live face images. It is shown that the pre-processing scheme proposed, using inpainting and denoising techniques, significantly increases the cross-document face recognition performance

Nonlinear and distributed sensory estimation

Methods to improve performance of sensors with regard to sensor nonlinearity, sensor noise and sensor bandwidths are investigated and new algorithms are developed. The necessity of the proposed research has evolved from the ever-increasing need for greater precision and improved reliability in sensor measurements. After describing the current state of the art of sensor related issues like nonlinearity and bandwidth, research goals are set to create a new trend on the usage of sensors. We begin the investigation with a detailed distortion analysis of nonlinear sensors. A need for efficient distortion compensation procedures is further justified by showing how a slight deviation from the linearity assumption leads to a very severe distortion in time and in frequency domains. It is argued that with a suitable distortion compensation technique the danger of having an infinite bandwidth nonlinear sensory operation, which is dictated by nonlinear distortion, can be avoided. Several distortion compensation techniques are developed and their performance is validated by simulation and experimental results. Like any other model-based technique, modeling errors or model uncertainty affects performance of the proposed scheme, this leads to the innovation of robust signal reconstruction. A treatment for this problem is given and a novel technique, which uses a nominal model instead of an accurate model and produces the results that are robust to model uncertainty, is developed. The means to attain a high operating bandwidth are developed by utilizing several low bandwidth pass-band sensors. It is pointed out that instead of using a single sensor to measure a high bandwidth signal, there are many advantages of using an array of several pass-band sensors. Having shown that employment of sensor arrays is an economic incentive and practical, several multi-sensor fusion schemes are developed to facilitate their implementation. Another aspect of this dissertation is to develop means to deal with outliers in sensor measurements. As fault sensor data detection is an essential element of multi-sensor network implementation, which is used to improve system reliability and robustness, several sensor scheduling configurations are derived to identify and to remove outliers