Audio-Visual Learning for Scene Understanding

Multimodal deep learning aims at combining the complementary information of different modalities. Among all modalities, audio and video are the predominant ones that humans use to explore the world. In this thesis, we decided to focus our study on audio-visual deep learning to mimic with our networks how humans perceive the world. Our research includes images, audio signals and acoustic images. The latter provide spatial audio information and are obtained from a planar array of microphones combining their raw audios with the beamforming algorithm. They better mimic human auditory systems, which cannot be replicated using just one microphone, not able alone to give spatial sound cues. However, as microphones arrays are not so widespread, we also study how to handle the missing spatialized audio modality at test time. As a solution, we propose to distill acoustic images content to audio features during the training in order to handle their absence at test time. This is done for supervised audio classification using the generalized distillation framework, which we also extend for self-supervised learning. Next, we devise a method for reconstructing acoustic images given a single microphone and an RGB frame. Therefore, in case we just dispose of a standard video, we are able to synthesize spatial audio, which is useful for many audio-visual tasks, including sound localization. Lastly, as another example of restoring one modality from available ones, we inpaint degraded images providing audio features, to reconstruct the missing region not only to be visually plausible but also semantically consistent with the related sound. This includes also cross-modal generation, in the limit case of completely missing or hidden visual modality: our method naturally deals with it, being able to generate images from sound. In summary we show how audio can help visual learning and vice versa, by transferring knowledge between the two modalities at training time, in order to distill, reconstruct, or restore the missing modality at test time

Effects of ventilator settings, nebulizer and exhalation port position on albuterol delivery during non-invasive ventilation: an in-vitro study.

BACKGROUND:Few studies have investigated the factors affecting aerosol delivery during non-invasive ventilation (NIV). Our aim was to investigate, using a bench-top model, the effect of different ventilator settings and positions of the exhalation port and nebulizer on the amount of albuterol delivered to a lung simulator. METHODS: A lung model simulating spontaneous breathing was connected to a single-limb NIV ventilator, set in bi-level positive airway pressure (BIPAP) with inspiratory/expiratory pressures of 10/5, 15/10, 15/5, and 20/10 cmH2O, or continuous positive airway pressure (CPAP) of 5 and 10 cmH2O. Three delivery circuits were tested: a vented mask with the nebulizer directly connected to the mask, and an unvented mask with a leak port placed before and after the nebulizer. Albuterol was collected on a filter placed after the mask and then the delivered amount was measured with infrared spectrophotometry. RESULTS: Albuterol delivery during NIV varied between 6.7\u2009\ub1\u20090.4% to 37.0\u2009\ub1\u20094.3% of the nominal dose. The amount delivered in CPAP and BIPAP modes was similar (22.1\u2009\ub1\u200910.1 vs. 24.0\u2009\ub1\u200910.0%, p\u2009=\u20090.070). CPAP level did not affect delivery (p\u2009=\u20090.056); in BIPAP with 15/5 cmH2O pressure the delivery was higher compared to 10/5 cmH2O (p\u2009=\u20090.033) and 20/10 cmH2O (p\u2009=\u20090.014). Leak port position had a major effect on delivery in both CPAP and BIPAP, the best performances were obtained with the unvented mask, and the nebulizer placed between the leak port and the mask (p\u2009<\u20090.001). CONCLUSIONS: In this model, albuterol delivery was marginally affected by ventilatory settings in NIV, while position of the leak port had a major effect. Nebulizers should be placed between an unvented mask and the leak port in order to maximize aerosol delivery

Antibacterial activity of standard and N-doped titanium dioxide-coated endotracheal tubes: an in vitro study

Objective: The aim of this study was to assess the antibacterial activity against Staphylococcus aureus and Pseudomonas aeruginosa of two nanoparticle endotracheal tube coatings with visible light-induced photocatalysis. Methods: Two types of titanium dioxide nanoparticles were tested: standard anatase (TiO2) and N-doped TiO2 (N-TiO2). Nanoparticles were placed on the internal surface of a segment of commercial endotracheal tubes, which were loaded on a cellulose acetate filter; control endotracheal tubes were left without a nanoparticle coating. A bacterial inoculum of 150 colony forming units was placed in the endotracheal tubes and then exposed to a fluorescent light source (3700 lux, 300-700 nm wavelength) for 5, 10, 20, 40, 60 and 80 minutes. Colony forming units were counted after 24 hours of incubation at 37\ub0C. Bacterial inactivation was calculated as the percentage reduction of bacterial growth compared to endotracheal tubes not exposed to light. Results: In the absence of light, no relevant antibacterial activity was shown against neither strain. For P. aeruginosa, both coatings had a higher bacterial inactivation than controls at any time point (p < 0.001), and no difference was observed between TiO2 and N-TiO2. For S. aureus, inactivation was higher than for controls starting at 5 minutes for N-TiO2 (p = 0.018) and 10 minutes for TiO2 (p = 0.014); inactivation with N-TiO2 was higher than that with TiO2 at 20 minutes (p < 0.001), 40 minutes (p < 0.001) and 60 minutes (p < 0.001). Conclusions: Nanosized commercial and N-doped TiO2 inhibit bacterial growth under visible fluorescent light. N-TiO2 has higher antibacterial activity against S. aureus compared to TiO2

Self-adaptive robot training of stroke survivors for continuous tracking movements

<p>Abstract</p> <p>Background</p> <p>Although robot therapy is progressively becoming an accepted method of treatment for stroke survivors, few studies have investigated how to adapt the robot/subject interaction forces in an automatic way. The paper is a feasibility study of a novel self-adaptive robot controller to be applied with continuous tracking movements.</p> <p>Methods</p> <p>The haptic robot Braccio di Ferro is used, in relation with a tracking task. The proposed control architecture is based on three main modules: 1) a force field generator that combines a non linear attractive field and a viscous field; 2) a performance evaluation module; 3) an adaptive controller. The first module operates in a continuous time fashion; the other two modules operate in an intermittent way and are triggered at the end of the current block of trials. The controller progressively decreases the gain of the force field, within a session, but operates in a non monotonic way between sessions: it remembers the minimum gain achieved in a session and propagates it to the next one, which starts with a block whose gain is greater than the previous one. The initial assistance gains are chosen according to a minimal assistance strategy. The scheme can also be applied with closed eyes in order to enhance the role of proprioception in learning and control.</p> <p>Results</p> <p>The preliminary results with a small group of patients (10 chronic hemiplegic subjects) show that the scheme is robust and promotes a statistically significant improvement in performance indicators as well as a recalibration of the visual and proprioceptive channels. The results confirm that the minimally assistive, self-adaptive strategy is well tolerated by severely impaired subjects and is beneficial also for less severe patients.</p> <p>Conclusions</p> <p>The experiments provide detailed information about the stability and robustness of the adaptive controller of robot assistance that could be quite relevant for the design of future large scale controlled clinical trials. Moreover, the study suggests that including continuous movement in the repertoire of training is acceptable also by rather severely impaired subjects and confirms the stabilizing effect of alternating vision/no vision trials already found in previous studies.</p

UNet and MobileNet CNN-based model observers for CT protocol optimization: comparative performance evaluation by means of phantom CT images

Purpose: The aim of this work is the development and characterization of a model observer (MO) based on convolutional neural networks (CNNs), trained to mimic human observers in image evaluation in terms of detection and localization of low-contrast objects in CT scans acquired on a reference phantom. The final goal is automatic image quality evaluation and CT protocol optimization to fulfill the ALARA principle. Approach: Preliminary work was carried out to collect localization confidence ratings of human observers for signal presence/absence from a dataset of 30,000 CT images acquired on a PolyMethyl MethAcrylate phantom containing inserts filled with iodinated contrast media at different concentrations. The collected data were used to generate the labels for the training of the artificial neural networks. We developed and compared two CNN architectures based respectively on Unet and MobileNetV2, specifically adapted to achieve the double tasks of classification and localization. The CNN evaluation was performed by computing the area under localization-ROC curve (LAUC) and accuracy metrics on the test dataset. Results: The mean of absolute percentage error between the LAUC of the human observer and MO was found to be below 5% for the most significative test data subsets. An elevated inter-rater agreement was achieved in terms of S-statistics and other common statistical indices. Conclusions: Very good agreement was measured between the human observer and MO, as well as between the performance of the two algorithms. Therefore, this work is highly supportive of the feasibility of employing CNN-MO combined with a specifically designed phantom for CT protocol optimization programs

Adaptive robot training for the treatment of incoordination in Multiple Sclerosis

<p>Abstract</p> <p>Background</p> <p>Cerebellar symptoms are extremely disabling and are common in Multiple Sclerosis (MS) subjects. In this feasibility study, we developed and tested a robot therapy protocol, aimed at the rehabilitation of incoordination in MS subjects.</p> <p>Methods</p> <p>Eight subjects with clinically defined MS performed planar reaching movements while grasping the handle of a robotic manipulandum, which generated forces that either reduced (error-reducing, ER) or enhanced (error-enhancing, EE) the curvature of their movements, assessed at the beginning of each session. The protocol was designed to adapt to the individual subjects' impairments, as well as to improvements between sessions (if any). Each subject went through a total of eight training sessions. To compare the effect of the two variants of the training protocol (ER and EE), we used a cross-over design consisting of two blocks of sessions (four ER and four EE; 2 sessions/week), separated by a 2-weeks rest period. The order of application of ER and EE exercises was randomized across subjects. The primary outcome measure was the modification of the Nine Hole Peg Test (NHPT) score. Other clinical scales and movement kinematics were taken as secondary outcomes.</p> <p>Results</p> <p>Most subjects revealed a preserved ability to adapt to the robot-generated forces. No significant differences were observed in EE and ER training. However over sessions, subjects exhibited an average 24% decrease in their NHPT score. The other clinical scales showed small improvements for at least some of the subjects. After training, movements became smoother, and their curvature decreased significantly over sessions.</p> <p>Conclusions</p> <p>The results point to an improved coordination over sessions and suggest a potential benefit of a short-term, customized, and adaptive robot therapy for MS subjects.</p

A retrospective multicentric observational study of trastuzumab emtansine in HER2 positive metastatic breast cancer: A real-world experience

We addressed trastuzumab emtansine (T-DM1) efficacy in HER2+ metastatic breast cancer patients treated in real-world practice, and its activity in pertuzumab-pretreated patients. We conducted a retrospective, observational study involving 23 cancer centres, and 250 patients. Survival data were analyzed by Kaplan Meier curves and log rank test. Factors testing significant in univariate analysis were tested in multivariate models. Median follow-up was 15 months and median T-DM1 treatment-length 4 months. Response rate was 41.6%, clinical benefit 60.9%. Median progression-free and median overall survival were 6 and 20 months, respectively. Overall, no differences emerged by pertuzumab pretreatment, with median progression-free and median overall survival of 4 and 17 months in pertuzumab-pretreated (p=0.13), and 6 and 22 months in pertuzumab-na\uc3\uafve patients (p=0.27). Patients who received second-line T-DM1 had median progression-free and median overall survival of 3 and 12 months (p=0.0001) if pertuzumab-pretreated, and 8 and 26 months if pertuzumab-na\uc3\uafve (p=0.06). In contrast, in third-line and beyond, median progression-free and median overall survival were 16 and 18 months in pertuzumab-pretreated (p=0.05) and 6 and 17 months in pertuzumab-na\uc3\uafve patients (p=0.30). In multivariate analysis, lower ECOG performance status was associated with progression-free survival benefit (p &lt; 0.0001), while overall survival was positively affected by lower ECOG PS (p &lt; 0.0001), absence of brain metastases (p 0.05), and clinical benefit (p &lt; 0.0001). Our results are comparable with those from randomized trials. Further studies are warranted to confirm and interpret our data on apparently lower T-DM1 efficacy when given as second-line treatment after pertuzumab, and on the optimal sequence order

Unsupervised Synthetic Acoustic Image Generation for Audio-Visual Scene Understanding

Acoustic images are an emergent data modality for multimodal scene understanding. Such images have the peculiarity of distinguishing the spectral signature of the sound coming from different directions in space, thus providing a richer information as compared to that derived from single or binaural microphones. However, acoustic images are typically generated by cumbersome and costly microphone arrays which are not as widespread as ordinary microphones. This paper shows that it is still possible to generate acoustic images from off-the-shelf cameras equipped with only a single microphone and how they can be exploited for audio-visual scene understanding. We propose three architectures inspired by Variational Autoencoder, U-Net and adversarial models, and we assess their advantages and drawbacks. Such models are trained to generate spatialized audio by conditioning them to the associated video sequence and its corresponding monaural audio track. Our models are trained using the data collected by a microphone array as ground truth. Thus they learn to mimic the output of an array of microphones in the very same conditions. We assess the quality of the generated acoustic images considering standard generation metrics and different downstream tasks (classification, cross-modal retrieval and sound localization). We also evaluate our proposed models by considering multimodal datasets containing acoustic images, as well as datasets containing just monaural audio signals and RGB video frames. In all of the addressed downstream tasks we obtain notable performances using the generated acoustic data, when compared to the state of the art and to the results obtained using real acoustic images as input