The Natural Statistics of Audiovisual Speech

Humans, like other animals, are exposed to a continuous stream of signals, which are dynamic, multimodal, extended, and time varying in nature. This complex input space must be transduced and sampled by our sensory systems and transmitted to the brain where it can guide the selection of appropriate actions. To simplify this process, it's been suggested that the brain exploits statistical regularities in the stimulus space. Tests of this idea have largely been confined to unimodal signals and natural scenes. One important class of multisensory signals for which a quantitative input space characterization is unavailable is human speech. We do not understand what signals our brain has to actively piece together from an audiovisual speech stream to arrive at a percept versus what is already embedded in the signal structure of the stream itself. In essence, we do not have a clear understanding of the natural statistics of audiovisual speech. In the present study, we identified the following major statistical features of audiovisual speech. First, we observed robust correlations and close temporal correspondence between the area of the mouth opening and the acoustic envelope. Second, we found the strongest correlation between the area of the mouth opening and vocal tract resonances. Third, we observed that both area of the mouth opening and the voice envelope are temporally modulated in the 2–7 Hz frequency range. Finally, we show that the timing of mouth movements relative to the onset of the voice is consistently between 100 and 300 ms. We interpret these data in the context of recent neural theories of speech which suggest that speech communication is a reciprocally coupled, multisensory event, whereby the outputs of the signaler are matched to the neural processes of the receiver

Speech and Non-Speech Audio-Visual Illusions: A Developmental Study

It is well known that simultaneous presentation of incongruent audio and visual stimuli can lead to illusory percepts. Recent data suggest that distinct processes underlie non-specific intersensory speech as opposed to non-speech perception. However, the development of both speech and non-speech intersensory perception across childhood and adolescence remains poorly defined. Thirty-eight observers aged 5 to 19 were tested on the McGurk effect (an audio-visual illusion involving speech), the Illusory Flash effect and the Fusion effect (two audio-visual illusions not involving speech) to investigate the development of audio-visual interactions and contrast speech vs. non-speech developmental patterns. Whereas the strength of audio-visual speech illusions varied as a direct function of maturational level, performance on non-speech illusory tasks appeared to be homogeneous across all ages. These data support the existence of independent maturational processes underlying speech and non-speech audio-visual illusory effects

Value of prominent flow voids without cord edema in the detection of spinal arteriovenous fistulae

Purpose: To determine the prevalence of spinal dural arteriovenous fistulae (SDAVF) in patients presenting with prominent vascular flow voids on imaging without other imaging findings suggestive of SDAVF. Methods: We retrospectively identified patients from January 1, 2005 to March 1, 2012 who underwent spinal angiography for suspected SDAVF with prominent vascular flow voids on prior imaging. We excluded patients with other major spinal pathology or other imaging findings of SDAVF including cord hyperintensity, enhancement, or expansion. We calculated the proportion of patients with positive findings for SDAVF on angiography and evaluated the prevalence of SDAVF for this finding alone and in correlation with clinical findings. Results: 18 patients underwent spinal angiography for prominent flow voids on imaging without other spinal pathology or imaging findings of SDAVF. Three had a SDAVF detected on angiography. The prevalence of SDAVF in this population was low, only 17% (95% CI 6-39%). All of the patients with positive angiography findings had myelopathy, increasing the prevalence to 100% if the additional clinical finding of myelopathy was present. Conclusions: Prominent flow voids without other imaging findings suggestive of SDAVF is poorly predictive of the presence of a SDAVF, unless myelopathy is present clinically. © 2014 Alhilali et al

Semantic Object Prediction and Spatial Sound Super-Resolution with Binaural Sounds

Humans can robustly recognize and localize objects by integrating visual and auditory cues. While machines are able to do the same now with images, less work has been done with sounds. This work develops an approach for dense semantic labelling of sound-making objects, purely based on binaural sounds. We propose a novel sensor setup and record a new audio-visual dataset of street scenes with eight professional binaural microphones and a 360 degree camera. The co-existence of visual and audio cues is leveraged for supervision transfer. In particular, we employ a cross-modal distillation framework that consists of a vision `teacher' method and a sound `student' method -- the student method is trained to generate the same results as the teacher method. This way, the auditory system can be trained without using human annotations. We also propose two auxiliary tasks namely, a) a novel task on Spatial Sound Super-resolution to increase the spatial resolution of sounds, and b) dense depth prediction of the scene. We then formulate the three tasks into one end-to-end trainable multi-tasking network aiming to boost the overall performance. Experimental results on the dataset show that 1) our method achieves promising results for semantic prediction and the two auxiliary tasks; and 2) the three tasks are mutually beneficial -- training them together achieves the best performance and 3) the number and orientations of microphones are both important. The data and code will be released to facilitate the research in this new direction.Comment: Project page: https://www.trace.ethz.ch/publications/2020/sound_perception/index.htm

Judging Time-to-Passage of looming sounds: evidence for the use of distance-based information

Perceptual judgments are an essential mechanism for our everyday interaction with other moving agents or events. For instance, estimation of the time remaining before an object contacts or passes us is essential to act upon or to avoid that object. Previous studies have demonstrated that participants use different cues to estimate the time to contact or the time to passage of approaching visual stimuli. Despite the considerable number of studies on the judgment of approaching auditory stimuli, not much is known about the cues that guide listeners’ performance in an auditory Time-to-Passage (TTP) task. The present study evaluates how accurately participants judge approaching white-noise stimuli in a TTP task that included variable occlusion periods (portion of the presentation time where the stimulus is not audible). Results showed that participants were able to accurately estimate TTP and their performance, in general, was weakly affected by occlusion periods. Moreover, we looked into the psychoacoustic variables provided by the stimuli and analysed how binaural cues related with the performance obtained in the psychophysical task. The binaural temporal difference seems to be the psychoacoustic cue guiding participants’ performance for lower amounts of occlusion, while the binaural loudness difference seems to be the cue guiding performance for higher amounts of occlusion. These results allowed us to explain the perceptual strategies used by participants in a TTP task (maintaining accuracy by shifting the informative cue for TTP estimation), and to demonstrate that the psychoacoustic cue guiding listeners’ performance changes according to the occlusion period.This study was supported by: Bial FoundationGrant 143/14 (https://www.bial.com/en/bial_foundation.11/11th_symposium.219/ fellows_preliminary_results.235/fellows_ preliminary_results.a569.html); FCT PTDC/EEAELC/112137/2009 (https://www.fct.pt/apoios/projectos/consulta/vglobal_projecto?idProjecto=112137&idElemConcurso=3628); and COMPETE: POCI-01-0145-FEDER-007043 and FCT – Fundação para a Ciência e Tecnologia within the Project Scope: UID/CEC/00319/2013.info:eu-repo/semantics/publishedVersio

Sensory substitution information informs locomotor adjustments when walking through apertures

The study assessed the ability of the central nervous system (CNS) to use echoic information from sensory substitution devices (SSDs) to rotate the shoulders and safely pass through apertures of different width. Ten visually normal participants performed this task with full vision, or blindfolded using an SSD to obtain information regarding the width of an aperture created by two parallel panels. Two SSDs were tested. Participants passed through apertures of +0%, +18%, +35%, and +70% of measured body width. Kinematic indices recorded movement time, shoulder rotation, average walking velocity across the trial, peak walking velocities before crossing, after crossing and throughout a whole trial. Analyses showed participants used SSD information to regulate shoulder rotation, with greater rotation associated with narrower apertures. Rotations made using an SSD were greater compared to vision, movement times were longer, average walking velocity lower and peak velocities before crossing, after crossing and throughout the whole trial were smaller, suggesting greater caution. Collisions sometimes occurred using an SSD but not using vision, indicating that substituted information did not always result in accurate shoulder rotation judgements. No differences were found between the two SSDs. The data suggest that spatial information, provided by sensory substitution, allows the relative position of aperture panels to be internally represented, enabling the CNS to modify shoulder rotation according to aperture width. Increased buffer space indicated by greater rotations (up to approximately 35% for apertures of +18% of body width), suggests that spatial representations are not as accurate as offered by full vision

Cross-Modal Prediction in Speech Perception

Speech perception often benefits from vision of the speaker's lip movements when they are available. One potential mechanism underlying this reported gain in perception arising from audio-visual integration is on-line prediction. In this study we address whether the preceding speech context in a single modality can improve audiovisual processing and whether this improvement is based on on-line information-transfer across sensory modalities. In the experiments presented here, during each trial, a speech fragment (context) presented in a single sensory modality (voice or lips) was immediately continued by an audiovisual target fragment. Participants made speeded judgments about whether voice and lips were in agreement in the target fragment. The leading single sensory context and the subsequent audiovisual target fragment could be continuous in either one modality only, both (context in one modality continues into both modalities in the target fragment) or neither modalities (i.e., discontinuous). The results showed quicker audiovisual matching responses when context was continuous with the target within either the visual or auditory channel (Experiment 1). Critically, prior visual context also provided an advantage when it was cross-modally continuous (with the auditory channel in the target), but auditory to visual cross-modal continuity resulted in no advantage (Experiment 2). This suggests that visual speech information can provide an on-line benefit for processing the upcoming auditory input through the use of predictive mechanisms. We hypothesize that this benefit is expressed at an early level of speech analysis

Orange jasmine as a trap crop to control Diaphorina citri

[EN] Novel, suitable and sustainable alternative control tactics that have the potential to reduce migration of Diaphorina citri into commercial citrus orchards are essential to improve management of huanglongbing (HLB). In this study, the effect of orange jasmine (Murraya paniculata) as a border trap crop on psyllid settlement and dispersal was assessed in citrus orchards. Furthermore, volatile emission profiles and relative attractiveness of both orange jasmine and sweet orange (Citrus¿×¿aurantium L., syn. Citrus sinensis (L.) Osbeck) nursery flushes to D. citri were investigated. In newly established citrus orchards, the trap crop reduced the capture of psyllids in yellow sticky traps and the number of psyllids that settled on citrus trees compared to fallow mowed grass fields by 40% and 83%, respectively. Psyllids were attracted and killed by thiamethoxam-treated orange jasmine suggesting that the trap crop could act as a `sink¿ for D. citri. Additionally, the presence of the trap crop reduced HLB incidence by 43%. Olfactometer experiments showed that orange jasmine plays an attractive role on psyllid behavior and that this attractiveness may be associated with differences in the volatile profiles emitted by orange jasmine in comparison with sweet orange. Results indicated that insecticide-treated M. paniculata may act as a trap crop to attract and kill D. citri before they settled on the edges of citrus orchards, which significantly contributes to the reduction of HLB primary spread.This work was supported by Fund for Citrus Protection (Fundecitrus) and by Fundacao de Amparo a Pesquisa do Estado de Sao Paulo (FAPESP) (Proc. 2015/07011-3). We thank Moacir Celio Vizone, Felipe Marinho Martini and Joao Pedro Ancoma Lopes for technical support with experiments. Furthermore, we thank Cambuhy Agricola Ltda. and University of Araraquara (Uniara) for providing the areas in which the field experiments were performed. Second author received scholarship from National Council for Scientific and Technological Development (CNPq)/Brazil (Proc. 300153/2011-2).Tomaseto, AF.; Marques, RN.; Fereres, A.; Zanardi, OZ.; Volpe, HXL.; Alquézar-García, B.; Peña, L.... (2019). Orange jasmine as a trap crop to control Diaphorina citri. Scientific Reports. 9:1-11. https://doi.org/10.1038/s41598-019-38597-5S1119Bové, J. M. Huanglongbing: a destructive, newly-emerging, century-old disease of citrus. J Plant Pathol. 88, 7–37 (2006).Alvarez, S., Rohrig, E., Solís, D. & Thomas, M. H. Citrus greening disease (Huanglongbing) in Florida: economic impact, management and the potential for biological control. Agric. Res. 5, 109–118 (2016).Belasque, J. Jr. et al. Lessons from huanglongbing management in São Paulo state, Brazil. J. Plant Pathol. 92, 285–302 (2010).Boina, D. R., Meyer, W. L., Onagbola, E. O. & Stelinski, L. L. Quantifying dispersal of Diaphorina citri (Hemiptera: Psyllidae) by immunomarking and potential impact of unmanaged groves on commercial citrus management. Environ. Entomol. 38, 1250–8 (2009).Lewis-Rosenblum, H., Martini, X., Tiwari, S. & Stelinski, L. L. Seasonal movement patterns and long-range dispersal of Asian citrus psyllid in Florida citrus. J. Econ. Entomol. 108, 3–10 (2015).Hall, D. G. & Hentz, M. G. Seasonal flight activity by the Asian citrus psyllid in east central Florida. Entomol. Exp. Appl. 139, 75–85 (2011).Tomaseto, A. F., Krugner, R. & Lopes, J. R. S. Effect of plant barriers and citrus leaf age on dispersal of Diaphorina citri (Hemiptera: Liviidae). J. Appl. Entomol. 140, 91–102 (2016).Gottwald, T. R. Current epidemiological understanding of citrus huanglongbing. Annu. Rev. Phytopathol. 48, 119–139 (2010).Bassanezi, R. B. et al. Efficacy of area-wide inoculum reduction and vector control on temporal progress of huanglongbing in young sweet orange plantings. Plant Dis. 97, 789–796 (2013).Sétamou, M. & Bartels, D. W. Living on the edges: spatial niche occupation of Asian citrus psyllid, Diaphorina citri Kuwayama (Hemiptera: Liviidae), in citrus groves. PLoS One 10, 1–21 (2015).Gottwald, T., Irey, M., Gast, T. & Parnell, S. Spatio-temporal analysis of an HLB epidemic in Florida and implications for spread. In Proceedings of the 17 th Conference of International Organization of Citrus Virologists, IOCV, University of California, Riverside, CA, 84–97 (2010).Shelton, A. M. & Badenes-Perez, F. R. Concepts and applications of trap cropping in pest management. Annu. Rev. Entomol 51, 285–308 (2006).Hokkanen, H. M. T. Trap cropping in pest management. Annu. Rev. Entomol. 36, 119–138 (1991).Stern, V. M., Mueller, A., Sevacherian, V. & Way, M. Lygus bug control in cotton through alfalfa interplanting. Calif. Agric. 8–10 (1969).Godfrey, L. D. & Leigh, T. F. Alfalfa harvest strategy effect on lygus bug (Hemiptera: Miridae) and insect predator population density: Implications for use as trap crop in cotton. Environ. Entomol. 23, 1106–1118 (1994).Gonsalves, D. & Ferreira, S. Transgenic papaya: a case for managing risks of Papaya ringspot virus in Hawaii. Plant Heal. Prog. 1–6, https://doi.org/10.1094/PHP-2003-1113-03-RV (2003)Aubert, B. Trioza erytheae del Guercio and Diaphorina citri Kuwayama (Homoptera: Psylloidea), the two vectors of citrus greening disease: biological aspects and possible control strategies. Fruits 42, 149–162 (1987).Leong, S. C. T., Fatimah, A., Beattie, A., Heng, R. K. J. & King, W. S. Influence of host plant species and flush growth stage on the Asian citrus psyllid, Diaphorina citri Kuwayama. Am. J. Agric. Biol. Sci. 6, 536–543 (2011).Patt, J. M. & Sétamou, M. Responses of the Asian citrus psyllid to volatiles emitted by the flushing shoots of its rutaceous host plants. Environ. Entomol. 39, 618–24 (2010).Damsteegt, V. D. et al. Murraya paniculata and related species as potential hosts and inoculum reservoirs of ‘Candidatus Liberibacter asiaticus’, causal agent of huanglongbing. Plant Dis. 94, 528–533 (2010).Lopes, S. A. et al. Liberibacters associated with orange jasmine in Brazil: Incidence in urban areas and relatedness to citrus liberibacters. Plant Pathol. 59, 1044–1053 (2010).Cifuentes-Arenas, J. C. Huanglongbing e Diaphorina citri: Estudos das relações patógeno-vetor-hospedeiro. Ph.D. Thesis. Faculdade de Ciências Agrárias e Veterinárias/Universidade Estadual Paulista (UNESP), Jaboticabal, SP, Brazil. 1–133 (2017).Morilla, G. et al. Pepper (Capsicum annuum) is a dead-end host for Tomato yellow leaf curl virus. Phytopathology 95, 1089–1097 (2005).Midega, C. A. O., Pittchar, J. O., Pickett, J. A., Hailu, G. W. & Khan, Z. R. A climate-adapted push-pull system effectively controls fall armyworm, Spodoptera frugiperda (J. E. Smith), in maize in East Africa. Crop Prot. 105, 10–15 (2018).Miranda, M. P. et al. Processed kaolin affects the probing and settling behavior of Diaphorina citri (Hemiptera: Liviidae). Pest Manag. Sci. 74, 1964–1972 (2018).Kobori, Y., Nakata, T., Ohto, Y. & Takasu, F. Dispersal of adult Asian citrus psyllid, Diaphorina citri Kuwayama (Homoptera: Psyllidae), the vector of citrus greening disease, in artificial release experiments. Appl. Entomol. Zool. 46, 27–30 (2011).Sétamou, M. et al. Diurnal patterns of flight activity and effects of light on host finding behavior of the Asian citrus psyllid. J. Insect Behav. 25, 264–276 (2012).Wenninger, E. J., Stelinski, L. L. & Hall, D. G. Roles of olfactory cues, visual cues, and mating status in orientation of Diaphorina citri Kuwayama (Hemiptera: Psyllidae) to four different host plants. Environ. Entomol. 38, 225–234 (2009).Miranda, M. P., Dos Santos, F. L., Felippe, M. R., Moreno, A. & Fereres, A. Effect of UV-blocking plastic films on take-off and host plant finding ability of Diaphorina citri (Hemiptera: Liviidae). J. Econ. Entomol. 108, 245–251 (2015).Visser, J. H. Host odor perception in phytophagous insects. Annu. Rev. Entomol. 31, 121–144 (1986).Robbins, P. S., Alessandro, R. T., Stelinski, L. L. & Lapointe, S. L. Volatile profiles of young leaves of Rutaceae spp. varying in susceptibility to the Asian citrus psyllid (Hemiptera: Psyllidae). Florida Entomol. 95, 774–776 (2012).Fancelli, M. et al. Attractiveness of host plant volatile extracts to the Asian citrus psyllid, Diaphorina citri, is reduced by terpenoids from the non-host cashew. J. Chem. Ecol. 44, 397–405 (2018).Alquézar, B. et al. β-caryophyllene emitted from a transgenic Arabidopsis or chemical dispenser repels Diaphorina citri, vector of Candidatus Liberibacters. Sci. Rep. 7, 5639 (2017).Jones, R. A. C. Effects of cereal borders, admixture with cereals and plant density on the spread of bean yellow mosaic potyvirus into narrow‐leafed lupins (Lupinus angustifolius). Ann. Appl. Biol. 122, 501–518 (1993).Beloti, V. H., Alves, G. R., Coletta-Filho, H. D. & Yamamoto, P. T. The Asian citrus psyllid host Murraya koenigii is immune to citrus huanglongbing pathogen ‘Candidatus Liberibacter asiaticus’. Phytopathology 108, 1089–1094 (2018).Walter, A. J., Duan, Y. & Hall, D. G. Titers of ‘Ca. Liberibacter asiaticus’ in Murraya paniculata and Murraya-reared Diaphorina citri are much lower than in Citrus and Citrus-reared psyllids. HortScience 47, 1449–1452 (2012).Walter, A. J., Hall, D. G. & Duan, Y. P. Low incidence of ‘Candidatus Liberibacter asiaticus’ in Murraya paniculata and associated Diaphorina citri. Plant Dis. 96, 827–832 (2012).Ammar, E.-D. D., Ramos, J. E., Hall, D. G., Dawson, W. O. & Shatters, R. G. Acquisition, replication and inoculation of Candidatus Liberibacter asiaticus following various acquisition periods on huanglongbing-infected citrus by nymphs and adults of the Asian citrus psyllid. PLoS One 11, e0159594 (2016).Inoue, H. et al. Enhanced proliferation and efficient transmission of Candidatus Liberibacter asiaticus by adult Diaphorina citri after acquisition feeding in the nymphal stage. Ann. Appl. Biol. 155, 29–36 (2009).Pelz-Stelinski, K. S., Brlansky, R. H., Ebert, T. A. & Rogers, M. E. Transmission parameters for Candidatus Liberibacter asiaticus by Asian citrus psyllid (Hemiptera: Psyllidae). J. Econ. Entomol. 103, 1531–1541 (2010).Canale, M. C. et al. Latency and persistence of ‘Candidatus Liberibacter asiaticus’ in its psyllid vector, Diaphorina citri (Hemiptera: Liviidae). Phytopathology 107, 264–272 (2017).Li, W., Hartung, J. S. & Levy, L. Quantitative real-time PCR for detection and identification of Candidatus Liberibacter species associated with citrus huanglongbing. J. Microbiol. Methods 66, 104–115 (2006).Nakata, T. Effectiveness of micronized fluorescent powder for marking citrus psyllid. Diaphorina citri. Appl. Entomol. Zool. 43, 33–36 (2008).Tomaseto, A. F. et al. Environmental conditions for Diaphorina citri Kuwayama (Hemiptera: Liviidae) take-off. J. Appl. Entomol. 142, 104–113 (2018).Paris, T. M., Croxton, S. D., Stansly, P. A. & Allan, S. A. Temporal response and attraction of Diaphorina citri to visual stimuli. Entomol. Exp. Appl. 155, 137–147 (2015).Zanardi, O. Z. et al. Putative sex pheromone of the Asian citrus psyllid, Diaphorina citri, breaks down into an attractant. Sci. Rep. 8, 455 (2018).Metsalu, T. & Vilo, J. ClustVis: A web tool for visualizing clustering of multivariate data using Principal Component Analysis and heatmap. Nucleic Acids Res. 43, W566–W570 (2015).Fournier, D. A. et al. AD Model Builder: using automatic differentiation for statistical inference of highly parameterized complex nonlinear models. Optim. Methods Softw. 27, 233–249 (2012).Bates, D., Mächler, M., Bolker, B. & Walker, S. Fitting linear mixed-effects models using lme4. J. Stat. Softw. 67, 1–48 (2015).Nelder, J. A. & Wedderburn, R. W. M. Generalized linear models. J. R. Stat. Soc. 135, 370–384 (1972).Demétrio, C. G. B., Hinde, J. & Moral, R. A. In Ecological Modelling Applied to Entomology (eds Ferreira, C. P. & Godoy, W. A. C.) 219–259 (Springer, 2014).Lenth, R. V. Least-Squares Means: the R package lsmeans. J. Stat. Softw. 69, (2016).R Core Team R: A language and environment for statistical computing. 2015. R Foundation for Statistical Computing, Vienna, Austria (2015). Available at, http://www.r-project.org/ . (Accessed: 20th July 2017)