Chinese Tones: Can You Listen With Your Eyes?:The Influence of Visual Information on Auditory Perception of Chinese Tones

CHINESE TONES: CAN YOU LISTEN WITH YOUR EYES? The Influence of Visual Information on Auditory Perception of Chinese Tones YUEQIAO HAN Summary Considering the fact that more than half of the languages spoken in the world (60%-70%) are so-called tone languages (Yip, 2002), and tone is notoriously difficult to learn for westerners, this dissertation focused on tone perception in Mandarin Chinese by tone-naïve speakers. Moreover, it has been shown that speech perception is more than just an auditory phenomenon, especially in situations when the speaker’s face is visible. Therefore, the aim of this dissertation is to also study the value of visual information (over and above that of acoustic information) in Mandarin tone perception for tone-naïve perceivers, in combination with other contextual (such as speaking style) and individual factors (such as musical background). Consequently, this dissertation assesses the relative strength of acoustic and visual information in tone perception and tone classification. In the first two empirical and exploratory studies in Chapter 2 and 3 , we set out to investigate to what extent tone-naïve perceivers are able to identify Mandarin Chinese tones in isolated words, and whether or not they can benefit from (seeing) the speakers’ face, and what the contribution is of a hyperarticulated speaking style, and/or their own musical experience. Respectively, in Chapter 2 we investigated the effect of visual cues (comparing audio-only with audio-visual presentations) and speaking style (comparing a natural speaking style with a teaching speaking style) on the perception of Mandarin tones by tone-naïve listeners, looking both at the relative strength of these two factors and their possible interactions; Chapter 3 was concerned with the effects of musicality of the participants (combined with modality) on Mandarin tone perception. In both of these studies, a Mandarin Chinese tone identification experiment was conducted: native speakers of a non-tonal language were asked to distinguish Mandarin Chinese tones based on audio (-only) or video (audio-visual) materials. In order to include variations, the experimental stimuli were recorded using four different speakers in imagined natural and teaching speaking scenarios. The proportion of correct responses (and average reaction times) of the participants were reported. The tone identification experiment presented in Chapter 2 showed that the video conditions (audio-visual natural and audio-visual teaching) resulted in an overall higher accuracy in tone perception than the auditory-only conditions (audio-only natural and audio-only teaching), but no better performance was observed in the audio-visual conditions in terms of reaction time, compared to the auditory-only conditions. Teaching style turned out to make no difference on the speed or accuracy of Mandarin tone perception (as compared to a natural speaking style). Further on, we presented the same experimental materials and procedure in Chapter 3 , but now with musicians and non-musicians as participants. The Goldsmith Musical Sophistication Index (Gold-MSI) was used to assess the musical aptitude of the participants. The data showed that overall, musicians outperformed non-musicians in the tone identification task in both auditory-visual and auditory-only conditions. Both groups identified tones more accurately in the auditory-visual conditions than in the auditory-only conditions. These results provided further evidence for the view that the availability of visual cues along with auditory information is useful for people who have no knowledge of Mandarin Chinese tones when they need to learn to identify these tones. Out of all the musical skills measured by Gold-MSI, the amount of musical training was the only predictor that had an impact on the accuracy of Mandarin tone perception. These findings suggest that learning to perceive Mandarin tones benefits from musical expertise, and visual information can facilitate Mandarin tone identification, but mainly for tone-naïve non-musicians. In addition, performance differed by tone: musicality improves accuracy for every tone; some tones are easier to identify than others: in particular, the identification of tone 3 (a low-falling-rising) proved to be the easiest, while tone 4 (a high-falling tone) was the most difficult to identify for all participants. The results of the first two experiments presented in chapters 2 and 3 showed that adding visual cues to clear auditory information facilitated the tone identification for tone-naïve perceivers (there is a significantly higher accuracy in audio-visual condition(s) than in auditory-only condition(s)). This visual facilitation was unaffected by the presence of (hyperarticulated) speaking style or the musical skill of the participants. Moreover, variations in speakers and tones had effects on the accurate identification of Mandarin tones by tone-naïve perceivers. In Chapter 4 , we compared the relative contribution of auditory and visual information during Mandarin Chinese tone perception. More specifically, we aimed to answer two questions: firstly, whether or not there is audio-visual integration at the tone level (i.e., we explored perceptual fusion between auditory and visual information). Secondly, we studied how visual information affects tone perception for native speakers and non-native (tone-naïve) speakers. To do this, we constructed various tone combinations of congruent (e.g., an auditory tone 1 paired with a visual tone 1, written as AxVx) and incongruent (e.g., an auditory tone 1 paired with a visual tone 2, written as AxVy) auditory-visual materials and presented them to native speakers of Mandarin Chinese and speakers of tone-naïve languages. Accuracy, defined as the percentage correct identification of a tone based on its auditory realization, was reported. When comparing the relative contribution of auditory and visual information during Mandarin Chinese tone perception with congruent and incongruent auditory and visual Chinese material for native speakers of Chinese and non-tonal languages, we found that visual information did not significantly contribute to the tone identification for native speakers of Mandarin Chinese. When there is a discrepancy between visual cues and acoustic information, (native and tone-naïve) participants tend to rely more on the auditory input than on the visual cues. Unlike the native speakers of Mandarin Chinese, tone-naïve participants were significantly influenced by the visual information during their auditory-visual integration, and they identified tones more accurately in congruent stimuli than in incongruent stimuli. In line with our previous work, the tone confusion matrix showed that tone identification varies with individual tones, with tone 3 (the low-dipping tone) being the easiest one to identify, whereas tone 4 (the high-falling tone) was the most difficult one. The results did not show evidence for auditory-visual integration among native participants, while visual information was helpful for tone-naïve participants. However, even for this group, visual information only marginally increased the accuracy in the tone identification task, and this increase depended on the tone in question. Chapter 5 is another chapter that zooms in on the relative strength of auditory and visual information for tone-naïve perceivers, but from the aspect of tone classification. In this chapter, we studied the acoustic and visual features of the tones produced by native speakers of Mandarin Chinese. Computational models based on acoustic features, visual features and acoustic-visual features were constructed to automatically classify Mandarin tones. Moreover, this study examined what perceivers pick up (perception) from what a speaker does (production, facial expression) by studying both production and perception. To be more specific, this chapter set out to answer: (1) which acoustic and visual features of tones produced by native speakers could be used to automatically classify Mandarin tones. Furthermore, (2) whether or not the features used in tone production are similar to or different from the ones that have cue value for tone-naïve perceivers when they categorize tones; and (3) whether and how visual information (i.e., facial expression and facial pose) contributes to the classification of Mandarin tones over and above the information provided by the acoustic signal. To address these questions, the stimuli that had been recorded (and described in chapter 2) and the response data that had been collected (and reported on in chapter 3) were used. Basic acoustic and visual features were extracted. Based on them, we used Random Forest classification to identify the most important acoustic and visual features for classifying the tones. The classifiers were trained on produced tone classification (given a set of auditory and visual features, predict the produced tone) and on perceived/responded tone classification (given a set of features, predict the corresponding tone as identified by the participant). The results showed that acoustic features outperformed visual features for tone classification, both for the classification of the produced and the perceived tone. However, tone-naïve perceivers did revert to the use of visual information in certain cases (when they gave wrong responses). So, visual information does not seem to play a significant role in native speakers’ tone production, but tone-naïve perceivers do sometimes consider visual information in their tone identification. These findings provided additional evidence that auditory information is more important than visual information in Mandarin tone perception and tone classification. Notably, visual features contributed to the participants’ erroneous performance. This suggests that visual information actually misled tone-naïve perceivers in their task of tone identification. To some extent, this is consistent with our claim that visual cues do influence tone perception. In addition, the ranking of the auditory features and visual features in tone perception showed that the factor perceiver (i.e., the participant) was responsible for the largest amount of variance explained in the responses by our tone-naïve participants, indicating the importance of individual differences in tone perception. To sum up, perceivers who do not have tone in their language background tend to make use of visual cues from the speakers’ faces for their perception of unknown tones (Mandarin Chinese in this dissertation), in addition to the auditory information they clearly also use. However, auditory cues are still the primary source they rely on. There is a consistent finding across the studies that the variations between tones, speakers and participants have an effect on the accuracy of tone identification for tone-naïve speaker

Investigations of Crystallization of Molecular Sieves and their Behaviour under High Pressures

The behaviour of completely siliceous zeolite ZSM-5 under high pressure up to 16 GPa were investigated by in situ Raman spectroscopy, Infrared spectroscopy and X-ray diffraction with synchrotron radiation using diamond anvil cell. Pressure-induced amorphization was observed in both as-made and calcined siliceous ZSM-5 which transform to a low-density amorphous silica first and then to a high-density amorphous silica. However, transition pressures and reversibility were different for the as-made and calcined siliceous ZSM-5. It was found that the occluded template molecules are the major factor for these differences. The crystallization mechanism of AIPO4-I6 molecular sieve was preliminarily investigated under dry gel conversion method by X-ray diffraction and Solid-State NMR spectroscopy. The AIPO4-I6 and a crystalline by-product which has similar structure were formed via a semi-crystalline layered A1PO intermediate. The by-product could be transformed to amorphous materials with different structures by calcination and washing with distilled water

ARF-OCE for mapping mechanical properties of ocular and vascular tissues

Elastography is an imaging modality for clinical diagnosis based on the tissue stiffness. Benefiting from the high resolution, three-dimensional, and noninvasive nature of optical coherence tomography (OCT), optical coherence elastography (OCE) has the ability to determine elastic properties with a resolution of ~10 μm in 3D. Typical OCE imaging includes excitation for inducing mechanical vibrations, measurement of the sample response using OCT, and estimation of elastic parameters. Acoustic radiation force (ARF) generated by an ultrasonic transducer can noninvasively excite internal tissues without contact; thus, ARF-OCE is suitable for measuring the mechanical properties in deeper tissues. For assessment of the elastic properties of tissues using ARF-OCE, the shear wave velocity, resonant frequency, and vibrational displacement can be measured. Shear wave velocity measurements can be conveniently used for quantitative calculation of the elastic modulus.1-3 The resonant frequency of a tissue has a squared relationship with the Young\u27s modulus, and thus can quantify the elasticity.4 Vibrational displacement can be compared directly when the same pressure is applied to different samples.5 Several diseases are accompanied by and result in the changes in composition and local geometry of tissues. Keratoconus, which causes vision distortions and blurriness, will change the geometry of the cornea. The development of presbyopia is generally caused by the loss of elasticity in the lens. The composition and biomechanical properties of vessels will usually be altered when atherosclerosis occurs. The ARF-OCE technology provides a new opportunity for the early diagnosis of ocular and vascular diseases. Based on the shear wave measurements, our system can be used to quantify the elastic modulus of the cornea and the crystalline lens. By comparing the vibrational displacement, we have detected the differences between normal and cross-linked cornea.6 Recently we developed a miniature probe for mapping the mechanical properties of vascular lesions using ARF-OCE. It has the ability to detect the a vulnerable plaque due to its higher stiffness.7 Because of the noninvasive nature, ARF-OCE has the potential to perform in vivo imaging of deep tissues for the early diagnosis of ocular and vascular diseases. 1. Zhu, J., Qu, Y., Ma, T., Li, R., Du, Y., Huang, S., Shung, K.K., Zhou, Q. and Chen, Z., 2015. Imaging and characterizing shear wave and shear modulus under orthogonal acoustic radiation force excitation using OCT Doppler variance method. Optics letters, 40(9): 2099-2102. 2. Zhu, J., Qi, L., Miao, Y., Ma, T., Dai, C., Qu, Y., He, Y., Gao, Y., Zhou, Q. and Chen, Z., 2016. 3D mapping of elastic modulus using shear wave optical micro-elastography. Scientific reports, 6: 35499. 3. Xu, X., Zhu, J. and Chen, Z., 2016. Dynamic and quantitative assessment of blood coagulation using optical coherence elastography. Scientific reports, 6: 24294. 4. Qi, W., Li, R., Ma, T., Li, J., Kirk Shung, K., Zhou, Q. and Chen, Z., 2013. Resonant acoustic radiation force optical coherence elastography. Applied physics letters, 103(10): 103704. 5. Qi, W., Li, R., Ma, T., Kirk Shung, K., Zhou, Q. and Chen, Z., 2014. Confocal acoustic radiation force optical coherence elastography using a ring ultrasonic transducer. Applied physics letters, 104(12): 123702. 6. Qu, Y., Ma, T., He, Y., Zhu, J., Dai, C., Yu, M., Huang, S., Lu, F., Shung, K.K., Zhou, Q. and Chen, Z., 2016. Acoustic radiation force optical coherence elastography of corneal tissue. IEEE Journal of Selected Topics in Quantum Electronics, 22(3): 288-294. Qu, Y., Ma, T., He, Y., Yu, M., Zhu, J., Miao, Y., Dai, C., Patel, P., Shung, K.K., Zhou, Q. and Chen, Z., 2017. Miniature probe for mapping mechanical properties of vascular lesions using acoustic radiation force optical coherence elastography. Scientific Reports, 7: 473

By-example Synthesis of Compactly Stored Textures

AbstractThis paper proposes a novel by-example texture synthesis algorithm. Unlike most present algorithms whose synthesized textures of different sizes must be stored in memory until rendering, our results are compactly stored as paths of the generative graphs. This is achieved by adapting the graph-based synthesis framework to be suitable for arbitrary types of textures. Two synthesis processes are carried out successively to grow the texture strip by strip to the desired horizontal and vertical dimensions, each of which is cast as a constrained shortest path problem that can be solved efficiently by our proposed algorithm. The strips are formed by consecutive cuts which are precomputed in the preprocessing step, and are selected with a redesigned mechanism accounting for both global and local matching errors. The variances make the graph-based synthesis framework also applicable to a wide variety of textures besides architectural textures, and the synthesis quality for them is greatly improved

Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Review

Educational technology innovations leveraging large language models (LLMs) have shown the potential to automate the laborious process of generating and analysing textual content. While various innovations have been developed to automate a range of educational tasks (e.g., question generation, feedback provision, and essay grading), there are concerns regarding the practicality and ethicality of these innovations. Such concerns may hinder future research and the adoption of LLMs-based innovations in authentic educational contexts. To address this, we conducted a systematic scoping review of 118 peer-reviewed papers published since 2017 to pinpoint the current state of research on using LLMs to automate and support educational tasks. The findings revealed 53 use cases for LLMs in automating education tasks, categorised into nine main categories: profiling/labelling, detection, grading, teaching support, prediction, knowledge representation, feedback, content generation, and recommendation. Additionally, we also identified several practical and ethical challenges, including low technological readiness, lack of replicability and transparency, and insufficient privacy and beneficence considerations. The findings were summarised into three recommendations for future studies, including updating existing innovations with state-of-the-art models (e.g., GPT-3/4), embracing the initiative of open-sourcing models/systems, and adopting a human-centred approach throughout the developmental process. As the intersection of AI and education is continuously evolving, the findings of this study can serve as an essential reference point for researchers, allowing them to leverage the strengths, learn from the limitations, and uncover potential research opportunities enabled by ChatGPT and other generative AI models

A New Three-Dimensional Indoor Positioning Mechanism Based on Wireless LAN

The researches on two-dimensional indoor positioning based on wireless LAN and the location fingerprint methods have become mature, but in the actual indoor positioning situation, users are also concerned about the height where they stand. Due to the expansion of the range of three-dimensional indoor positioning, more features must be needed to describe the location fingerprint. Directly using a machine learning algorithm will result in the reduced ability of classification. To solve this problem, in this paper, a “divide and conquer” strategy is adopted; that is, first through k-medoids algorithm the three-dimensional location space is clustered into a number of service areas, and then a multicategory SVM with less features is created for each service area for further positioning. Our experiment shows that the error distance resolution of the approach with k-medoids algorithm and multicategory SVM is higher than that of the approach only with SVM, and the former can effectively decrease the “crazy prediction.

Separating and characterizing functional alkane degraders from crude-oil-contaminated sites via magnetic nanoparticle-mediated isolation

Uncultivable microorganisms account for over 99% of all species on the planet, but their functions are yet not well characterized. Though many cultivable degraders for n-alkanes have been intensively investigated, the roles of functional n-alkane degraders remain hidden in the natural environment. This study introduces the novel magnetic nanoparticle-mediated isolation (MMI) technology in Nigerian soils and successfully separates functional microbes belonging to the families Oxalobacteraceae and Moraxellaceae, which were dominant and responsible for alkane metabolism in situ. The alkR-type n-alkane monooxygenase genes, instead of alkA- or alkP-type, were the key functional genes involved in the n-alkane degradation process. Further physiological investigation via a BIOLOG PM plate revealed some carbon (Tween 20, Tween 40 and Tween 80) and nitrogen (tyramine, L-glutamine and D-aspartic acid) sources promoting microbial respiration and n-alkane degradation. With further addition of promoter carbon or nitrogen sources, the separated functional alkane degraders significantly improved n-alkane biodegradation rates. This suggests that MMI is a promising technology for separating functional microbes from complex microbiota, with deeper insight into their ecological functions and influencing factors. The technique also broadens the application of the BIOLOG PM plate for physiological research on functional yet uncultivable microorganisms

Modification of PEDOT: PSS to enhance device efficiency and stability of the Quasi-2D perovskite light-emitting diodes

Poly(3,4-ethylenedioxy thiophene): poly(styrene sulfonate) (PEDOT: PSS) is a hole transport layer (HTL) that is often employed in a diverse array of optoelectronic devices, such as perovskite solar cells and perovskite light-emitting diodes (PeLEDs). By simply doping lithium fluoride (LiF) into PEDOT: PSS, we demonstrate that the electrical characteristics of the HTL can be modified. Especially in quasi-2D perovskite LEDs, the crystallization process is regulated by LiF modification, leading to reduced phase impurity defects and improved carrier transport in the perovskite emission layer. Therefore, the luminance and efficiency of the quasi-2D PeLEDs are notably enhanced. The optimized PeLED with LiF modification exhibits a peak luminance of 21517 cd m−2 with 317% higher than the standard PeLED; and a high current efficiency of 39.8 cd A−1 with 237% higher than the standard PeLED. Moreover, the device stability is also improved with a nearly doubled half lifetime due to the reduced phase impurities. The work demonstrates a facile yet effective method for altering PEDOT: PSS hole transport layer, emphasizing the critical role of the underneath layer in the crystallization of quasi-2D perovskites