SGGNet2^2: Speech-Scene Graph Grounding Network for Speech-guided Navigation

The spoken language serves as an accessible and efficient interface, enabling non-experts and disabled users to interact with complex assistant robots. However, accurately grounding language utterances gives a significant challenge due to the acoustic variability in speakers' voices and environmental noise. In this work, we propose a novel speech-scene graph grounding network (SGGNet$^2$) that robustly grounds spoken utterances by leveraging the acoustic similarity between correctly recognized and misrecognized words obtained from automatic speech recognition (ASR) systems. To incorporate the acoustic similarity, we extend our previous grounding model, the scene-graph-based grounding network (SGGNet), with the ASR model from NVIDIA NeMo. We accomplish this by feeding the latent vector of speech pronunciations into the BERT-based grounding network within SGGNet. We evaluate the effectiveness of using latent vectors of speech commands in grounding through qualitative and quantitative studies. We also demonstrate the capability of SGGNet$^2$ in a speech-based navigation task using a real quadruped robot, RBQ-3, from Rainbow Robotics.Comment: 7 pages, 6 figures, Paper accepted for the Special Session at the 2023 International Symposium on Robot and Human Interactive Communication (RO-MAN), [Dohyun Kim, Yeseung Kim, Jaehwi Jang, and Minjae Song] contributed equally to this wor

Ultra-Low-Power IoT Solutions for Sound Source Localization: Combining Mixed-Signal Processing and Machine Learning

With the prevalence of smartphones, pedestrians and joggers today often walk or run while listening to music. Since they are deprived of auditory stimuli that could provide important cues to dangers, they are at a much greater risk of being hit by cars or other vehicles. We start this research into building a wearable system that uses multichannel audio sensors embedded in a headset to help detect and locate cars from their honks and engine and tire noises. Based on this detection, the system can warn pedestrians of the imminent danger of approaching cars. We demonstrate that using a segmented architecture and implementation consisting of headset-mounted audio sensors, front-end hardware that performs signal processing and feature extraction, and machine-learning-based classification on a smartphone, we are able to provide early danger detection in real time, from up to 80m distance, with greater than 80% precision and 90% recall, and alert the user on time (about 6s in advance for a car traveling at 30mph). The time delay between audio signals in a microphone array is the most important feature for sound-source localization. This work also presents a polarity-coincidence, adaptive time-delay estimation (PCC-ATDE) mixed-signal technique that uses 1-bit quantized signals and a negative-feedback architecture to directly determine the time delay between signals in the analog inputs and convert it to a digital number. This direct conversion, without a multibit ADC and further digital-signal processing, allows for ultra low power consumption. A prototype chip in 0:18μm CMOS with 4 analog inputs consumes 78nW with a 3-channel 8-bit digital time-delay output while sampling at 50kHz with a 20μs resolution and 6.06 ENOB. We present a theoretical analysis for the nonlinear, signal-dependent feedback loop of the PCC-ATDE. A delay-domain model of the system is developed to estimate the power bandwidth of the converter and predict its dynamic response. Results are validated with experiments using real-life stimuli, captured with a microphone array, that demonstrate the technique’s ability to localize a sound source. The chip is further integrated in an embedded platform and deployed as an audio-based vehicle-bearing IoT system. Finally, we investigate the signal’s envelope, an important feature for a host of applications enabled by machine-learning algorithms. Conventionally, the raw analog signal is digitized first, followed by feature extraction in the digital domain. This work presents an ultra-low-power envelope-to-digital converter (EDC) consisting of a passive switched-capacitor envelope detector and an inseparable successive approximation-register analog-to-digital converter (ADC). The two blocks integrate directly at different sampling rates without a buffer between them thanks to the ping-pong operation of their sampling capacitors. An EDC prototype was fabricated in 180nm CMOS. It provides 7.1 effective bits of ADC resolution and supports input signal bandwidth up to 5kHz and an envelope bandwidth up to 50Hz while consuming 9.6nW

Calibration Instrumentation for the Hydrogen Intensity and Real-Time Analysis eXperiment

The Hydrogen Intensity and Real-time Analysis eXperiment (HIRAX) is a 21 cm neutral hydrogen intensity mapping experiment to be deployed in the Karoo Desert in South Africa. It aims to improve constraints on the dark energy equation of state through measurements of large-scale structure at high redshift, while doubling as a state-of-the-art fast radio burst (FRB) detector. This dissertation focuses on two aspects of the HIRAX instrument characterization: (1) optimizing the signal-to-noise of antennas, through the design and implementation of a custom test-bed for determining the noise temperature of radio antennas operating between 400-800MHz, and (2) mapping the HIRAX telescope beam pattern with a custom drone calibration system. The work described is critical to HIRAX\u27s development, both by informing final antenna design and providing the tools to generate beam maps that will factor into all cosmological analysis

Design, implementation and data analysis of an embedded system for measuring environmental quantities

The article describes the development and implementation of a complex monitoring system for measuring the concentration of carbon dioxide, ambient temperature, relative humidity and atmospheric pressure. The presented system was installed at two locations. The first was in the rooms at the Department of Cybernetics and Biomedical Engineering, Faculty of Electrical Engineering and Computer Science, VSB-Technical University of Ostrava. The second was in the classrooms of the Grammar School and Secondary School of Electrical Engineering and Computer Science in Frentat pod Radhotem. The article contains a detailed description of the entire measurement network, whose basic component was a device for measuring carbon dioxide concentration, temperature and relative humidity in ambient air and atmospheric pressure via wireless data transmission using IQRF((R)) technology. Measurements were conducted continuously for several months. The data were archived in a database. The article also describes the methods for processing the data with statistical analysis. Carbon dioxide concentration was selected for data analysis. Data were selected from at least two different rooms at each location. The processed results represent the time periods for the given carbon dioxide concentrations. The graphs display in percent how much of the time students or employees spent exposed to safe or dangerous concentrations of carbon dioxide. The collected data were used for the future improvement of air quality in the rooms.Web of Science208art. no. 230

Integrating passive ubiquitous surfaces into human-computer interaction

Mobile technologies enable people to interact with computers ubiquitously. This dissertation investigates how ordinary, ubiquitous surfaces can be integrated into human-computer interaction to extend the interaction space beyond the edge of the display. It turns out that acoustic and tactile features generated during an interaction can be combined to identify input events, the user, and the surface. In addition, it is shown that a heterogeneous distribution of different surfaces is particularly suitable for realizing versatile interaction modalities. However, privacy concerns must be considered when selecting sensors, and context can be crucial in determining whether and what interaction to perform.Mobile Technologien ermöglichen den Menschen eine allgegenwärtige Interaktion mit Computern. Diese Dissertation untersucht, wie gewöhnliche, allgegenwärtige Oberflächen in die Mensch-Computer-Interaktion integriert werden können, um den Interaktionsraum über den Rand des Displays hinaus zu erweitern. Es stellt sich heraus, dass akustische und taktile Merkmale, die während einer Interaktion erzeugt werden, kombiniert werden können, um Eingabeereignisse, den Benutzer und die Oberfläche zu identifizieren. Darüber hinaus wird gezeigt, dass eine heterogene Verteilung verschiedener Oberflächen besonders geeignet ist, um vielfältige Interaktionsmodalitäten zu realisieren. Bei der Auswahl der Sensoren müssen jedoch Datenschutzaspekte berücksichtigt werden, und der Kontext kann entscheidend dafür sein, ob und welche Interaktion durchgeführt werden soll

Deep learning for internet of underwater things and ocean data analytics

The Internet of Underwater Things (IoUT) is an emerging technological ecosystem developed for connecting objects in maritime and underwater environments. IoUT technologies are empowered by an extreme number of deployed sensors and actuators. In this thesis, multiple IoUT sensory data are augmented with machine intelligence for forecasting purposes

Towards a Common Software/Hardware Methodology for Future Advanced Driver Assistance Systems

The European research project DESERVE (DEvelopment platform for Safe and Efficient dRiVE, 2012-2015) had the aim of designing and developing a platform tool to cope with the continuously increasing complexity and the simultaneous need to reduce cost for future embedded Advanced Driver Assistance Systems (ADAS). For this purpose, the DESERVE platform profits from cross-domain software reuse, standardization of automotive software component interfaces, and easy but safety-compliant integration of heterogeneous modules. This enables the development of a new generation of ADAS applications, which challengingly combine different functions, sensors, actuators, hardware platforms, and Human Machine Interfaces (HMI). This book presents the different results of the DESERVE project concerning the ADAS development platform, test case functions, and validation and evaluation of different approaches. The reader is invited to substantiate the content of this book with the deliverables published during the DESERVE project. Technical topics discussed in this book include:Modern ADAS development platforms;Design space exploration;Driving modelling;Video-based and Radar-based ADAS functions;HMI for ADAS;Vehicle-hardware-in-the-loop validation system

Hydrodynamics-Biology Coupling for Algae Culture and Biofuel Production

International audienceBiofuel production from microalgae represents an acute optimization problem for industry. There is a wide range of parameters that must be taken into account in the development of this technology. Here, mathematical modelling has a vital role to play. The potential of microalgae as a source of biofuel and as a technological solution for CO2 fixation is the subject of intense academic and industrial research. Large-scale production of microalgae has potential for biofuel applications owing to the high productivity that can be attained in high-rate raceway ponds. We show, through 3D numerical simulations, that our approach is capable of discriminating between situations where the paddle wheel is rapidly moving water or slowly agitating the process. Moreover, the simulated velocity fields can provide lagrangian trajectories of the algae. The resulting light pattern to which each cell is submitted when travelling from light (surface) to dark (bottom) can then be derived. It will then be reproduced in lab experiments to study photosynthesis under realistic light patterns

Recent Advances in Embedded Computing, Intelligence and Applications

The latest proliferation of Internet of Things deployments and edge computing combined with artificial intelligence has led to new exciting application scenarios, where embedded digital devices are essential enablers. Moreover, new powerful and efficient devices are appearing to cope with workloads formerly reserved for the cloud, such as deep learning. These devices allow processing close to where data are generated, avoiding bottlenecks due to communication limitations. The efficient integration of hardware, software and artificial intelligence capabilities deployed in real sensing contexts empowers the edge intelligence paradigm, which will ultimately contribute to the fostering of the offloading processing functionalities to the edge. In this Special Issue, researchers have contributed nine peer-reviewed papers covering a wide range of topics in the area of edge intelligence. Among them are hardware-accelerated implementations of deep neural networks, IoT platforms for extreme edge computing, neuro-evolvable and neuromorphic machine learning, and embedded recommender systems

MEMS enabled miniaturisation of photoacoustic imaging and sensing systems

This work presents multiple advances toward miniaturised photoacoustic imaging systems. Miniaturising the system is done in two steps. Firstly, by using novel custom arrays of piezoelectric miniaturised ultrasound transducers. The arrays were fabricated using a cost-efficient multi-user process. The achievable upper frequency limits were restricted by the design limitations of the multi-user process. The designs comprised of a single frequency and two frequency staggered arrays. They were characterised using laser Doppler velocimetry, pitch and catch technique as well as photoacoustic excitation. Additionally, the arrays were compared to commercial bulk ultrasound transducers. The custom-made PMUT arrays perform well compared to commercial transducer, despite their significantly smaller (two orders of magnitude) detection area. Secondly, an optical resolution photoacoustic microscope consisting consisting of MEMS based excitation - using a fast-scanning micro-mirror for Q-switching - and detection schemes is built and used to image synthetic targets and phantoms. Furthermore, a simulation model of the system is developed to evaluate influences of the miniaturised elements on the photoacoustic signal generation and received spectra and signal strength. Finally, a novel photoacoustic excitation scheme based on CW - laser excitation and a MEMS based fast-scanning micro-mirror is presented and its performance relative to pulsed excitation photoacoustic imaging is evaluated. Here, the photoacoustic excitation is not due to fast pulsed laser excitation, but caused by scanning a focused CW - beam over a sample.This work presents multiple advances toward miniaturised photoacoustic imaging systems. Miniaturising the system is done in two steps. Firstly, by using novel custom arrays of piezoelectric miniaturised ultrasound transducers. The arrays were fabricated using a cost-efficient multi-user process. The achievable upper frequency limits were restricted by the design limitations of the multi-user process. The designs comprised of a single frequency and two frequency staggered arrays. They were characterised using laser Doppler velocimetry, pitch and catch technique as well as photoacoustic excitation. Additionally, the arrays were compared to commercial bulk ultrasound transducers. The custom-made PMUT arrays perform well compared to commercial transducer, despite their significantly smaller (two orders of magnitude) detection area. Secondly, an optical resolution photoacoustic microscope consisting consisting of MEMS based excitation - using a fast-scanning micro-mirror for Q-switching - and detection schemes is built and used to image synthetic targets and phantoms. Furthermore, a simulation model of the system is developed to evaluate influences of the miniaturised elements on the photoacoustic signal generation and received spectra and signal strength. Finally, a novel photoacoustic excitation scheme based on CW - laser excitation and a MEMS based fast-scanning micro-mirror is presented and its performance relative to pulsed excitation photoacoustic imaging is evaluated. Here, the photoacoustic excitation is not due to fast pulsed laser excitation, but caused by scanning a focused CW - beam over a sample