Active Tactile Sensing for Texture Perception in Robotic Systems

This thesis presents a comprehensive study of tactile sensing, particularly on the prob- lem of active texture perception. It includes a brief introduction to tactile sensing technology and the neural basis for tactile perception. It follows the literature review of textural percep- tion with tactile sensing. I propose a decoding and perception pipeline to tackle fine-texture classification/identification problems via active touching. Experiments are conducted using a 7DOF robotic arm with a finger-shaped tactile sensor mounted on the end-effector to per- form sliding/rubbing movements on multiple fabrics. Low-dimensional frequency features are extracted from the raw signals to form a perceptive feature space, where tactile signals are mapped and segregated into fabric classes. Fabric classes can be parameterized and sim- plified in the feature space using elliptical equations. Results from experiments of varied control parameters are compared and visualized to show that different exploratory move- ments have an apparent impact on the perceived tactile information. It implies the possibil- ity of optimising the robotic movements to improve the textural classification/identification performance

Embedded Electronic Systems for Electronic Skin Applications

The advances in sensor devices are potentially providing new solutions to many applications including prosthetics and robotics. Endowing upper limb prosthesis with tactile sensors (electronic/sensitive skin) can be used to provide tactile sensory feedback to the amputees. In this regard, the prosthetic device is meant to be equipped with tactile sensing system allowing the user limb to receive tactile feedback about objects and contact surfaces. Thus, embedding tactile sensing system is required for wearable sensors that should cover wide areas of the prosthetics. However, embedding sensing system involves set of challenges in terms of power consumption, data processing, real-time response and design scalability (e-skin may include large number of tactile sensors). The tactile sensing system is constituted of: (i) a tactile sensor array, (ii) an interface electronic circuit, (iii) an embedded processing unit, and (iv) a communication interface to transmit tactile data. The objective of the thesis is to develop an efficient embedded tactile sensing system targeting e-skin application (e.g. prosthetic) by: 1) developing a low power and miniaturized interface electronics circuit, operating in real-time; 2) proposing an efficient algorithm for embedded tactile data processing, affecting the system time latency and power consumption; 3) implementing an efficient communication channel/interface, suitable for large amount of data generated from large number of sensors. Most of the interface electronics for tactile sensing system proposed in the literature are composed of signal conditioning and commercial data acquisition devices (i.e. DAQ). However, these devices are bulky (PC-based) and thus not suitable for portable prosthetics from the size, power consumption and scalability point of view. Regarding the tactile data processing, some works have exploited machine learning methods for extracting meaningful information from tactile data. However, embedding these algorithms poses some challenges because of 1) the high amount of data to be processed significantly affecting the real time functionality, and 2) the complex processing tasks imposing burden in terms of power consumption. On the other hand, the literature shows lack in studies addressing data transfer in tactile sensing system. Thus, dealing with large number of sensors will pose challenges on the communication bandwidth and reliability. Therefore, this thesis exploits three approaches: 1) Developing a low power and miniaturized Interface Electronics (IE), capable of interfacing and acquiring signals from large number of tactile sensors in real-time. We developed a portable IE system based on a low power arm microcontroller and a DDC232 A/D converter, that handles an array of 32 tactile sensors. Upon touch applied to the sensors, the IE acquires and pre-process the sensor signals at low power consumption achieving a battery lifetime of about 22 hours. Then we assessed the functionality of the IE by carrying out Electrical and electromechanical characterization experiments to monitor the response of the interface electronics with PVDF-based piezoelectric sensors. The results of electrical and electromechanical tests validate the correct functionality of the proposed system. In addition, we implemented filtering methods on the IE that reduced the effect of noise in the system. Furthermore, we evaluated our proposed IE by integrating it in tactile sensory feedback system, showing effective deliver of tactile data to the user. The proposed system overcomes similar state of art solutions dealing with higher number of input channels and maintaining real time functionality. 2) Optimizing and implementing a tensorial-based machine learning algorithm for touch modality classification on embedded Zynq System-on-chip (SoC). The algorithm is based on Support Vector Machine classifier to discriminate between three input touch modality classes \u201cbrushing\u201d, \u201crolling\u201d and \u201csliding\u201d. We introduced an efficient algorithm minimizing the hardware implementation complexity in terms of number of operations and memory storage which directly affect time latency and power consumption. With respect to the original algorithm, the proposed approach \u2013 implemented on Zynq SoC \u2013 achieved reduction in the number of operations per inference from 545 M-ops to 18 M-ops and the memory storage from 52.2 KB to 1.7 KB. Moreover, the proposed method speeds up the inference time by a factor of 43 7 at a cost of only 2% loss in accuracy, enabling the algorithm to run on embedded processing unit and to extract tactile information in real-time. 3) Implementing a robust and efficient data transfer channel to transfer aggregated data at high transmission data rate and low power consumption. In this approach, we proposed and demonstrated a tactile sensory feedback system based on an optical communication link for prosthetic applications. The optical link features a low power and wide transmission bandwidth, which makes the feedback system suitable for large number of tactile sensors. The low power transmission is due to the employed UWB-based optical modulation. We implemented a system prototype, consisting of digital transmitter and receiver boards and acquisition circuits to interface 32 piezoelectric sensors. Then we evaluated the system performance by measuring, processing and transmitting data of the 32 piezoelectric sensors at 100 Mbps data rate through the optical link, at 50 pJ/bit communication energy consumption. Experimental results have validated the functionality and demonstrated the real time operation of the proposed sensory feedback system

Soft mechanical sensors for wearable and implantable applications

Wearable and implantable sensing of biomechanical signals such as pressure, strain, shear, and vibration can enable a multitude of human-integrated applications, including on-skin monitoring of vital signs, motion tracking, monitoring of internal organ condition, restoration of lost/impaired mechanoreception, among many others. The mechanical conformability of such sensors to the human skin and tissue is critical to enhancing their biocompatibility and sensing accuracy. As such, in the recent decade, significant efforts have been made in the development of soft mechanical sensors. To satisfy the requirements of different wearable and implantable applications, such sensors have been imparted with various additional properties to make them better suited for the varied contexts of human-integrated applications. In this review, focusing on the four major types of soft mechanical sensors for pressure, strain, shear, and vibration, we discussed the recent material and device design innovations for achieving several important properties, including flexibility and stretchability, bioresorbability and biodegradability, self-healing properties, breathability, transparency, wireless communication capabilities, and high-density integration. We then went on to discuss the current research state of the use of such novel soft mechanical sensors in wearable and implantable applications, based on which future research needs were further discussed. This article is categorized under: Diagnostic Tools > Biosensing Diagnostic Tools > Diagnostic Nanodevices Implantable Materials and Surgical Technologies > Nanomaterials and Implants</p

Multi-touch Detection and Semantic Response on Non-parametric Rear-projection Surfaces

The ability of human beings to physically touch our surroundings has had a profound impact on our daily lives. Young children learn to explore their world by touch; likewise, many simulation and training applications benefit from natural touch interactivity. As a result, modern interfaces supporting touch input are ubiquitous. Typically, such interfaces are implemented on integrated touch-display surfaces with simple geometry that can be mathematically parameterized, such as planar surfaces and spheres; for more complicated non-parametric surfaces, such parameterizations are not available. In this dissertation, we introduce a method for generalizable optical multi-touch detection and semantic response on uninstrumented non-parametric rear-projection surfaces using an infrared-light-based multi-camera multi-projector platform. In this paradigm, touch input allows users to manipulate complex virtual 3D content that is registered to and displayed on a physical 3D object. Detected touches trigger responses with specific semantic meaning in the context of the virtual content, such as animations or audio responses. The broad problem of touch detection and response can be decomposed into three major components: determining if a touch has occurred, determining where a detected touch has occurred, and determining how to respond to a detected touch. Our fundamental contribution is the design and implementation of a relational lookup table architecture that addresses these challenges through the encoding of coordinate relationships among the cameras, the projectors, the physical surface, and the virtual content. Detecting the presence of touch input primarily involves distinguishing between touches (actual contact events) and hovers (near-contact proximity events). We present and evaluate two algorithms for touch detection and localization utilizing the lookup table architecture. One of the algorithms, a bounded plane sweep, is additionally able to estimate hover-surface distances, which we explore for interactions above surfaces. The proposed method is designed to operate with low latency and to be generalizable. We demonstrate touch-based interactions on several physical parametric and non-parametric surfaces, and we evaluate both system accuracy and the accuracy of typical users in touching desired targets on these surfaces. In a formative human-subject study, we examine how touch interactions are used in the context of healthcare and present an exploratory application of this method in patient simulation. A second study highlights the advantages of touch input on content-matched physical surfaces achieved by the proposed approach, such as decreases in induced cognitive load, increases in system usability, and increases in user touch performance. In this experiment, novice users were nearly as accurate when touching targets on a 3D head-shaped surface as when touching targets on a flat surface, and their self-perception of their accuracy was higher

Piezoelectric Transducers Based on Aluminum Nitride and Polyimide for Tactile Applications

The development of micro systems with smart sensing capabilities is paving the way to progresses in the technology for humanoid robotics. The importance of sensory feedback has been recognized the enabler of a high degree of autonomy for robotic systems. In tactile applications, it can be exploited not only to avoid objects slipping during their manipulation but also to allow safe interaction with humans and unknown objects and environments. In order to ensure the minimal deformation of an object during subtle manipulation tasks, information not only on contact forces between the object and fingers but also on contact geometry and contact friction characteristics has to be provided. Touch, unlike other senses, is a critical component that plays a fundamental role in dexterous manipulation capabilities and in the evaluation of objects properties such as type of material, shape, texture, stiffness, which is not easily possible by vision alone. Understanding of unstructured environments is made possible by touch through the determination of stress distribution in the surrounding area of physical contact. To this aim, tactile sensing and pressure detection systems should be integrated as an artificial tactile system. As illustrated in the Chapter I, the role of external stimuli detection in humans is provided by a great number of sensorial receptors: they are specialized endings whose structure and location in the skin determine their specific signal transmission characteristics. Especially, mechanoreceptors are specialized in the conversion of the mechanical deformations caused by force, vibration or slip on skin into electrical nerve impulses which are processed and encoded by the central nervous system. Highly miniaturized systems based on MEMS technology seem to imitate properly the large number of fast responsive mechanoreceptors present in human skin. Moreover, an artificial electronic skin should be lightweight, flexible, soft and wearable and it should be fabricated with compliant materials. In this respect a big challenge of bio-inspired technologies is the efficient application of flexible active materials to convert the mechanical pressure or stress into a usable electric signal (voltage or current). In the emerging field of soft active materials, able of large deformation, piezoelectrics have been recognized as a really promising and attractive material in both sensing and actuation applications. As outlined in Chapter II, there is a wide choice of materials and material forms (ceramics: PZT; polycrystalline films: ZnO, AlN; polymers and copolymers: PVDF, PVDF-TrFe) which are actively piezoelectric and exhibit features more or less attractive. Among them, aluminum nitride is a promising piezoelectric material for flexible technology. It has moderate piezoelectric coefficient, when available in c-axis oriented polycrystalline columnar structure, but, at same time, it exhibits low dielectric constant, high temperature stability, large band gap, large electrical resistivity, high breakdown voltage and low dielectric loss which make it suitable for transducers and high thermal conductivity which implies low thermal drifts. The high chemical stability allows AlN to be used in humid environments. Moreover, all the above properties and its deposition method make AlN compatible with CMOS technology. Exploiting the features of the AlN, three-dimensional AlN dome-shaped cells, embedded between two metal electrodes, are proposed in this thesis. They are fabricated on general purpose Kapton™ substrate, exploiting the flexibility of the polymer and the electrical stability of the semiconductor at the same time. As matter of fact, the crystalline layers release a compressive stress over the polymer, generating three-dimensional structures with reduced stiffness, compared to the semiconductor materials. In Chapter III, a mathematical model to calculate the residual stresses which arise because of mismatch in coefficient of thermal expansion between layers and because of mismatch in lattice constants between the substrate and the epitaxially grown films is adopted. The theoretical equation is then used to evaluate the dependence of geometrical features of the fabricated three-dimensional structures on compressive residual stress. Moreover, FEM simulations and theoretical models analysis are developed in order to qualitative explore the operation principle of curved membranes, which are labelled dome-shaped diaphragm transducers (DSDT), both as sensors and as piezo-actuators and for the related design optimization. For the reliability of the proposed device as a force/pressure sensor and piezo-actuator, an exhaustive electromechanical characterization of the devices is carried out. A complete description of the microfabrication processes is also provided. As shown in Chapter IV, standard microfabrication techniques are employed to fabricate the array of DSDTs. The overall microfabrication process involves deposition of metal and piezoelectric films, photolithography and plasma-based dry and wet etching to pattern thin films with the desired features. The DSDT devices are designed and developed according to FEM and theoretical analysis and following the typical requirements of force/pressure systems for tactile applications. Experimental analyses are also accomplished to extract the relationship between the compressive residual stress due to the aluminum nitride and the geometries of the devices. They reveal different deformations, proving the dependence of the geometrical features of the three-dimensional structures on residual stress. Moreover, electrical characterization is performed to determine capacitance and impedance of the DSDTs and to experimentally calculate the relative dielectric constant of sputtered AlN piezoelectric film. In order to investigate the mechanical behaviour of the curved circular transducers, a characterization of the flexural deflection modes of the DSDT membranes is carried out. The natural frequency of vibrations and the corresponding displacements are measured by a Laser Doppler Vibrometer when a suitable oscillating voltage, with known amplitude, is applied to drive the piezo-DSDTs. Finally, being developed for tactile sensing purpose, the proposed technology is tested in order to explore the electromechanical response of the device when impulsive dynamic and/or long static forces are applied. The study on the impulsive dynamic and long static stimuli detection is then performed by using an ad hoc setup measuring both the applied loading forces and the corresponding generated voltage and capacitance variation. These measurements allow a thorough test of the sensing abilities of the AlN-based DSDT cells. Finally, as stated in Chapter V, the proposed technology exhibits an improved electromechanical coupling with higher mechanical deformation per unit energy compared with the conventional plate structures, when the devices are used as piezo-actuator. On the other hand, it is well suited to realize large area tactile sensors for robotics applications, opening up new perspectives to the development of latest generation biomimetic sensors and allowing the design and the fabrication of miniaturized devices

Electronic systems for the restoration of the sense of touch in upper limb prosthetics

In the last few years, research on active prosthetics for upper limbs focused on improving the human functionalities and the control. New methods have been proposed for measuring the user muscle activity and translating it into the prosthesis control commands. Developing the feed-forward interface so that the prosthesis better follows the intention of the user is an important step towards improving the quality of life of people with limb amputation. However, prosthesis users can neither feel if something or someone is touching them over the prosthesis and nor perceive the temperature or roughness of objects. Prosthesis users are helped by looking at an object, but they cannot detect anything otherwise. Their sight gives them most information. Therefore, to foster the prosthesis embodiment and utility, it is necessary to have a prosthetic system that not only responds to the control signals provided by the user, but also transmits back to the user the information about the current state of the prosthesis. This thesis presents an electronic skin system to close the loop in prostheses towards the restoration of the sense of touch in prosthesis users. The proposed electronic skin system inlcudes an advanced distributed sensing (electronic skin), a system for (i) signal conditioning, (ii) data acquisition, and (iii) data processing, and a stimulation system. The idea is to integrate all these components into a myoelectric prosthesis. Embedding the electronic system and the sensing materials is a critical issue on the way of development of new prostheses. In particular, processing the data, originated from the electronic skin, into low- or high-level information is the key issue to be addressed by the embedded electronic system. Recently, it has been proved that the Machine Learning is a promising approach in processing tactile sensors information. Many studies have been shown the Machine Learning eectiveness in the classication of input touch modalities.More specically, this thesis is focused on the stimulation system, allowing the communication of a mechanical interaction from the electronic skin to prosthesis users, and the dedicated implementation of algorithms for processing tactile data originating from the electronic skin. On system level, the thesis provides design of the experimental setup, experimental protocol, and of algorithms to process tactile data. On architectural level, the thesis proposes a design ow for the implementation of digital circuits for both FPGA and integrated circuits, and techniques for the power management of embedded systems for Machine Learning algorithms

Physical sketching tools and techniques for customized sensate surfaces

Sensate surfaces are a promising avenue for enhancing human interaction with digital systems due to their inherent intuitiveness and natural user interface. Recent technological advancements have enabled sensate surfaces to surpass the constraints of conventional touchscreens by integrating them into everyday objects, creating interactive interfaces that can detect various inputs such as touch, pressure, and gestures. This allows for more natural and intuitive control of digital systems. However, prototyping interactive surfaces that are customized to users' requirements using conventional techniques remains technically challenging due to limitations in accommodating complex geometric shapes and varying sizes. Furthermore, it is crucial to consider the context in which customized surfaces are utilized, as relocating them to fabrication labs may lead to the loss of their original design context. Additionally, prototyping high-resolution sensate surfaces presents challenges due to the complex signal processing requirements involved. This thesis investigates the design and fabrication of customized sensate surfaces that meet the diverse requirements of different users and contexts. The research aims to develop novel tools and techniques that overcome the technical limitations of current methods and enable the creation of sensate surfaces that enhance human interaction with digital systems.Sensorische Oberflächen sind aufgrund ihrer inhärenten Intuitivität und natürlichen Benutzeroberfläche ein vielversprechender Ansatz, um die menschliche Interaktionmit digitalen Systemen zu verbessern. Die jüngsten technologischen Fortschritte haben es ermöglicht, dass sensorische Oberflächen die Beschränkungen herkömmlicher Touchscreens überwinden, indem sie in Alltagsgegenstände integriert werden und interaktive Schnittstellen schaffen, die diverse Eingaben wie Berührung, Druck, oder Gesten erkennen können. Dies ermöglicht eine natürlichere und intuitivere Steuerung von digitalen Systemen. Das Prototyping interaktiver Oberflächen, die mit herkömmlichen Techniken an die Bedürfnisse der Nutzer angepasst werden, bleibt jedoch eine technische Herausforderung, da komplexe geometrische Formen und variierende Größen nur begrenzt berücksichtigt werden können. Darüber hinaus ist es von entscheidender Bedeutung, den Kontext, in dem diese individuell angepassten Oberflächen verwendet werden, zu berücksichtigen, da eine Verlagerung in Fabrikations-Laboratorien zum Verlust ihres ursprünglichen Designkontextes führen kann. Zudem stellt das Prototyping hochauflösender sensorischer Oberflächen aufgrund der komplexen Anforderungen an die Signalverarbeitung eine Herausforderung dar. Diese Arbeit erforscht dasDesign und die Fabrikation individuell angepasster sensorischer Oberflächen, die den diversen Anforderungen unterschiedlicher Nutzer und Kontexte gerecht werden. Die Forschung zielt darauf ab, neuartigeWerkzeuge und Techniken zu entwickeln, die die technischen Beschränkungen derzeitigerMethoden überwinden und die Erstellung von sensorischen Oberflächen ermöglichen, die die menschliche Interaktion mit digitalen Systemen verbessern

Printing Technologies on Flexible Substrates for Printed Electronics

Printing technologies have been demonstrated to be highly efficient and compatible with polymeric materials (both inks and substrates) enabling a new generation of flexible electronics applications. Conductive flexible polymers are a new class of materials that are prepared for a wide range of applications, such as photovoltaic solar cells, transistors molecular devices, and sensors and actuators. There are many possible printing techniques. This chapter provides an opportunity to review the most common printing techniques used at the industrial level, the most commonly used substrates and electronic materials, giving an overall vision for a better understanding and evaluation of their different features. Several technological solutions (contact/noncontact) and its critical challenges are also presented. Inkjet Printing Technology (IPT) has been receiving a great attention and therefore higher focus is given to this technology. An overview of IPT is presented to evidence its importance and potential as a key-technology on the research field for printed electronics development, as well as on large scale industrial manufacturing. A background and a review on prior work are presented along with used materials, developed applications and potential of IPT technology. The main features of the different printing technologies, advantages and main challenges are also compared

Machine Learning in Sensors and Imaging

Machine learning is extending its applications in various fields, such as image processing, the Internet of Things, user interface, big data, manufacturing, management, etc. As data are required to build machine learning networks, sensors are one of the most important technologies. In addition, machine learning networks can contribute to the improvement in sensor performance and the creation of new sensor applications. This Special Issue addresses all types of machine learning applications related to sensors and imaging. It covers computer vision-based control, activity recognition, fuzzy label classification, failure classification, motor temperature estimation, the camera calibration of intelligent vehicles, error detection, color prior model, compressive sensing, wildfire risk assessment, shelf auditing, forest-growing stem volume estimation, road management, image denoising, and touchscreens