Low-fi skin vision: A case study in rapid prototyping a sensory substitution system

We describe the design process we have used to develop a minimal, twenty vibration motor Tactile Vision Sensory Substitution (TVSS) system which enables blind-folded subjects to successfully track and bat a rolling ball and thereby experience 'skin vision'. We have employed a low-fi rapid prototyping approach to build this system and argue that this methodology is particularly effective for building embedded interactive systems. We support this argument in two ways. First, by drawing on theoretical insights from robotics, a discipline that also has to deal with the challenge of building complex embedded systems that interact with their environments; second, by using the development of our TVSS as a case study: describing the series of prototypes that led to our successful design and highlighting what we learnt at each stage

LifeChair: A Conductive Fabric Sensor-Based Smart Cushion for Actively Shaping Sitting Posture.

The LifeChair is a smart cushion that provides vibrotactile feedback by actively sensing and classifying sitting postures to encourage upright posture and reduce slouching. The key component of the LifeChair is our novel conductive fabric pressure sensing array. Fabric sensors have been explored in the past, but a full sensing solution for embedded real world use has not been proposed. We have designed our system with commercial use in mind, and as a result, it has a high focus on manufacturability, cost-effectiveness and adaptiveness. We demonstrate the performance of our fabric sensing system by installing it into the LifeChair and comparing its posture detection accuracy with our previous study that implemented a conventional flexible printed PCB-sensing system. In this study, it is shown that the LifeChair can detect all 11 postures across 20 participants with an improved average accuracy of 98.1%, and it demonstrates significantly lower variance when interfacing with different users. We also conduct a performance study with 10 participants to evaluate the effectiveness of the LifeChair device in improving upright posture and reducing slouching. Our performance study demonstrates that the LifeChair is effective in encouraging users to sit upright with an increase of 68.1% in time spent seated upright when vibrotactile feedback is activated

Cortical Orchestra Conducted by Purpose and Function

학위논문(박사)--서울대학교 대학원 :자연과학대학 협동과정 뇌과학전공,2020. 2. 정천기.촉각과 자기수용감각은 우리의 생존 및 일상생활에 절대적인 영향을 미치는 중요한 감각 기능이다. 말초신경계에서 이 두 가지 기능들에 필요한 정보를 수집하고 전달하는 기계적 수용기 및 그 구심성 신경들에 대한 신호 전달 메커니즘 및 그 특징들은 상대적으로 잘 알려져 있는 편이다. 그러나, 촉각과 자기수용감각을 형성하기 위한 인간 뇌의 피질에서의 정보 처리 메커니즘에 대하여 우리가 현재 알고 있는 바는 극히 일부분이다. 이 논문에서 제시하는 일련의 연구들은 인간 뇌 피질 단계에서 촉각과 자기수용감각의 지각적 처리과정에 대한 거시적 신경계 정보처리 메커니즘을 다룬다. 첫 번째 연구에서는 뇌피질뇌파를 이용하여 인간 일차 및 이차 체성감각 피질에서 인공적인 자극과 일상생활에서 접할 수 있는 자극을 포함하는 다양한 진동촉감각 및 질감 자극에 대한 거시적 신경계 정보처리 특성을 밝혔다. 이 연구에서는 일차 및 이차 체성감각 피질의 촉감각 주파수 특이적인 하이-감마 영역 신경활동이 자극 주파수에 따라 각각 상이한 시간적 다이나믹스를 가지고 변화하는 것을 확인하였다. 또한, 이러한 하이-감마 활동은 성긴 질감과 미세한 입자감을 가진 자연스러운 질감 자극에 대해서도 진동촉감각의 경우와 유사한 패턴을 보였다. 이러한 결과들은 인간의 진동촉감각이 매우 단순한 형태에 자극일지라도 대뇌 체성감각 시스템에 있어 거시적인 다중 영역에서의 계층적 정보처리를 동반한다는 점을 시사한다. 두 번째 연구에서는 인간의 움직임과 관련된 두정엽 영역에서의 하이-감마 뇌활성이 자기수용감각과 같은 말초신경계로부터의 체성감각 피드백을 주로 반영하는지, 아니면 움직임 준비 및 제어를 위한 피질 간 신경 프로세스에 대한 활동을 반영하는지를 조사하였다. 연구 결과, 자발적 운동 중 대뇌 운동감각령에서의 하이-감마 활동은 일차 체성감각피질이 일차 운동피질보다 더 지배적인 것으로 나타났다. 또한 이 연구에서는, 움직임과 관련된 일차 체성감각피질에서의 하이-감마 뇌활동은 말초신경계로부터의 자기수용감각과 촉각에 대한 신경계 정보처리를 주로 반영하는 것을 밝혔다. 이러한 연구들을 바탕으로, 마지막 연구에서는 인간 대뇌에서의 체성감각 지각 프로세스에 대한 거시적 피질 간 네트워크를 규명하고자 하였다. 이를 위해, 51명의 뇌전증 환자에게서 체성감각을 유발했던 뇌피질전기자극 데이터와 46명의 환자에게서 촉감각 자극 및 운동 수행 중에 측정한 뇌피질뇌파 하이-감마 매핑 데이터를 종합적으로 분석하였다. 그 결과, 체성감각 지각 프로세스는 대뇌에서 넓은 영역에 걸쳐 분포하는 체성감각 관련 네트워크의 신경 활성을 수반한다는 것을 알아냈다. 또한, 뇌피질전기자극을 통한 대뇌 지도와 하이-감마 매핑을 통한 대뇌 지도는 서로 상당한 유사성을 보였다. 흥미롭게도, 뇌피질전기자극과 하이-감마 활동을 종합한 뇌지도들로부터 체성감각 관련 뇌 영역의 공간적 분포가 체성감각 기능에 따라 서로 달랐고, 그에 해당하는 각 영역들은 서로 뚜렷하게 다른 시간적 다이나믹스를 가지고 순차적으로 활성화되었다. 이러한 결과들은 체성감각에 대한 거시적 신경계 프로세스가 그 지각적 기능에 따라 뚜렷이 다른 계층적 네트워크를 가진다는 점을 시사한다. 더 나아가, 본 연구에서의 결과들은 체성감각 시스템의 지각-행동 관련 신경활동 흐름에 관한 이론적인 가설에 대하여 설득력 있는 증거를 제시하고 있다.Tactile and proprioceptive perceptions are crucial for our daily life as well as survival. At the peripheral level, the transduction mechanisms and characteristics of mechanoreceptive afferents containing information required for these functions, have been well identified. However, our knowledge about the cortical processing mechanism for them in human is limited. The present series of studies addressed the macroscopic neural mechanism for perceptual processing of tactile and proprioceptive perception in human cortex. In the first study, I investigated the macroscopic neural characteristics for various vibrotactile and texture stimuli including artificial and naturalistic ones in human primary and secondary somatosensory cortices (S1 and S2, respectively) using electrocorticography (ECoG). I found robust tactile frequency-specific high-gamma (HG, 50–140 Hz) activities in both S1 and S2 with different temporal dynamics depending on the stimulus frequency. Furthermore, similar HG patterns of S1 and S2 were found in naturalistic stimulus conditions such as coarse/fine textures. These results suggest that human vibrotactile sensation involves macroscopic multi-regional hierarchical processing in the somatosensory system, even during the simplified stimulation. In the second study, I tested whether the movement-related HG activities in parietal region mainly represent somatosensory feedback such as proprioception from periphery or primarily indicate cortico-cortical neural processing for movement preparation and control. I found that sensorimotor HG activities are more dominant in S1 than in M1 during voluntary movement. Furthermore, the results showed that movement-related HG activities in S1 mainly represent proprioceptive and tactile feedback from periphery. Given the results of previous two studies, the final study aimed to identify the large-scale cortical networks for perceptual processing in human. To do this, I combined direct cortical stimulation (DCS) data for eliciting somatosensation and ECoG HG band (50 to 150 Hz) mapping data during tactile stimulation and movement tasks, from 51 (for DCS mapping) and 46 patients (for HG mapping) with intractable epilepsy. The results showed that somatosensory perceptual processing involves neural activation of widespread somatosensory-related network in the cortex. In addition, the spatial distributions of DCS and HG functional maps showed considerable similarity in spatial distribution between high-gamma and DCS functional maps. Interestingly, the DCS-HG combined maps showed distinct spatial distributions depending on the somatosensory functions, and each area was sequentially activated with distinct temporal dynamics. These results suggest that macroscopic neural processing for somatosensation has distinct hierarchical networks depending on the perceptual functions. In addition, the results of the present study provide evidence for the perception and action related neural streams of somatosensory system. Throughout this series of studies, I suggest that macroscopic somatosensory network and structures of our brain are intrinsically organized by perceptual function and its purpose, not by somatosensory modality or submodality itself. Just as there is a purpose for human behavior, so is our brain.PART I. INTRODUCTION 1 CHAPTER 1: Somatosensory System 1 1.1. Mechanoreceptors in the Periphery 2 1.2. Somatosensory Afferent Pathways 4 1.3. Cortico-cortical Connections among Somatosensory-related Areas 7 1.4. Somatosensory-related Cortical Regions 8 CHAPTER 2: Electrocorticography 14 2.1. Intracranial Electroencephalography 14 2.2. High-Gamma Band Activity 18 CHAPTER 3: Purpose of This Study 24 PART II. EXPERIMENTAL STUDY 26 CHAPTER 4: Apparatus Design 26 4.1. Piezoelectric Vibrotactile Stimulator 26 4.2. Magnetic Vibrotactile Stimulator 29 4.3. Disc-type Texture Stimulator 33 4.4. Drum-type Texture Stimulator 36 CHAPTER 5: Vibrotactile and Texture Study 41 5.1. Introduction 42 5.2. Materials and Methods 46 5.2.1. Patients 46 5.2.2. Apparatus 47 5.2.3. Experimental Design 49 5.2.4. Data Acquisition and Preprocessing 50 5.2.5. Analysis 51 5.3. Results 54 5.3.1. Frequency-specific S1/S2 HG Activities 54 5.3.2. S1 HG Attenuation during Flutter and Vibration 62 5.3.3. Single-trial Vibration Frequency Classification 64 5.3.4. S1/S2 HG Activities during Texture Stimuli 65 5.4. Discussion 69 5.4.1. Comparison with Previous Findings 69 5.4.2. Tactile Frequency-dependent Neural Adaptation 70 5.4.3. Serial vs. Parallel Processing between S1 and S2 72 5.4.4. Conclusion of Chapter 5 73 CHAPTER 6: Somatosensory Feedback during Movement 74 6.1. Introduction 75 6.2. Materials and Methods 79 6.2.1. Subjects 79 6.2.2. Tasks 80 6.2.3. Data Acquisition and Preprocessing 82 6.2.4. S1-M1 HG Power Difference 85 6.2.5. Classification 86 6.2.6. Timing of S1 HG Activity 86 6.2.7. Correlation between HG and EMG signals 87 6.3. Results 89 6.3.1. HG Activities Are More Dominant in S1 than in M1 89 6.3.2. HG Activities in S1 Mainly Represent Somatosensory Feedback 94 6.4. Discussion 100 6.4.1. S1 HG Activity Mainly Represents Somatosensory Feedback 100 6.4.2. Further Discussion and Future Direction in BMI 102 6.4.3. Conclusion of Chapter 6 103 CHAPTER 7: Cortical Maps of Somatosensory Function 104 7.1. Introduction 106 7.2. Materials and Methods 110 7.2.1. Participants 110 7.2.2. Direct Cortical Stimulation 114 7.2.3. Classification of Verbal Feedbacks 115 7.2.4. Localization of Electrodes 115 7.2.5. Apparatus 116 7.2.6. Tasks 117 7.2.7. Data Recording and Processing 119 7.2.8. Mapping on the Brain 120 7.2.9. ROI-based Analysis 122 7.3. Results 123 7.3.1. DCS Mapping 123 7.3.2. Three and Four-dimensional HG Mapping 131 7.3.3. Neural Characteristics among Somatosensory-related Areas 144 7.4. Discussion 146 7.4.1. DCS on the Non-Primary Areas 146 7.4.2. Two Streams of Somatosensory System 148 7.4.3. Functional Role of ventral PM 151 7.4.4. Limitation and Perspective 152 7.4.5. Conclusion of Chapter 7 155 PART III. CONCLUSION 156 CHAPTER 8: Conclusion and Perspective 156 8.1. Perspective and Future Work 157 References 160 Abstract in Korean 173Docto

How brain-computer interface technology may improve the diagnosis of the disorders of consciousness: A comparative study

ObjectiveClinical assessment of consciousness relies on behavioural assessments, which have several limitations. Hence, disorder of consciousness (DOC) patients are often misdiagnosed. In this work, we aimed to compare the repetitive assessment of consciousness performed with a clinical behavioural and a Brain-Computer Interface (BCI) approach. Materials and methodsFor 7 weeks, sixteen DOC patients participated in weekly evaluations using both the Coma Recovery Scale-Revised (CRS-R) and a vibrotactile P300 BCI paradigm. To use the BCI, patients had to perform an active mental task that required detecting specific stimuli while ignoring other stimuli. We analysed the reliability and the efficacy in the detection of command following resulting from the two methodologies. ResultsOver repetitive administrations, the BCI paradigm detected command following before the CRS-R in seven patients. Four clinically unresponsive patients consistently showed command following during the BCI assessments. ConclusionBrain-Computer Interface active paradigms might contribute to the evaluation of the level of consciousness, increasing the diagnostic precision of the clinical bedside approach. SignificanceThe integration of different diagnostic methods leads to a better knowledge and care for the DOC

Tactile Data Entry for Extravehicular Activity

In the task-saturated environment of extravehicular activity (EVA), an astronaut's ability to leverage suit-integrated information systems is limited by a lack of options for data entry. In particular, bulky gloves inhibit the ability to interact with standard computing interfaces such as a mouse or keyboard. This paper presents the results of a preliminary investigation into a system that permits the space suit gloves themselves to be used as data entry devices. Hand motion tracking is combined with simple finger gesture recognition to enable use of a virtual keyboard, while tactile feedback provides touch-based context to the graphical user interface (GUI) and positive confirmation of keystroke events. In human subject trials, conducted with twenty participants using a prototype system, participants entered text significantly faster with tactile feedback than without (p = 0.02). The results support incorporation of vibrotactile information in a future system that will enable full touch typing and general mouse interactions using instrumented EVA gloves

Biomechanical Texture Coding and Transmission of Texture Information in Rat Whiskers

Classically, texture discrimination has been thought to be based on ‘global’ codes, i.e. frequency (signal analysis based on Fourier analysis) or intensity (signal analysis based on averaging), which both rely on integration of the vibrotactile signal across time and/or space. Recently, a novel ‘local’ coding scheme based on the waveform of frictional movements, discrete short- lasting kinematic events (i.e. stick-slip movements called slips) has been formulated. In the first part of my study I performed biomechanical measurements of relative movements of a rat vibrissa across sandpapers of different roughness. My major finding is that the classic global codes convey some information about texture identity but are consistently outperformed by the slip-based local code. Moreover, the slip code also surpasses the global ones in coding for active scanning parameters. This is remarkable as it suggests that the slip code would explicitly allow the whisking rat to optimize perception by selecting goal-specific scanning strategies. I therefore provide evidence that short stick-slip events may contribute to the perceptual mechanism by which rodent vibrissa code surface roughness. In the second part, I studied the biomechanics of how such events are transmitted from tip to follicle where mechano-transduction occurs. For this purpose, ultra-fast videography recording of the entire beam of a plucked rat whisker rubbing across sandpaper was employed. I found that slip events are conveyed almost instantly from tip to follicle while amplifying moments by a factor of about 1000. From these results, I argue that the mechanics of the whisker serve as a passive amplification device that faithfully represents stick-slip events to the neuronal receptors. Using measures of correlation, I moreover found that amongst the kinematic 8 variables, acceleration portrays dynamic variables (forces) best. The time series of acceleration at the base of the whisker provided a fair proxy to the time series of forces (dynamical variables) acting on the whisker base. Acceleration measurements (easily done via videography) may therefore provide an access to at least the relative amplitude of forces. This may be important for future work in behaving animals, where dynamical variables are notoriously difficult to measure