Computational design and optimization of electro-physiological sensors

Electro-physiological sensing devices are becoming increasingly common in diverse applications. However, designing such sensors in compact form factors and for high-quality signal acquisition is a challenging task even for experts, is typically done using heuristics, and requires extensive training. Our work proposes a computational approach for designing multi-modal electro-physiological sensors. By employing an optimization-based approach alongside an integrated predictive model for multiple modalities, compact sensors can be created which offer an optimal trade-off between high signal quality and small device size. The task is assisted by a graphical tool that allows to easily specify design preferences and to visually analyze the generated designs in real-time, enabling designer-in-the-loop optimization. Experimental results show high quantitative agreement between the prediction of the optimizer and experimentally collected physiological data. They demonstrate that generated designs can achieve an optimal balance between the size of the sensor and its signal acquisition capability, outperforming expert generated solutions

PhysioSkin: Rapid Fabrication of Skin-Conformal Physiological Interfaces

Advances in rapid prototyping platforms have made physiological sensing accessible to a wide audience. However, off-the-shelf electrodes commonly used for capturing biosignals are typically thick, non-conformal and do not support customization. We present PhysioSkin, a rapid, do-it-yourself prototyping method for fabricating custom multi-modal physiological sensors, using commercial materials and a commodity desktop inkjet printer. It realizes ultrathin skin-conformal patches (~1μm) and interactive textiles that capture sEMG, EDA and ECG signals. It further supports fabricating devices with custom levels of thickness and stretchability. We present detailed fabrication explorations on multiple substrate materials, functional inks and skin adhesive materials. Informed from the literature, we also provide design recommendations for each of the modalities. Evaluation results show that the sensor patches achieve a high signal-to-noise ratio. Example applications demonstrate the functionality and versatility of our approach for prototyping a next generation of physiological devices that intimately couple with the human body

Psychophysiology in the digital age

The research I performed for my thesis revolved around the question how affect-physiology dynamics can be best measured in daily life. In my thesis I focused on three aspects of this question: 1) Do wearable wristband devices have sufficient validity to capture ANS activity? 2) To what extent is the laboratory design suitable to measure affect-ANS dynamics? 3) Are the affect-ANS dynamics subject to individual differences, both in the laboratory and in daily life? In chapter 2, I validated a shortened version of the Sing-a-Song Stress (SSST) test, the SSSTshort. The purpose of this test is to create social-evaluative stress in participants through a simple and brief design that does not require the involvement of multiple confederates. The results indicated that the SSSTshort was effective in inducing ANS and affective reactivity. This makes the SSSTshort a cost-effective alternative to the well-known Trier-Social-Stress task (TSST), which can be easily incorporated into large-scale studies to expand the range of stress types that can be studied in laboratory designs. In chapter 3, I validated a new wrist worn technology for measuring electrodermal activity (EDA). As expected, the overall EDA levels measured on the wrist were lower than those measured on the palm, likely due to the lower density of sweat glands on the wrist. The analysis demonstrated that the frequency measure of non-specific skin conductance response (ns.SCR) was superior to the commonly used measure of skin conductance level (SCL) for both the palm and wrist. The wrist-based ns.SCR measure was sensitive to the experimental manipulations and showed similar correspondence to the pre-ejection period (PEP) as palm-based ns.SCR. Moreover, wrist-based ns.SCR demonstrated similar predictive validity for affective state as PEP. However, the predictive validity of both wrist-based ns.SCR and PEP was lower compared to palm-based ns.SCR. These findings suggest that wrist-based ns.SCR EDA parameter has a promising future for use in psychophysiological research. In Chapter 4 of my thesis, I conducted the first study to directly compare the relationship between affect and ANS activity in a laboratory setting to that in daily life. To elicit stress in the laboratory, four different stress paradigms were employed, while stressful events in daily life were left to chance. In both settings, a valence and arousal scale was constructed from a nine-item affect questionnaire, and ANS activity was collected using the same devices. Data was collected from a single population, and the affect-ANS dynamics were analyzed using the same methodology for both laboratory and daily life settings. The results showed a remarkable similarity between the laboratory and daily life affect-ANS relationships. In Chapter 5 of my thesis, I investigated the influence of individual differences in physical activity and aerobic fitness on ANS and affective stress reactivity. Previous research has yielded inconsistent results due to heterogeneity issues in the population studied, stressor type, and the way fitness was measured. My study made a unique contribution to this field by measuring physical activity in three ways: 1) as objective aerobic fitness, 2) leisure time exercise behavior, and 3) total moderate-to-vigorous exercise (including both exercise and all other regular physical activity behaviors). In addition, we measured the physiological and affective stress response in both a laboratory and daily life setting. The total amount of physical activity showed more relationships with stress reactivity compared to exercise behavior alone, suggesting that future research should include a total physical activity variable. Our results did not support the cross-stressor adaptation hypotheses, suggesting that if exercise has a stress-reducing effect, it is unlikely to be mediated by altered ANS regulation due to repeated exposure to physical stress

Electrodermal Activity: Simultaneous Recordings

Electrodermal activity (EDA) is a sensitive measure of the sympathetic nervous system activity. It is used to describe changes in the skin electrical properties. This chapter aimed to show advantages of simultaneous recordings of EDA parameters at the same skin site over other recordings. The literature databases, Web of Science and Google Scholar, were searched using terms like “electrodermal activity,” “sequential recording,” “simultaneous recording,” “skin conductance,” “skin potential,” and “skin susceptance.” Articles that include sequential and/or simultaneous recording of EDA parameters were analyzed. The chapter presents a description of the oldest and current methods used for recording EDA parameters and an explanation of the newest techniques used in EDA researches. Although sequential recordings are predominant and widely spreading, much effort has been made to simultaneously record skin conductance (SC) and skin potential (SP), and recently researchers realized the capability of simultaneously recording SC, SP, and skin susceptance (SS) at the same skin site. The advantage of simultaneous over the sequence measurements is that the latter must be manually time realigned when measured by different instruments, which means it is time-consuming. Although the simultaneous measurements are used exclusively for research purposes at this stage, this may open horizons in the modern trends of psychophysiology applications in the near future

An Observational Study With the Janssen Autism Knowledge Engine (JAKE((R))) in Individuals With Autism Spectrum Disorder

Objective: The Janssen Autism Knowledge Engine (JAKE(R)) is a clinical research outcomes assessment system developed to more sensitively measure treatment outcomes and identify subpopulations in autism spectrum disorder (ASD). Here we describe JAKE and present results from its digital phenotyping (My JAKE) and biosensor (JAKE Sense) components. Methods: An observational, non-interventional, prospective study of JAKE in children and adults with ASD was conducted at nine sites in the United States. Feedback on JAKE usability was obtained from caregivers. JAKE Sense included electroencephalography, eye tracking, electrocardiography, electrodermal activity, facial affect analysis, and actigraphy. Caregivers of individuals with ASD reported behaviors using My JAKE. Results from My JAKE and JAKE Sense were compared to traditional ASD symptom measures. Results: Individuals with ASD (N = 144) and a cohort of typically developing (TD) individuals (N = 41) participated in JAKE Sense. Most caregivers reported that overall use and utility of My JAKE was easy (69%, 74/108) or very easy (74%, 80/108). My JAKE could detect differences in ASD symptoms as measured by traditional methods. The majority of biosensors included in JAKE Sense captured sizable amounts of quality data (i.e., 93-100% of eye tracker, facial affect analysis, and electrocardiogram data was of good quality), demonstrated differences between TD and ASD individuals, and correlated with ASD symptom scales. No significant safety events were reported. Conclusions: My JAKE was viewed as easy or very easy to use by caregivers participating in research outside of a clinical study. My JAKE sensitively measured a broad range of ASD symptoms. JAKE Sense biosensors were well-tolerated. JAKE functioned well when used at clinical sites previously inexperienced with some of the technologies. Lessons from the study will optimize JAKE for use in clinical trials to assess ASD interventions. Additionally, because biosensors were able to detect features differentiating TD and ASD individuals, and also were correlated with standardized symptom scales, these measures could be explored as potential biomarkers for ASD and as endpoints in future clinical studies. Clinical Trial Registration: https://clinicaltrials.gov/ct2/show/NCT02668991 identifier: NCT02668991

From wearable towards epidermal computing : soft wearable devices for rich interaction on the skin

Human skin provides a large, always available, and easy to access real-estate for interaction. Recent advances in new materials, electronics, and human-computer interaction have led to the emergence of electronic devices that reside directly on the user's skin. These conformal devices, referred to as Epidermal Devices, have mechanical properties compatible with human skin: they are very thin, often thinner than human hair; they elastically deform when the body is moving, and stretch with the user's skin. Firstly, this thesis provides a conceptual understanding of Epidermal Devices in the HCI literature. We compare and contrast them with other technical approaches that enable novel on-skin interactions. Then, through a multi-disciplinary analysis of Epidermal Devices, we identify the design goals and challenges that need to be addressed for advancing this emerging research area in HCI. Following this, our fundamental empirical research investigated how epidermal devices of different rigidity levels affect passive and active tactile perception. Generally, a correlation was found between the device rigidity and tactile sensitivity thresholds as well as roughness discrimination ability. Based on these findings, we derive design recommendations for realizing epidermal devices. Secondly, this thesis contributes novel Epidermal Devices that enable rich on-body interaction. SkinMarks contributes to the fabrication and design of novel Epidermal Devices that are highly skin-conformal and enable touch, squeeze, and bend sensing with co-located visual output. These devices can be deployed on highly challenging body locations, enabling novel interaction techniques and expanding the design space of on-body interaction. Multi-Touch Skin enables high-resolution multi-touch input on the body. We present the first non-rectangular and high-resolution multi-touch sensor overlays for use on skin and introduce a design tool that generates such sensors in custom shapes and sizes. Empirical results from two technical evaluations confirm that the sensor achieves a high signal-to-noise ratio on the body under various grounding conditions and has a high spatial accuracy even when subjected to strong deformations. Thirdly, Epidermal Devices are in contact with the skin, they offer opportunities for sensing rich physiological signals from the body. To leverage this unique property, this thesis presents rapid fabrication and computational design techniques for realizing Multi-Modal Epidermal Devices that can measure multiple physiological signals from the human body. Devices fabricated through these techniques can measure ECG (Electrocardiogram), EMG (Electromyogram), and EDA (Electro-Dermal Activity). We also contribute a computational design and optimization method based on underlying human anatomical models to create optimized device designs that provide an optimal trade-off between physiological signal acquisition capability and device size. The graphical tool allows for easily specifying design preferences and to visually analyze the generated designs in real-time, enabling designer-in-the-loop optimization. Experimental results show high quantitative agreement between the prediction of the optimizer and experimentally collected physiological data. Finally, taking a multi-disciplinary perspective, we outline the roadmap for future research in this area by highlighting the next important steps, opportunities, and challenges. Taken together, this thesis contributes towards a holistic understanding of Epidermal Devices}: it provides an empirical and conceptual understanding as well as technical insights through contributions in DIY (Do-It-Yourself), rapid fabrication, and computational design techniques.Die menschliche Haut bietet eine große, stets verfügbare und leicht zugängliche Fläche für Interaktion. Jüngste Fortschritte in den Bereichen Materialwissenschaft, Elektronik und Mensch-Computer-Interaktion (Human-Computer-Interaction, HCI) [so that you can later use the Englisch abbreviation] haben zur Entwicklung elektronischer Geräte geführt, die sich direkt auf der Haut des Benutzers befinden. Diese sogenannten Epidermisgeräte haben mechanische Eigenschaften, die mit der menschlichen Haut kompatibel sind: Sie sind sehr dünn, oft dünner als ein menschliches Haar; sie verformen sich elastisch, wenn sich der Körper bewegt, und dehnen sich mit der Haut des Benutzers. Diese Thesis bietet, erstens, ein konzeptionelles Verständnis von Epidermisgeräten in der HCI-Literatur. Wir vergleichen sie mit anderen technischen Ansätzen, die neuartige Interaktionen auf der Haut ermöglichen. Dann identifizieren wir durch eine multidisziplinäre Analyse von Epidermisgeräten die Designziele und Herausforderungen, die angegangen werden müssen, um diesen aufstrebenden Forschungsbereich voranzubringen. Im Anschluss daran untersuchten wir in unserer empirischen Grundlagenforschung, wie epidermale Geräte unterschiedlicher Steifigkeit die passive und aktive taktile Wahrnehmung beeinflussen. Im Allgemeinen wurde eine Korrelation zwischen der Steifigkeit des Geräts und den taktilen Empfindlichkeitsschwellen sowie der Fähigkeit zur Rauheitsunterscheidung festgestellt. Basierend auf diesen Ergebnissen leiten wir Designempfehlungen für die Realisierung epidermaler Geräte ab. Zweitens trägt diese Thesis zu neuartigen Epidermisgeräten bei, die eine reichhaltige Interaktion am Körper ermöglichen. SkinMarks trägt zur Herstellung und zum Design neuartiger Epidermisgeräte bei, die hochgradig an die Haut angepasst sind und Berührungs-, Quetsch- und Biegesensoren mit gleichzeitiger visueller Ausgabe ermöglichen. Diese Geräte können an sehr schwierigen Körperstellen eingesetzt werden, ermöglichen neuartige Interaktionstechniken und erweitern den Designraum für die Interaktion am Körper. Multi-Touch Skin ermöglicht hochauflösende Multi-Touch-Eingaben am Körper. Wir präsentieren die ersten nicht-rechteckigen und hochauflösenden Multi-Touch-Sensor-Overlays zur Verwendung auf der Haut und stellen ein Design-Tool vor, das solche Sensoren in benutzerdefinierten Formen und Größen erzeugt. Empirische Ergebnisse aus zwei technischen Evaluierungen bestätigen, dass der Sensor auf dem Körper unter verschiedenen Bedingungen ein hohes Signal-Rausch-Verhältnis erreicht und eine hohe räumliche Auflösung aufweist, selbst wenn er starken Verformungen ausgesetzt ist. Drittens, da Epidermisgeräte in Kontakt mit der Haut stehen, bieten sie die Möglichkeit, reichhaltige physiologische Signale des Körpers zu erfassen. Um diese einzigartige Eigenschaft zu nutzen, werden in dieser Arbeit Techniken zur schnellen Herstellung und zum computergestützten Design von multimodalen Epidermisgeräten vorgestellt, die mehrere physiologische Signale des menschlichen Körpers messen können. Die mit diesen Techniken hergestellten Geräte können EKG (Elektrokardiogramm), EMG (Elektromyogramm) und EDA (elektrodermale Aktivität) messen. Darüber hinaus stellen wir eine computergestützte Design- und Optimierungsmethode vor, die auf den zugrunde liegenden anatomischen Modellen des Menschen basiert, um optimierte Gerätedesigns zu erstellen. Diese Designs bieten einen optimalen Kompromiss zwischen der Fähigkeit zur Erfassung physiologischer Signale und der Größe des Geräts. Das grafische Tool ermöglicht die einfache Festlegung von Designpräferenzen und die visuelle Analyse der generierten Designs in Echtzeit, was eine Optimierung durch den Designer im laufenden Betrieb ermöglicht. Experimentelle Ergebnisse zeigen eine hohe quantitative Übereinstimmung zwischen den Vorhersagen des Optimierers und den experimentell erfassten physiologischen Daten. Schließlich skizzieren wir aus einer multidisziplinären Perspektive einen Fahrplan für zukünftige Forschung in diesem Bereich, indem wir die nächsten wichtigen Schritte, Möglichkeiten und Herausforderungen hervorheben. Insgesamt trägt diese Arbeit zu einem ganzheitlichen Verständnis von Epidermisgeräten bei: Sie liefert ein empirisches und konzeptionelles Verständnis sowie technische Einblicke durch Beiträge zu DIY (Do-It-Yourself), schneller Fertigung und computergestützten Entwurfstechniken

An Observational Study With the Janssen Autism Knowledge Engine (JAKE®) in Individuals With Autism Spectrum Disorder

Objective: The Janssen Autism Knowledge Engine (JAKE®) is a clinical research outcomes assessment system developed to more sensitively measure treatment outcomes and identify subpopulations in autism spectrum disorder (ASD). Here we describe JAKE and present results from its digital phenotyping (My JAKE) and biosensor (JAKE Sense) components.Methods: An observational, non-interventional, prospective study of JAKE in children and adults with ASD was conducted at nine sites in the United States. Feedback on JAKE usability was obtained from caregivers. JAKE Sense included electroencephalography, eye tracking, electrocardiography, electrodermal activity, facial affect analysis, and actigraphy. Caregivers of individuals with ASD reported behaviors using My JAKE. Results from My JAKE and JAKE Sense were compared to traditional ASD symptom measures.Results: Individuals with ASD (N = 144) and a cohort of typically developing (TD) individuals (N = 41) participated in JAKE Sense. Most caregivers reported that overall use and utility of My JAKE was “easy” (69%, 74/108) or “very easy” (74%, 80/108). My JAKE could detect differences in ASD symptoms as measured by traditional methods. The majority of biosensors included in JAKE Sense captured sizable amounts of quality data (i.e., 93–100% of eye tracker, facial affect analysis, and electrocardiogram data was of good quality), demonstrated differences between TD and ASD individuals, and correlated with ASD symptom scales. No significant safety events were reported.Conclusions: My JAKE was viewed as easy or very easy to use by caregivers participating in research outside of a clinical study. My JAKE sensitively measured a broad range of ASD symptoms. JAKE Sense biosensors were well-tolerated. JAKE functioned well when used at clinical sites previously inexperienced with some of the technologies. Lessons from the study will optimize JAKE for use in clinical trials to assess ASD interventions. Additionally, because biosensors were able to detect features differentiating TD and ASD individuals, and also were correlated with standardized symptom scales, these measures could be explored as potential biomarkers for ASD and as endpoints in future clinical studies.Clinical Trial Registration:https://clinicaltrials.gov/ct2/show/NCT02668991 identifier: NCT0266899

The Smartphone Brain Scanner: A Portable Real-Time Neuroimaging System

Combining low cost wireless EEG sensors with smartphones offers novel opportunities for mobile brain imaging in an everyday context. We present a framework for building multi-platform, portable EEG applications with real-time 3D source reconstruction. The system - Smartphone Brain Scanner - combines an off-the-shelf neuroheadset or EEG cap with a smartphone or tablet, and as such represents the first fully mobile system for real-time 3D EEG imaging. We discuss the benefits and challenges of a fully portable system, including technical limitations as well as real-time reconstruction of 3D images of brain activity. We present examples of the brain activity captured in a simple experiment involving imagined finger tapping, showing that the acquired signal in a relevant brain region is similar to that obtained with standard EEG lab equipment. Although the quality of the signal in a mobile solution using a off-the-shelf consumer neuroheadset is lower compared to that obtained using high density standard EEG equipment, we propose that mobile application development may offset the disadvantages and provide completely new opportunities for neuroimaging in natural settings