Multi-Channel Neural Network for Assessing Neonatal Pain from Videos

Neonates do not have the ability to either articulate pain or communicate it non-verbally by pointing. The current clinical standard for assessing neonatal pain is intermittent and highly subjective. This discontinuity and subjectivity can lead to inconsistent assessment, and therefore, inadequate treatment. In this paper, we propose a multi-channel deep learning framework for assessing neonatal pain from videos. The proposed framework integrates information from two pain indicators or channels, namely facial expression and body movement, using convolutional neural network (CNN). It also integrates temporal information using a recurrent neural network (LSTM). The experimental results prove the efficiency and superiority of the proposed temporal and multi-channel framework as compared to existing similar methods.Comment: Accepted to IEEE SMC 201

Personalized Pain Study Platform Using Evidence-Based Continuous Learning Tool

With the increased accessibility to mobile technologies, research utilizing mobile technologies in medical and public health area has also increased. The efficiency and effectiveness of healthcare services are also improved by introduction of mobile technologies. Effective pain treatment requires regular and continuous pain assessment of the patients. Mobile Health or mHealth has been an active interdisciplinary research area for more than a decade to research pain assessment through different software research tools. Different mHealth support systems are developed to assess pain level of patient using different techniques. Close attention to participant’s self- reported pain along with data mining based pain level detection could help the healthcare industry and researchers to deliver effective health services in pain treatment. Pain expression recognition can be a good way for data mining based approach though pain expression recognition itself may utilize different approach based on the research study scope. Most of the pain research tools are study or disease specific. Some of the tools are pain specific (lumber pain, cancer pain etc) and some are patient group specific (neonatal, adult, woman etc). This results in recurrent but potentially avoidable costs such as time, money, and workforce to develop similar service or software research tools for each research study. Based on the pain study research characteristics, it is possible to design and implement a customizable and extensible generic pain research tool. In this thesis, we have proposed, designed, and implemented a customizable personalized pain study platform tool following a micro service architecture. It has most of the common software research modules that are needed for a pain research study. These include real-time data collection, research participant management, role based access control, research data anonymization etc. This software research tool is also used to investigate pain level detection accuracy using evidence-based continuous learning from facial expression which yielded about 71% classification accuracy. This tool is also HIPAA compliant and platform independent which makes it device independent, privacy-aware, and security-aware

Vedel-objektiiv abil salvestatud kaugseire piltide analüüs kasutades super-resolutsiooni meetodeid

Väitekirja elektrooniline versioon ei sisalda publikatsiooneKäesolevas doktoritöös uuriti nii riist- kui ka tarkvaralisi lahendusi piltide töötlemiseks. Riist¬varalise poole pealt pakuti lahenduseks uudset vedelläätse, milles on dielekt¬rilisest elastomeerist kihilise täituriga membraan otse optilisel teljel. Doktoritöö käigus arendati välja kaks prototüüpi kahe erineva dielektrilisest elastomeerist ki¬hilise täituriga, mille aktiivne ala oli ühel juhul 40 ja teisel 20 mm. Läätse töö vas¬tas elastomeeri deformatsiooni mehaanikale ja suhtelistele muutustele fookuskau¬guses. Muutuste demonstreerimiseks meniskis ja läätse fookuskauguse mõõtmiseks kasutati laserkiirt. Katseandmetest selgub, et muutuste tekitamiseks on vajalik pinge vahemikus 50 kuni 750 volti. Tarkvaralise poole pealt pakuti uut satelliitpiltide parandamise süsteemi. Paku¬tud süsteem jagas mürase sisendpildi DT-CWT laineteisenduse abil mitmeteks sagedusalamribadeks. Pärast müra eemaldamist LA-BSF funktsiooni abil suu¬rendati pildi resolutsiooni DWT-ga ja kõrgsagedusliku alamriba piltide interpo¬leerimisega. Interpoleerimise faktor algsele pildile oli pool sellest, mida kasutati kõrgsagedusliku alamriba piltide interpoleerimisel ning superresolutsiooniga pilt rekonst¬rueeriti IDWT abil. Käesolevas doktoritöös pakuti tarkvaraliseks lahenduseks uudset sõnastiku baasil töötavat super-resolutsiooni (SR) meetodit, milles luuakse paarid suure resolutsiooniga (HR) ja madala resolut-siooniga (LR) piltidest. Kõigepealt jagati vastava sõnastiku loomiseks HR ja LR paarid omakorda osadeks. Esialgse HR kujutise saamiseks LR sisendpildist kombineeriti HR osi. HR osad valiti sõnastikust nii, et neile vastavad LR osad oleksid võimalikult lähedased sisendiks olevale LR pil¬dile. Iga valitud HR osa heledust korrigeeriti, et vähendada kõrvuti asuvate osade heleduse erine¬vusi superresolutsiooniga pildil. Plokkide efekti vähendamiseks ar¬vutati saadud SR pildi keskmine ning bikuupinterpolatsiooni pilt. Lisaks pakuti käesolevas doktoritöös välja kernelid, mille tulemusel on võimalik saadud SR pilte teravamaks muuta. Pakutud kernelite tõhususe tõestamiseks kasutati [83] ja [50] poolt pakutud resolutsiooni parandamise meetodeid. Superreso¬lutsiooniga pilt saadi iga kerneli tehtud HR pildi kombineerimise teel alpha blen¬dingu meetodit kasutades. Pakutud meetodeid ja kerneleid võrreldi erinevate tavaliste ja kaasaegsete meetoditega. Kvantita-tiivsetest katseandmetest ja saadud piltide kvaliteedi visuaal¬sest hindamisest selgus, et pakutud meetodid on tavaliste kaasaegsete meetoditega võrreldes paremad.In this thesis, a study of both hardware and software solutions for image enhance¬ment has been done. On the hardware side, a new liquid lens design with a DESA membrane located directly in the optical path has been demonstrated. Two pro¬totypes with two different DESA, which have a 40 and 20 mm active area in diameter, were developed. The lens performance was consistent with the mechan¬ics of elastomer deformation and relative focal length changes. A laser beam was used to show the change in the meniscus and to measure the focal length of the lens. The experimental results demonstrate that voltage in the range of 50 to 750 V is required to create change in the meniscus. On the software side, a new satellite image enhancement system was proposed. The proposed technique decomposed the noisy input image into various frequency subbands by using DT-CWT. After removing the noise by applying the LA-BSF technique, its resolution was enhanced by employing DWT and interpolating the high-frequency subband images. An original image was interpolated with half of the interpolation factor used for interpolating the high-frequency subband images, and the super-resolved image was reconstructed by using IDWT. A novel single-image SR method based on a generating dictionary from pairs of HR and their corresponding LR images was proposed. Firstly, HR and LR pairs were divided into patches in order to make HR and LR dictionaries respectively. The initial HR representation of an input LR image was calculated by combining the HR patches. These HR patches are chosen from the HR dictionary corre-sponding to the LR patches that have the closest distance to the patches of the in¬put LR image. Each selected HR patch was processed further by passing through an illumination enhancement processing order to reduce the noticeable change of illumination between neighbor patches in the super-resolved image. In order to reduce the blocking effect, the average of the obtained SR image and the bicubic interpolated image was calculated. The new kernels for sampling have also been proposed. The kernels can improve the SR by resulting in a sharper image. In order to demonstrate the effectiveness of the proposed kernels, the techniques from [83] and [50] for resolution enhance¬ment were adopted. The super-resolved image was achieved by combining the HR images produced by each of the proposed kernels using the alpha blending tech-nique. The proposed techniques and kernels are compared with various conventional and state-of-the-art techniques, and the quantitative test results and visual results on the final image quality show the superiority of the proposed techniques and ker¬nels over conventional and state-of-art technique

Chronic-Pain Protective Behavior Detection with Deep Learning

In chronic pain rehabilitation, physiotherapists adapt physical activity to patients' performance based on their expression of protective behavior, gradually exposing them to feared but harmless and essential everyday activities. As rehabilitation moves outside the clinic, technology should automatically detect such behavior to provide similar support. Previous works have shown the feasibility of automatic protective behavior detection (PBD) within a specific activity. In this paper, we investigate the use of deep learning for PBD across activity types, using wearable motion capture and surface electromyography data collected from healthy participants and people with chronic pain. We approach the problem by continuously detecting protective behavior within an activity rather than estimating its overall presence. The best performance reaches mean F1 score of 0.82 with leave-one-subject-out cross validation. When protective behavior is modelled per activity type, performance is mean F1 score of 0.77 for bend-down, 0.81 for one-leg-stand, 0.72 for sit-to-stand, 0.83 for stand-to-sit, and 0.67 for reach-forward. This performance reaches excellent level of agreement with the average experts' rating performance suggesting potential for personalized chronic pain management at home. We analyze various parameters characterizing our approach to understand how the results could generalize to other PBD datasets and different levels of ground truth granularity.Comment: 24 pages, 12 figures, 7 tables. Accepted by ACM Transactions on Computing for Healthcar

Multimodaalsel emotsioonide tuvastamisel põhineva inimese-roboti suhtluse arendamine

Väitekirja elektrooniline versioon ei sisalda publikatsiooneÜks afektiivse arvutiteaduse peamistest huviobjektidest on mitmemodaalne emotsioonituvastus, mis leiab rakendust peamiselt inimese-arvuti interaktsioonis. Emotsiooni äratundmiseks uuritakse nendes süsteemides nii inimese näoilmeid kui kakõnet. Käesolevas töös uuritakse inimese emotsioonide ja nende avaldumise visuaalseid ja akustilisi tunnuseid, et töötada välja automaatne multimodaalne emotsioonituvastussüsteem. Kõnest arvutatakse mel-sageduse kepstri kordajad, helisignaali erinevate komponentide energiad ja prosoodilised näitajad. Näoilmeteanalüüsimiseks kasutatakse kahte erinevat strateegiat. Esiteks arvutatakse inimesenäo tähtsamate punktide vahelised erinevad geomeetrilised suhted. Teiseks võetakse emotsionaalse sisuga video kokku vähendatud hulgaks põhikaadriteks, misantakse sisendiks konvolutsioonilisele tehisnärvivõrgule emotsioonide visuaalsekseristamiseks. Kolme klassifitseerija väljunditest (1 akustiline, 2 visuaalset) koostatakse uus kogum tunnuseid, mida kasutatakse õppimiseks süsteemi viimasesetapis. Loodud süsteemi katsetati SAVEE, Poola ja Serbia emotsionaalse kõneandmebaaside, eNTERFACE’05 ja RML andmebaaside peal. Saadud tulemusednäitavad, et võrreldes olemasolevatega võimaldab käesoleva töö raames loodudsüsteem suuremat täpsust emotsioonide äratundmisel. Lisaks anname käesolevastöös ülevaate kirjanduses väljapakutud süsteemidest, millel on võimekus tunda äraemotsiooniga seotud ̆zeste. Selle ülevaate eesmärgiks on hõlbustada uute uurimissuundade leidmist, mis aitaksid lisada töö raames loodud süsteemile ̆zestipõhiseemotsioonituvastuse võimekuse, et veelgi enam tõsta süsteemi emotsioonide äratundmise täpsust.Automatic multimodal emotion recognition is a fundamental subject of interest in affective computing. Its main applications are in human-computer interaction. The systems developed for the foregoing purpose consider combinations of different modalities, based on vocal and visual cues. This thesis takes the foregoing modalities into account, in order to develop an automatic multimodal emotion recognition system. More specifically, it takes advantage of the information extracted from speech and face signals. From speech signals, Mel-frequency cepstral coefficients, filter-bank energies and prosodic features are extracted. Moreover, two different strategies are considered for analyzing the facial data. First, facial landmarks' geometric relations, i.e. distances and angles, are computed. Second, we summarize each emotional video into a reduced set of key-frames. Then they are taught to visually discriminate between the emotions. In order to do so, a convolutional neural network is applied to the key-frames summarizing the videos. Afterward, the output confidence values of all the classifiers from both of the modalities are used to define a new feature space. Lastly, the latter values are learned for the final emotion label prediction, in a late fusion. The experiments are conducted on the SAVEE, Polish, Serbian, eNTERFACE'05 and RML datasets. The results show significant performance improvements by the proposed system in comparison to the existing alternatives, defining the current state-of-the-art on all the datasets. Additionally, we provide a review of emotional body gesture recognition systems proposed in the literature. The aim of the foregoing part is to help figure out possible future research directions for enhancing the performance of the proposed system. More clearly, we imply that incorporating data representing gestures, which constitute another major component of the visual modality, can result in a more efficient framework