Beigl, Michael

Schneegans, Tim

Simmon, Leon

English

KITopen

Privacy Preserving Continuous Speech Recording using ThroatMicrophonesTim Schneegansschneegans@teco.eduKarlsruhe Institute of TechnologyKarlsruhe, GermanyLeon Simmonsimmon@teco.eduKarlsruhe Institute of TechnologyKarlsruhe, GermanyMichael Beiglbeigl@teco.eduKarlsruhe Institute of TechnologyKarlsruhe, GermanyFigure 1: A) Functional Scheme, B) Prototype Device, C) Device worn on a person. ACM=Air-Conduction Mic., TM=Throat Mic.ABSTRACTA prerequisite for field-research on audio data are privacy preserv-ing recordings that exclusively contain the target speaker who gaveconsent. For this purpose, we investigated the potential of a simplebut robust wearable technology consisting of three parts: first, astandard air-conduction microphone provides the necessary audioquality for speech analysis; second, a throat microphone is used as aspeech activity filter; third, a custom ESP32 based recording deviceenables on-device real-time processing. The system was evaluatedin two challenging free discussion settings with two and four par-ticipants each (total N=16). Results from manual annotations showan Equal Error Rate of M=23.4-29.69 %. Based on simple instruc-tions, our participants managed to maintain a False AcceptanceRate below 5 % while recording more than half of their utterances.CCS CONCEPTS• Applied computing; • Law, social and behavioral sciences;KEYWORDSspeech processing; audio sensors; data privacy protection; Researchon behaviour, psychology, and cognition; Biometrics and securityACM Reference Format:Tim Schneegans, Leon Simmon, and Michael Beigl. 2022. Privacy Preserv-ing Continuous Speech Recording using Throat Microphones. In The 2022International Symposium on Wearable Computers (ISWC ’22), September 11–15, 2022, Cambridge, United Kingdom. ACM, New York, NY, USA, 3 pages.https://doi.org/10.1145/3544794.35584801 INTRODUCTIONSpeech is a major data source for understanding human behavior.Vocal features relate to psychiatric disorders, such as depression,ISWC ’22, September 11–15, 2022, Cambridge, United Kingdom© 2022 Copyright held by the owner/author(s).This is the author’s version of the work. It is posted here for your personal use. Not forredistribution. The definitive Version of Record was published in The 2022 InternationalSymposium on Wearable Computers (ISWC ’22), September 11–15, 2022, Cambridge,United Kingdom, https://doi.org/10.1145/3544794.3558480.schizophrenia, and bipolar disorder [16]. Everyday word use relatesto personality traits [18] and cognitive ageing processes [9]. Wear-able sensors help researchers to access such data [17, 25]. Aside ofcollecting data, speech is used for providing user-adaptive feedback,for example in voice assistants [11, 14].A prerequisite for everyday audio recordings is to separate targetspeakers who actively agreed to data collection from non-targetspeakers who did not agree. Recording latter is legally prohibitedin the EU [4, 5, 26]. Typically, an everyday life audio stream con-tains social situations with a mix of both of these groups. Mehlet al. [17, 19, 20] achieved approval from ethic committees for theUS only. Their system collects random speech samples from 5 %of the participant’s daytime, which suffices for examining certainresearch questions, such as social interactions [9, 18].Throat microphones (TMs) offer a computationally efficient wayfor detecting the target speaker. TMs use a piezoelectric sensor torecord speech via vibrations in the human body. This mechanismdecrements any external sound from non-target speakers. This isnot possible with air-conduction microphones (ACMs) even if theyare placed near the user. ACM signals can be filtered using SpeakerRecognition and Diarization systems [8, 24, 27, 29], which in prac-tice often require cloud computing and raise additional privacyissues. Compared to earbuds with active noise cancelling [2], TMsallow ear-free interactions. However, TMs do not meet the dataquality requirements for extensive speech analysis (but see [21, 22]).Our contributions are of two kind: First, we propose a wearablesystem for privacy preserving and continuous audio recordings.For that purpose, we combine the filtering capability of TMs withthe high audio quality of ACMs. Previous TM-ACM combinationshad the purpose of reducing random background noise but notspeech from non-target speakers [6, 28, 30]. Second, we evaluatedour filtering principle in a group study in two challenging freediscussion settings. While not recording any non-target speakers,we aim to collect more data from the target speaker than currentsampling methods (e.g., 5 % [17]).ISWC ’22, September 11–15, 2022, Cambridge, United Kingdom Schneegans, et al.2 PROTOTYPEWe propose a combination of off-the-shelf ACM and TM that areprocessed in real-time using an ESP board (Figure 1). The user (tar-get speaker) chooses a decibel-value for the TM signal that worksas a threshold value passing or muting the synchronous ACM sig-nal (Figure 1). For the calibration, the user is instructed to movetheir head, speak with a moderate volume, watch an LED that in-dicates the recording state, and set their personal threshold. Bothmicrophones were plugged into an ESP32 LyraT v4.3 board, whichintegrates several audio processing components (e.g., Audio CodecChip, control buttons). We replaced the built-in stereo microphoneswith 4-pin audio ports each in order to connect the ACM (left chan-nel) and TM (right channel). We implemented additional softwarefeatures to improve performance: a Schmitt-Trigger model for thethreshold to prevent unstable trigger behaviour; an exponentialsmooth volume control for the mute/unmute function to removecrackling in the recordings; and a time based unique file nameauto-save file writer for stable data storage.3 EVALUATIONProcedure. We recorded audio data from four groups of fourGerman-speaking participants each (N=16; n=4 female) from a sam-ple of convenience. Most participants (n=14) were 18-25 years old(n=2 were 25-30 years) and had an alto voice (n=8; n=3 bariton/bass,n=4 tenor, n=1 sopran). In a balanced Within-Subject-Design theysat on a table and played a collaborative game (www.keeptalking-game.com) in two conditions à 20 min: (i) in the entire group and (ii)in a dialogue. A ground truth audio was established by passing theunfiltered ACM signal through the ESP32 board to a smartphone.The study was preregistered on aspredicted.org (#89869). Prior todata collection, a data protection officer was consulted.Preprocessing. Speaker segments and identification were man-ually annotated. The annotations were framed into non-overlappingwindows of 25 ms each. Because human raters annotate with differ-ent levels of precision, labels within 0.25 s after a label change wereexcluded (15.9 %). Relabeling 13 files (20.3 %) showed an averageInter-Rater-Agreement (IRR) of Kappa=82.2 % (SD=15.5 %) [3], afterremoving one outlier file. Due to high annotation effort, only thefirst three minutes of each participant and condition were analyzed.Across participants, this results in 72 min of filtered and 72 minof ground truth annotated audio material. The users spoke forM=1.97 s (SD=0.78 s) in the filtered and M=2.23 s (SD=0.95 s) in theground truth stream. User and other speakers spoke simultaneouslyfor M=4.73 s (2.62 %) (SD=7.44 s; 4.12 %).Metrics. Three standard performance metrics were calculated[15, 23]: The (i) False Acceptance Rate (FAR) is the error rate ofrecording a non-target speaker. We consider the FAR as the privacylevel. The (ii) False Rejection Rate (FRR) is the error rate of rejectinga target speaker, i.e., the user. The (iii) Equal Error Rate is theminimum error rate when FAR and FRR are equal. These metricswere analyzed for the real-time filtered audio stream (RT) based onthe participant’s calibration and a post-hoc analysis (PH) for a rangeof threshold values based on the ground truth audio. Table 1 showsthe FAR and FRR for the self-calibrated threshold values (M=51.8 db,SD=4.2 db) based on RT and PH analysis (also see Figure 2). Onaverage, the participants managed to set a FAR of ca. 5 % or lower.Results. Overall, the system showed better performance scoresin the dialogue than the group condition. Figure 2 shows FAR andFRR for the Post-Hoc analysis and also the tradeoff between securityfor non-target speakers in speech recordings and coverage of thetarget speakers utterances.Table 1: FAR and FRR [ %] based on participant calibration.Group DialogueM SD Range M SD RangeFARRT 4.03 4.56 [0-17.4] 1.8 3.12 [0-12.7]FARPH 7.0 7.19 [0.7-25.5] 6.85 9.35 [0-33.8]FRRRT 46.76 38.16 [0.8-100] 33.42 30.18 [2.8-97.0]FRRPH 79.54 18.51 [42.7-100] 74.38 18.95 [41.9-99.7]Figure 2: Mean and Standard Deviations for FAR and FRRacross thresholds. Vertical lines highlight thresholds withFAR values of 2.5 %, 5 %, and 10 %.EERwasM=29.69 % (SD=17.23 %; group) andM=23.4 % (SD=13.2 %;dialogue). As a benchmark, wemodified an i-vector and PLDA basedSpeech Recognizer [7, 12] to analyze 400 ms frames (approx. twoEnglish syllables [1]) in real-time, which performed on EER=22.19 %(dialogue) to 34.57 % (group). By combining both systems wherei-Vectors are applied to pre-filtered audio from the TM, we couldimprove the EER to 20.70 % (dialogue) and 26.37 % (group).The users showed a moderate comfort score on an adapted ver-sion of the comfort rating scale (M=1.9, SD=0.5, range=1.1-2.9) [13].4 CONCLUSIONWe introduced a wearable speech recording system that provideshigh quality data from the user while protecting the data privacy ofnon-target speakers. Based on simple instructions, our participantscalibrated a small FAR<5 % while recording more than half of theirspeech. Although our system has a high FRR, it captures moredata than current sampling methods [17]. Our study limitationsleave room for further evaluation on (i) a larger sample with (ii)freely moving participants and (iii) other conversation scenarioswith longer utterances. What remains a challenge is that non-targetspeakers can be recorded if they speak at the same time as the targetspeaker. We plan to address this by further augmenting the audiosignals with embedded machine learning (e.g., [10]). Moreover,we plan to improve the user comfort of the TM. Finally, a perfectspeaker filtering system does not guarantee perfect privacy if thetarget speaker’s data contain personal details of their conversationalpartners. Future research should examine whether and to whatextent non-target speakers can be identified in such a scenario.Privacy Preserving Continuous Speech Recording using Throat Microphones ISWC ’22, September 11–15, 2022, Cambridge, United KingdomACKNOWLEDGMENTSFunded by the Deutsche Forschungsgemeinschaft (DFG, GermanResearch Foundation) – GRK2739/1 – Project Nr. 447089431 – Re-search Training Group: KD2School – Designing Adaptive Systemsfor Economic DecisionsREFERENCES[1] Melissa M Baese-Berk and Tuuli H Morrill. 2015. Speaking rate consistency innative and non-native speakers of English. The Journal of the Acoustical Societyof America 138, 3 (2015), EL223–EL228.[2] Ishan Chatterjee, Maruchi Kim, Vivek Jayaram, Shyamnath Gollakota, Ira Kemel-macher, Shwetak Patel, and Steven M Seitz. 2022. ClearBuds: wireless binauralearbuds for learning-based speech enhancement. In Proceedings of the 20th AnnualInternational Conference on Mobile Systems, Applications and Services. 384–396.[3] Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educationaland psychological measurement 20, 1 (1960), 37–46.[4] U.S. Congress. 2002. 18 USC 2511: Interception and disclosure of wire, oral, orelectronic communications prohibited. https://uscode.house.gov/view.xhtml?req=%28title:18%20section:2511%29. Accessed: 2022-07-09.[5] Council of Europe. 2018. 128th Session of the Committee of Ministers (Elsinore,Denmark, 17-18 May 2018). Modernised Convention for the Protection ofIndividuals with Regard to the Processing of Personal Data – Consolidated text.https://search.coe.int/cm/Pages/result_details.aspx?ObjectId=09000016807c65bf.[6] Engin Erzin. 2009. Improving throat microphone speech recognition by jointanalysis of throat and acoustic microphone recordings. IEEE transactions onaudio, speech, and language processing 17, 7 (2009), 1316–1324.[7] Daniel Garcia-Romero and Carol Y Espy-Wilson. 2011. Analysis of i-vector lengthnormalization in speaker recognition systems. In Twelfth annual conference ofthe international speech communication association.[8] Aleksei Gusev, Vladimir Volokhov, Alisa Vinogradova, Tseren Andzhukaev, An-drey Shulipa, Sergey Novoselov, Timur Pekhovsky, and Alexander Kozlov. 2020.STC-Innovation Speaker Recognition Systems for Far-Field Speaker VerificationChallenge 2020.. In INTERSPEECH. 3466–3470.[9] Maximilian Haas, Matthias R Mehl, Nicola Ballhausen, Sascha Zuber, MatthiasKliegel, and Alexandra Hering. 2022. The Sounds of Memory: Extending theAge–Prospective Memory Paradox to Everyday Behavior and Conversations. TheJournals of Gerontology: Series B 77, 4 (01 2022), 695–703. https://doi.org/10.1093/geronb/gbac012 arXiv:https://academic.oup.com/psychsocgerontology/article-pdf/77/4/695/43224411/gbac012.pdf[10] Simon Haykin and Zhe Chen. 2005. The cocktail party problem. Neural compu-tation 17, 9 (2005), 1875–1902.[11] Shin Katayama, Akhil Mathur, Marc Van den Broeck, Tadashi Okoshi, JinNakazawa, and Fahim Kawsar. 2019. Situation-Aware Emotion Regulation ofConversational Agents with Kinetic Earables. In 2019 8th International Conferenceon Affective Computing and Intelligent Interaction (ACII). IEEE, 725–731.[12] Patrick Kenny. 2010. Bayesian speaker verification with, heavy tailed priors. Proc.Odyssey 2010 (2010).[13] James F Knight and Chris Baber. 2005. A tool to assess the comfort of wearablecomputers. Human factors 47, 1 (2005), 77–91.[14] Effie Lai-Chong Law, Asbjørn Følstad, Jonathan Grudin, and Björn Schuller. 2022.Conversational Agent as Trustworthy Autonomous System (Trust-CA)(DagstuhlSeminar 21381). In Dagstuhl Reports, Vol. 11. Schloss Dagstuhl-Leibniz-Zentrumfür Informatik.[15] David A van Leeuwen and Niko Brümmer. 2007. An introduction to application-independent evaluation of speaker recognition systems. In Speaker classificationI. Springer, 330–353.[16] Daniel M Low, Kate H Bentley, and Satrajit S Ghosh. 2020. Automated assess-ment of psychiatric disorders using speech: A systematic review. LaryngoscopeInvestigative Otolaryngology 5, 1 (2020), 96–116.[17] Matthias R Mehl. 2017. The electronically activated recorder (EAR) a methodfor the naturalistic observation of daily social behavior. Current directions inpsychological science 26, 2 (2017), 184–190.[18] Matthias R Mehl, Samuel D Gosling, and James W Pennebaker. 2006. Personalityin its natural habitat: manifestations and implicit folk theories of personality indaily life. Journal of personality and social psychology 90, 5 (2006), 862.[19] Matthias R Mehl, James W Pennebaker, D Michael Crow, James Dabbs, andJohn H Price. 2001. The Electronically Activated Recorder (EAR): A devicefor sampling naturalistic daily activities and conversations. Behavior researchmethods, instruments, & computers 33, 4 (2001), 517–523.[20] Matthias R Mehl, Megan L Robbins, and Fenne große Deters. 2012. Naturalisticobservation of health-relevant social processes: The Electronically ActivatedRecorder (EAR) methodology in psychosomatics. Psychosomatic medicine 74, 4(2012), 410.[21] Yoshitaka Nakajima, Hideki Kashioka, Nick Campbell, and Kiyohiro Shikano.2006. Non-audible murmur (NAM) recognition. IEICE TRANSACTIONS onInformation and Systems 89, 1 (2006), 1–8.[22] YoshitakaNakajima, Hideki Kashioka, Kiyohiro Shikano, andNick Campbell. 2003.Non-audible murmur recognition input interface using stethoscopic microphoneattached to the skin. In 2003 IEEE International Conference on Acoustics, Speech,and Signal Processing, 2003. Proceedings.(ICASSP’03)., Vol. 5. IEEE, V–708.[23] National Institute of Standards and Technology. 2021. NIST 2021 Speaker Recog-nition Evaluation Plan. https://sre.nist.gov/[24] Wei Rao, Chenglin Xu, Eng Siong Chng, and Haizhou Li. 2019. Target speakerextraction for overlapped multi-talker speaker verification. arXiv preprintarXiv:1902.02546 (2019).[25] Timothy J Trull and Ulrich Ebner-Priemer. 2013. Ambulatory assessment. Annualreview of clinical psychology 9 (2013), 151.[26] European Union. 2016. Art. 6.1.a, Regulation (EU) 2016/679 of the EuropeanParliament and of the Council of 27 April 2016 on the protection of natural personswith regard to the processing of personal data and on the free movement of suchdata, and repealing Directive 95/46/EC (General Data Protection Regulation).[27] Chenglin Xu, Wei Rao, Eng Siong Chng, and Haizhou Li. 2019. Optimization ofspeaker extraction neural network with magnitude and temporal spectrum ap-proximation loss. In ICASSP 2019-2019 IEEE International Conference on Acoustics,Speech and Signal Processing (ICASSP). IEEE, 6990–6994.[28] Zhengyou Zhang, Zicheng Liu, Mike Sinclair, Alex Acero, Li Deng, Jasha Droppo,Xuedong Huang, and Yanli Zheng. 2004. Multi-sensory microphones for ro-bust speech detection, enhancement and recognition. In 2004 IEEE InternationalConference on Acoustics, Speech, and Signal Processing, Vol. 3. IEEE, iii–781.[29] Siqi Zheng, Weilong Huang, Xianliang Wang, Hongbin Suo, Jinwei Feng, andZhijie Yan. 2021. A real-time speaker diarization system based on spatial spectrum.In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and SignalProcessing (ICASSP). IEEE, 7208–7212.[30] Yanli Zheng, Zicheng Liu, Zhengyou Zhang, Mike Sinclair, Jasha Droppo, Li Deng,Alex Acero, and Xuedong Huang. 2003. Air-and bone-conductive integratedmicrophones for robust speech detection and enhancement. In 2003 IEEE Work-shop on Automatic Speech Recognition and Understanding (IEEE Cat. No. 03EX721).IEEE, 249–254.

Privacy Preserving Continuous Speech Recording using Throat Microphones

https://publikationen.bibliothek.kit.edu/1000150092/149192795

Privacy Preserving Continuous Speech Recording using Throat Microphones

Abstract

Similar works

Full text

Available Versions

KITopen