Angeler, David

Angeler, David G.

Berk, Michael

Eyre, Harris A.

English

Epsilon Open Archive

Bipolar Disorders. 2022;24:689–692.   | 689wileyonlinelibrary.com/journal/bdiDOI: 10.1111/bdi.13210  C L I N I C A L  C U R I O S I T YListening to the silent struggles of bipolar disorder through sonification of iMoodJournal dataDavid G. Angeler1,2,3,4  |   Harris A. Eyre5,6,7,8 |   Michael Berk3,9,10,111Department of Aquatic Sciences and Assessment, Swedish University of Agricultural Sciences, Uppsala, Sweden2The PRODEO Institute, San Francisco, California, USA3IMPACT, the Institute for Mental and Physical Health and Clinical Translation, Deakin University, Geelong, Victoria, Australia4University of Nebraska – Lincoln, School of Natural Resources, Lincoln, Nebraska, USA5Global Brain Health Institute at University of California, San Francisco (UCSF), San Francisco, California, USA6Trinity College Dublin, Dublin, Ireland7Department of Psychiatry and Behavioral Sciences, Baylor College of Medicine, Houston, Texas, USA8Neuroscience- inspired Policy Initiative, Organisation for Economic Co- Operation and Development (OECD) and the PRODEO Institute and Meadows Mental Health Policy Institute, Paris, France9Department of Psychiatry, University of Melbourne, Melbourne, Victoria, Australia10Orygen Youth Health, University of Melbourne, Melbourne, Victoria, Australia11The Florey Institute for Neuroscience and Mental Health, University of Melbourne, Melbourne, Victoria, AustraliaCorrespondenceDavid G. Angeler, Swedish University of Agricultural Sciences, Department of Aquatic Sciences and Assessment, Box 7050, 750 07 Uppsala, Sweden.Email: david.angeler@slu.seFunding informationNHMRC, Grant/Award Number: 11560721  |  C A SE PRESENTATIONThis paper reports a preliminary case study for demonstrating the potential of data sonification for telling a real- life narrative of ex-perienced mood swings through music. We obtained iMoodJournal data (2017– 2020) from a voluntarily participating male who was di-agnosed with type 2 bipolar disorder in 2015. The monitored pe-riod covers prolonged stretches of severe depression, particularly during fall, winter, and spring months. These “winter depressions” were usually superseded by remission during summer. These sea-sonal patterns were similar and recurring in the period between 2017 and 2019. In 2020, the depressions were relatively mild due to the patient spending winter in southern latitudes. However, another severe depression episode occurred during summer 2020 instead, which likely emanated from a period of medication discontinuation. The symptomatology was overall complex and highly dynamic, man-ifested in the combination of mood specifying tags that the user associated with determined mood scores in the iMoodJournal. This complexity was difficult to capture in the form of the numerical scores visualized in Figures S1 and S2.The patient considered the time series based on the weekly lowest scores to present the most suitably depression. These data were thus used for sonification. To facilitate the perception of mood states and trade off the underestimation of depression in the son-ified data, we used different dynamics (from piano (p) [periods of remission] to triple forte (fff) [severe depression]) and crescendos and decrescendos for transitions between these states. Also, a hy-pomanic mood state occurred once ephemerally in the sonified data; this was highlighted with an accidental (marcato).Two complementary scores were written (Data S1). The first score is based on representing years sequentially to familiarize the audience with the sound structure inherent in the data. The second score focused on the inter- annual variability in symptoms expres-sion which can be substantial in bipolar disorder and other mental health conditions. In this score, the individual yearly scores from the first composition were overlayed, in accordance with the visual This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and distribution in any medium, provided the original work is properly cited, the use is non-commercial and no modifications or adaptations are made.© 2022 The Authors. Bipolar Disorders published by John Wiley & Sons Ltd. 13995618, 2022, 6, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/bdi.13210 by Cochrane Sweden, Wiley Online Library on [17/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License690  |    CLINICAL CURIOSITYrepresentation in the data graph. Thus, each year became a differ-ent voice in the same score. To emphasize the melancholic nature of depression, we composed both scores in A minor. Both auditory versions were written for and are performed by violoncello, which, due to its sound quality (timbre), can be subjectively associated with sadness, an emotional state inherent in (bipolar) depression.Both sound files (in WAV and OGG format; Audio S1– S4) were created with the free software MuseScore 3.6.2. The auditory rep-resentations complement the underlying music sheets (Data S1). Two complementary spectrograms (Figures 1 and 2) that visually match the sound structure of the sound files with color patterns were created with Sonic Visualizer 4.4. The iMoodJournal data used for sonification are available in Data S2.Summarizing and briefly describing the results, the sound struc-ture of the first composition represents periods of depression and re-mission, with lower and higher pitches, respectively. These sequences were also manifested in the sound volume: depression (fff) and pe-riods of remissions (p) and transitions between these periods in the form of crescendos (relapse into depression) and decrescendos (shift to remission). These patterns were visually supported by the spec-trograms (Figure 1). The depressions were particularly highlighted by the yellow areas in the melodic range spectrogram (Figure 1, upper panel) and the more smeared- out bands in the spectrogram (Figure 1, lower panel). A high mood state was only scored once in August 2019 that “sticks out” from the remainder of the sound patterns in the form of a marcato in the sound file. This moment was captured by a distinct band in the spectrogram (Figure 1, lower panel) and a lighter point amidst the blue- purple areas in the melodic range spectrogram (Figure 1, upper panel). Lower, louder pitches (depression) can be heard especially at the beginning (declining mood state from January to February/March) and more subtly toward the end of 2017, 2018, and 2019. Despite these patterns being similar across these years in the sound files (see also Figure 1), there was also between- year vari-ability evident. This auditory variability not only emanated from the sequence of pitches but also in their relative lengths, which indicates the stability of mood states. 2020 differed from the previous years, with higher pitches superseded by lower ones at the beginning of the year (January and February), indicating more short- termed mood swings. These patterns are again supported by the spectrograms (Figure 1). A more stable emotional state followed between March and the middle of July revealed in relatively narrow pitch intervals and durations of pitches. The shift to lower and louder pitches, indicating a relapse to depression, was apparent until August. These patterns are also mirrored in the visuals (Figure 1). Afterward, the patient remitted to more functional mood states (crescendo), until the end of the year.The emerging sound structure in part 2 in which the 4 years were overplayed and performed simultaneously to emphasize the between- year variability of symptom expression in bipolar, despite seeming chaotic, carried a subjectively interpretable emergence of harmony and had esthetic appeal. None of the voices came across clearly for substantial parts across the years, despite different sound volumes. Only from October onwards were more synchronous sound patterns observed across years, evinced by the durations of pitches. Patterns in the spectrograms support these latter patterns (Figure 2). They also highlight the occasional dominance of individ-ual years (light areas in the melodic range spectrogram; stacked bands in the spectrogram) that were less obvious in the sound file.2  |  DISCUSSIONData sonification has been used across disparate disciplines, for in-stance, the social (e.g., political crises and science literacy), natural (e.g., astronomy and climate change), communication (Tweetscapes analyses), and health (e.g., seizure detection, biofeedback of facial EMG activity, and diabetes) sciences for an alternative (artistic) rep-resentation of complex scientific phenomena. Mental health com-prises undoubtedly an intricate field with multidimensional and multiscale interactions between science, clinics, technology, society, economy, and policy.1 Although transdisciplinary in a sense that our approach operates at the intersection of the arts (visual and music), science (data), technology (mental health tracking app), and the clin-ics (patient engagement), our approach is based on data from a single patient. Our study is therefore only a very simplistic representation of this complexity and thus an initial demonstration only.Despite being preliminary, we consider data sonification, how-ever, as a potentially useful tool for complementing traditional clin-ical approaches, such as, for instance, psychotherapy and cognitive behavioral therapy. Specifically, we consider the clinics as users of the approach and the arts as a channel conveying science, including monitored mental health scorings. The next step in applying data Key MessageData sonification may have potential to eviscerate the emo-tional strain of patients with mental disorders through music.Translating monitored mood states into sound may offer clinical opportunities for empowering patients to manage, predict and likely communicate their mood states.Learning Points• There is transdisciplinary, collaborative potential for data sonification due to the intersection of technology (mental health apps), science (data monitoring), the arts (visual and music) and the clinics (patient engagement).• Further research may be warranted to further assess the usefulness of data sonification as a clinical tool.• Although highly speculative at this stage, data sonifi-cation may even potentially be applicable beyond the clinics to inform through outreach about society- level challenges, such as an increasing mental health pan-demic and policy implications. 13995618, 2022, 6, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/bdi.13210 by Cochrane Sweden, Wiley Online Library on [17/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License    |  691CLINICAL CURIOSITYsonification may therefore be at the clinical level; for instance, in the form of prospective studies that may assess the broader potential of this approach for future research and clinical translation. More pre-cisely, data sonification is a novel way of communicating information about data, contrasting with a visual mood chart that graphs data points. The initial steps toward likely clinical application would be to determine whether and in what ways people with bipolar disor-der and healthcare providers find value in sonification beyond data presented in traditional mood charts. In essence, qualitative “market research” on the perceived value of this innovation could provide ideas for how data sonification might be integrated into routine.The clinical implementation of data sonification may be espe-cially useful because it simultaneously operates at two interacting psychological domains that benefit the learning process: the cog-nitive domain which relates to logic and thinking and predominates in science, and, prevailing in the arts, the affective domain which engages emotions and feeling.2 Being able to engage emotionally with “cold” monitoring data suggests that adding data sonification to the current therapeutical tool box for managing mental health may help empower patients, not only by better understanding their mental health conditions but also to manage, predict, and maybe even communicate their mood. That the method was emotionally appealing, highlighting the power of art for the clinics, was at least evident in our patient, helping him to learn about the self in novel ways beyond traditional mental health management forms. This suggests that further work may be warranted to further explore and develop this potential for improving health outcomes for pa-tients with mental disorders. Also, potential applications regarding other neuropathological and neurodegenerative conditions such as dementia and Alzheimer might likely be worth exploring.We acknowledge that the power of art to cause an impact and interpretations is subjective and can therefore vary with context. Specifically, subjective representations alone can be risky because societal challenges may be perceived as distant and unlinked to per-sonal experience, due to ambiguity, leading to misunderstanding and decontextualization. This has been shown for example in the environmental sciences.3 The problem of decontextualization may be exacerbated by social, economic, and demographic factors such as deficiencies in science literacy, graphical, and numeracy under-standing in the public.4 Data sonification comprises arguably an ob-jective approach for making the silent voices of mental illness heard and can therefore complement subjective artistic representation of emotions and feelings, such as drawings, paintings, classic, and other music genres, sculpture, and photography.F I G U R E  1  Melodic range spectrogram (upper panel) and spectrogram (lower panel) presented for sequential years. The graphs have been extracted from the WAV audio file (part 1). X- axes show frequencies in Hz.  13995618, 2022, 6, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/bdi.13210 by Cochrane Sweden, Wiley Online Library on [17/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License692  |    CLINICAL CURIOSITYWhile we see the most immediate application of data sonification in the clinics, there may be speculatively potential to further develop the approach in subsequent steps to address the pervasive gaps in mental health awareness and public literacy operating at the socie-tal level,5 especially those pertaining to an increasing global mental health crisis.1 However, we acknowledge that the societal scale has a significantly higher level of complexity than the patient/clinical level and other data requirements compared to those of the present study. The likely impact of data sonification as an outreach/information tool for societies is, at this stage, therefore difficult to envision.We conclude by highlighting that data sonification alone may not be a silver bullet in the clinics. However, adding it to the arsenal of therapeutical tools may allow us to give a targeted shot for refin-ing current mental health management by striving for more holistic models not only rooted in the sciences but also in the arts.ACKNOWLEDG MENTSWithout the generous sharing of data and the enthusiasm of our anonymous participant, this study would have been impossible. His support, interest, and engagement throughout this project are much appreciated. MB is supported by an NHMRC Senior Principal Research Fellowship (1156072). An anonymous reviewer is ac-knowledged for providing constructive criticism that improved the paper.CONFLIC T OF INTERE S TThe authors declare that no conflict of interest exists.DATA AVAIL ABILIT Y S TATEMENTThe iMoodJournal data used for sonification are available in Data S2.ORCIDDavid G. Angeler  https://orcid.org/0000-0003-2197-7470 R E FE R E N C E S 1. Eyre HA, Berk M, Lavretsky H, Reynolds CF III. Convergence Mental Health: A Transdisciplinary Approach to Innovation. Oxford University Press; 2021. 2. Lesen AE, Rogan A, Blum MJ. Science communication through art: ob-jectives, challenges, and outcomes. Trends Ecol Evol. 2016;31:657- 660. 3. Moser SC. Communicating climate change: history, challenges, pro-cess and future directions. WIRE Climate Change. 2010;1:31- 53. 4. Sawe N, Chafe C, Treviño J. Using data sonification to overcome science literacy, numeracy, and visualization barriers in science communication. Front Comm. 2020;5:46. 5. Furnham A, Swami V. Mental health literacy: a review of what it is and why it matters. Int Persp Psychol. 2018;7(4):240- 257.SUPPORTING INFORMATIONAdditional supporting information may be found in the online version of the article at the publisher’s website.F I G U R E  2  Melodic range spectrogram (upper panel) and spectrogram (lower panel) presented for overlaid years. The graphs have been extracted from the WAV audio file (part 2). X- axes show frequencies in Hz.  13995618, 2022, 6, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/bdi.13210 by Cochrane Sweden, Wiley Online Library on [17/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Listening to the silent struggles of bipolar disorder through sonification of iMoodJournal data

https://pub.epsilon.slu.se/29273/1/angeler-d-g-et-al-20221017.pdf

Listening to the silent struggles of bipolar disorder through sonification of iMoodJournal data

Abstract

Similar works

Full text

Available Versions

Epsilon Open Archive