Studies on community-based environmental monitoring, sustained civic engagement and new opportunities in journalism

In the presented work, Internet of Things (IoT) technologies, like sensors and their applications, have been investigated from a socio-technical perspective focusing on civic purposes of the technologies’ use. The aim of the work is to gain a better understanding of civic IoT and its transformative potential for society. Therefore, the overarching research question is: What are the societal implications of the technological object civic IoT? Three application areas identified as highly relevant have been studied: community-based environmental monitoring, sustained engagement of civic tech initiatives, and the “Journalism of Things.” In the first study, the civic IoT initiative Luftdaten.info dedicated to the community-based monitoring of particulate matter in the air was examined. The initiative is organized within limits of technical equipment, resources, and academic knowledge. The case study comprises a media content analysis, a web application analysis, and two expert interviews. The data illustrates that the local air pollution topic is a discursive long-term process striving towards a more sustainable city and community. The findings show that the information provided by Luftdaten motivates certain people to become aware and engaged for their local environment, and some to change their behavior. I found that the initiative itself is managed in a sustainable way being inclusive and resource-saving. The media content analysis shows that events around the emergence of the Luftdaten initiative had a certain influence on the local media agenda. Finally, a data map comparison shows how easily misunderstandings in the representation of monitoring data can occur when taking different decisions during the data analysis and visualization of almost identical datasets. In the second study, two long-lasting civic tech initiatives of global scale were investigated to understand what makes them sustain over time. We conducted two mixed-method case studies of the initiative Luftdaten.info from Germany and the initiative Safecast from Japan. We combined social network analysis and qualitative content analysis of Twitter data with insights from expert interviews. Drawing on our findings, we identified a set of key factors that help the studied civic tech initiatives to grow and last. Contributing to Digital Civics in HCI, we argue that the civic tech initiatives’ scaling and sustaining are configured through the entanglement of (1) civic data both captured and owned by the citizens for the citizens, (2) the use of open and accessible technology, and (3) the initiatives’ public narrative, giving them a voice on the environmental issue. In the third study, Journalism of Things (JoT) as a new paradigm in digital journalism was investigated. Three case studies on recent award-winning journalism projects in Germany were conducted with the analytical lenses of boundary work and objects of journalism. The study comprises interviews with journalists, media content analyses, and observations of virtual public events. The findings suggest four typical phases in JoT projects: formation, data work, presentation, and ramification. Blurred boundaries of journalism towards science and activism become apparent when co-creative JoT teams apply scientific methods and technology design while mobilizing communities. Findings further show that things (or objects) of JoT have implications on the configuration for collaborative arrangements and audience relations. By creating and disseminating new local knowledge on matters of common concern, JoT is also contributing to empowering both journalism and citizens. This dissertation is a cumulative work comprising three peer-reviewed scientific publications

Wizundry: A Cooperative Wizard of Oz Platform for Simulating Future Speech-based Interfaces with Multiple Wizards

Wizard of Oz (WoZ) as a prototyping method has been used to simulate intelligent user interfaces, particularly for speech-based systems. However, as our societies' expectations on artificial intelligence (AI) grows, the question remains whether a single Wizard is sufficient for it to simulate smarter systems and more complex interactions. Optimistic visions of 'what artificial intelligence (AI) can do' places demands on WoZ platforms to simulate smarter systems and more complex interactions. This raises the question of whether the typical approach of employing a single Wizard is sufficient. Moreover, while existing work has employed multiple Wizards in WoZ studies, a multi-Wizard approach has not been systematically studied in terms of feasibility, effectiveness, and challenges. We offer Wizundry, a real-time, web-based WoZ platform that allows multiple Wizards to collaboratively operate a speech-to-text based system remotely. We outline the design and technical specifications of our open-source platform, which we iterated over two design phases. We report on two studies in which participant-Wizards were tasked with negotiating how to cooperatively simulate an interface that can handle natural speech for dictation and text editing as well as other intelligent text processing tasks. We offer qualitative findings on the Multi-Wizard experience for Dyads and Triads of Wizards. Our findings reveal the promises and challenges of the multi-Wizard approach and open up new research questions.Comment: 34 page

NON-VERBAL COMMUNICATION WITH PHYSIOLOGICAL SENSORS. THE AESTHETIC DOMAIN OF WEARABLES AND NEURAL NETWORKS

Historically, communication implies the transfer of information between bodies, yet this phenomenon is constantly adapting to new technological and cultural standards. In a digital context, it’s commonplace to envision systems that revolve around verbal modalities. However, behavioural analysis grounded in psychology research calls attention to the emotional information disclosed by non-verbal social cues, in particular, actions that are involuntary. This notion has circulated heavily into various interdisciplinary computing research fields, from which multiple studies have arisen, correlating non-verbal activity to socio-affective inferences. These are often derived from some form of motion capture and other wearable sensors, measuring the ‘invisible’ bioelectrical changes that occur from inside the body. This thesis proposes a motivation and methodology for using physiological sensory data as an expressive resource for technology-mediated interactions. Initialised from a thorough discussion on state-of-the-art technologies and established design principles regarding this topic, then applied to a novel approach alongside a selection of practice works to compliment this. We advocate for aesthetic experience, experimenting with abstract representations. Atypically from prevailing Affective Computing systems, the intention is not to infer or classify emotion but rather to create new opportunities for rich gestural exchange, unconfined to the verbal domain. Given the preliminary proposition of non-representation, we justify a correspondence with modern Machine Learning and multimedia interaction strategies, applying an iterative, human-centred approach to improve personalisation without the compromising emotional potential of bodily gesture. Where related studies in the past have successfully provoked strong design concepts through innovative fabrications, these are typically limited to simple linear, one-to-one mappings and often neglect multi-user environments; we foresee a vast potential. In our use cases, we adopt neural network architectures to generate highly granular biofeedback from low-dimensional input data. We present the following proof-of-concepts: Breathing Correspondence, a wearable biofeedback system inspired by Somaesthetic design principles; Latent Steps, a real-time auto-encoder to represent bodily experiences from sensor data, designed for dance performance; and Anti-Social Distancing Ensemble, an installation for public space interventions, analysing physical distance to generate a collective soundscape. Key findings are extracted from the individual reports to formulate an extensive technical and theoretical framework around this topic. The projects first aim to embrace some alternative perspectives already established within Affective Computing research. From here, these concepts evolve deeper, bridging theories from contemporary creative and technical practices with the advancement of biomedical technologies.Historicamente, os processos de comunicação implicam a transferência de informação entre organismos, mas este fenómeno está constantemente a adaptar-se a novos padrões tecnológicos e culturais. Num contexto digital, é comum encontrar sistemas que giram em torno de modalidades verbais. Contudo, a análise comportamental fundamentada na investigação psicológica chama a atenção para a informação emocional revelada por sinais sociais não verbais, em particular, acções que são involuntárias. Esta noção circulou fortemente em vários campos interdisciplinares de investigação na área das ciências da computação, dos quais surgiram múltiplos estudos, correlacionando a actividade nãoverbal com inferências sócio-afectivas. Estes são frequentemente derivados de alguma forma de captura de movimento e sensores “wearable”, medindo as alterações bioeléctricas “invisíveis” que ocorrem no interior do corpo. Nesta tese, propomos uma motivação e metodologia para a utilização de dados sensoriais fisiológicos como um recurso expressivo para interacções mediadas pela tecnologia. Iniciada a partir de uma discussão aprofundada sobre tecnologias de ponta e princípios de concepção estabelecidos relativamente a este tópico, depois aplicada a uma nova abordagem, juntamente com uma selecção de trabalhos práticos, para complementar esta. Defendemos a experiência estética, experimentando com representações abstractas. Contrariamente aos sistemas de Computação Afectiva predominantes, a intenção não é inferir ou classificar a emoção, mas sim criar novas oportunidades para uma rica troca gestual, não confinada ao domínio verbal. Dada a proposta preliminar de não representação, justificamos uma correspondência com estratégias modernas de Machine Learning e interacção multimédia, aplicando uma abordagem iterativa e centrada no ser humano para melhorar a personalização sem o potencial emocional comprometedor do gesto corporal. Nos casos em que estudos anteriores demonstraram com sucesso conceitos de design fortes através de fabricações inovadoras, estes limitam-se tipicamente a simples mapeamentos lineares, um-para-um, e muitas vezes negligenciam ambientes multi-utilizadores; com este trabalho, prevemos um potencial alargado. Nos nossos casos de utilização, adoptamos arquitecturas de redes neurais para gerar biofeedback altamente granular a partir de dados de entrada de baixa dimensão. Apresentamos as seguintes provas de conceitos: Breathing Correspondence, um sistema de biofeedback wearable inspirado nos princípios de design somaestético; Latent Steps, um modelo autoencoder em tempo real para representar experiências corporais a partir de dados de sensores, concebido para desempenho de dança; e Anti-Social Distancing Ensemble, uma instalação para intervenções no espaço público, analisando a distância física para gerar uma paisagem sonora colectiva. Os principais resultados são extraídos dos relatórios individuais, para formular um quadro técnico e teórico alargado para expandir sobre este tópico. Os projectos têm como primeiro objectivo abraçar algumas perspectivas alternativas às que já estão estabelecidas no âmbito da investigação da Computação Afectiva. A partir daqui, estes conceitos evoluem mais profundamente, fazendo a ponte entre as teorias das práticas criativas e técnicas contemporâneas com o avanço das tecnologias biomédicas

Ih­mi­sen ja tie­to­ko­neen vä­li­nen yh­teis­luo­vuus : runoja kirjoittavien yh­teis­luo­vien jär­jes­tel­mien suun­nit­te­lu ja ar­vioin­ti sekä yh­teis­luo­van pro­ses­sin mal­lin­ta­mi­nen

Human-computer co-creativity examines creative collaboration between humans and artificially intelligent computational agents. Human-computer co-creativity researchers assume that instead of using computational systems to merely automate creative tasks, computational creativity methods can be leveraged to design computational collaborators capable of sharing creative responsibility with a human collaborator. This has potential for extending both human and computational creative capability. This thesis focuses on the case of one human and one computational collaborator. More specifically this thesis studies how children collaborate with a computational collaborator called the Poetry Machine in the linguistically creative task of writing poems. This thesis investigates three topics related to human-computer co-creativity: The design of human-computer co-creative systems, their evaluation and the modelling of human-computer co-creative processes. These topics are approached from two perspectives: an interaction design perspective and a computational creativity perspective. The interaction design perspective provides practical methods for the design and evaluation of interactive systems as well as methodological frameworks for analysing design practices in the field. The computational creativity perspective then again provides a theoretical view to the evaluation and modelling of human-computer co-creativity. The thesis itself consists of five papers. This thesis starts with an analysis of the interaction design process for computational collaborators. The design process is examined through a review of case studies, and a thorough description of the design process of the Poetry Machine system described in Paper I. The review shows that several researchers in the field have assumed a user-centered design approach, but some good design practices, including the reporting of design decisions, iterative design and early testing with users are not yet fulfilled according to the best standards. After illustrating the general design process, this thesis examines different approaches to the evaluation of human-computer co-creativity. Two case studies are conducted to evaluate the usability of and user experiences with the Poetry Machine system. The first evaluations are described in Paper II. They produced useful feedback for developing the system further. The second evaluation, described in Papers III and IV, investigates specific metrics for evaluating the co-creative writing experience in more detail. To promote the accumulation of design knowledge, special care is taken to report practical issues related to evaluating co-creative systems. These include, for example, issues related to formulating suitable evaluation tasks. Finally the thesis considers modelling human-computer co-creativity. Paper V approaches modelling from a computationally creative perspective, by extending the creativity-as-a-search paradigm into co-creative systems. The new model highlights specific issues for interaction designers to be aware of when designing new computational collaborators.Ihmisen ja tietokoneen välinen yhteisluovuus on tutkimusala, joka käsittelee ihmisten ja tekoälyagenttien välistä luovaa yhteistyötä. Tekoälyagenttien perustana toimivat uudet laskennallisen luovuuden metodit. Ne mahdollistavat pelkän luovien tehtävien automatisoinnin sijaan tasapainoisemman vastuunjaon ja vuorovaikutuksen ihmisen ja tekoälyagentin välillä. Tämä tarjoaa sekä ihmisille että laskennallisille agenteille uusia luovia mahdollisuuksia. Väitöskirja keskittyy erityisesti yhden ihmisen ja laskennallisesti luovan agentin yhteistyöhön. Väitöskirja koostuu viidestä erillisestä julkaisusta, ja siihen kuuluvissa tapaustutkimuksissa havainnoidaan lasten ja laskennalliseen kielelliseen luovuuteen perustuvan Runokone–nimisen laskennallisesti luovan agentin yhteistyötä. Väitöskirjassa käsitellään ihmisen ja tietokoneen välisen yhteisluovuuden kolmea teemaa: yhteisluovien järjestelmien suunnittelua, niiden arviointia ja ihmisen ja tietokoneen välisen yhteisluovan prosessin mallinnusta. Teemojen tutkimiseen käytetään vuorovaikutussuunnittelun ja laskennallisen luovuuden menetelmiä. Vuorovaikutussuunnittelu tarjoaa käytännönläheisiä menetelmiä järjestelmien suunnitteluun ja arviointiin sekä erilaisia teoreettisia näkökulmia alalla vallitsevien suunnittelukäytäntöjen tarkasteluun. Laskennallisen luovuuden tutkimus puolestaan tarjoaa teoreettisen näkökulman yhteisluovien järjestelmien arviointiin ja yhteisluovuuden mallinnukseen. Ensimmäistä teemaa, yhteisluovien järjestelmien suunnittelua, käsitellään väitöskirjan julkaisussa I. Julkaisussa kuvataan yhteisluovien järjestelmien yleistä vuorovaikutussuunnitteluprosessia tapaustutkimuskatsauksen kautta, ja tarkastellaan Runokoneen suunnitteluprosessia. Tutkimuskatsaus osoittaa alan tutkijoiden usein valitsevan tutkimuksensa lähtökohdaksi käyttäjäkeskeisen suunnittelun. He kuitenkin noudattavat parhaita vuorovaikutussuunnittelun käytäntöjä vain löyhästi. Tiedeyhteisön sisällä tulisikin siksi parantaa erityisesti suunnittelupäätösten dokumentointia, iteratiivista suunnittelua ja varhaista käyttäjätestausta. Toista teemaa, ihmisen ja koneen välisen yhteisluovuuden arviointia, tarkastellaan väitöksessä kahden tapaustutkimuksen kautta. Niistä ensimmäisessä keskitytään Runokoneen käytettävyyden arviointiin ja toisessa Runokoneen käyttäjien kokemusten arviointiin. Käytettävyyden arviointia on kuvattu tarkemmin julkaisussa II. Arviointi tuotti hyödyllistä palautetta järjestelmän jatkokehitystä varten. Julkaisuissa III ja IV tarkastellaan mittareita, joiden avulla voidaan arvioida tarkemmin käyttäjien käyttäjäkokemuksia erilaisissa yhteisluovan kirjoittamisen prosesseissa. Vuorovaikutussuunnittelun tutkimuksen ja käytännön suunnittelutyön tukemiseksi julkaisuissa paneudutaan erityisesti yhteisluovien järjestelmien arvioinnin käytännön ongelmiin. Näihin kuuluu esimerkiksi sopivien arviointitehtävien muodostaminen. Lopuksi väitöskirjassa käsitellään ihmisen ja koneen välisen yhteisluovuuden mallinnusta. Julkaisussa V tarkastellaan mallinnusta laskennallisen luovuuden näkökulmasta laajentamalla luovan haun paradigmaa yhteisluoviin järjestelmiin. Luovan haun paradigma kuvaa luomisprosessia sekä esteettisesti miellyttävien että luovaan kohdealaan sopivien artefaktien etsintänä hakuavaruudessa. Kuvatussa laajennuksessa painottuvat vuorovaikutussuunnittelun kannalta oleelliset ristiriitatilanteet, joiden ratkaisutavat vaikuttavat laskennallisesti luovien yhteistyökumppaneiden ominaisuuksiin

Collaborative video editing

Samarbeid i videoredigering Denne avhandlingen tar opp følgende spørsmål: Hvordan kan vi støtte samarbeid i videoredigering? I ulike anvendelsesområder, som skriving og design, er bruk av samarbeidsverktøy utbredt. Likevel er programvare for videoredigering i hovedsak utviklet for individuell bruk. Videoredigering bør forstås som en sosial aktivitet og blir i profesjonelle sammenhenger ofte utført som et samarbeid mellom ulike aktører. Basert på intervjuer og designverksteder, undersøker denne avhandlingen hvordan videoredigerere samarbeider og utforsker mulighetsrommet for å støtte samarbeid i videoredigering gjennom design av nye løsninger. I tre studier undersøker denne avhandlingen videoredigering fra tre perspektiver. Først undersøker den samarbeidspraksiser blant profesjonelle videoredigerere og identifiserer ulike strategier og sosiale mekanismer som brukes for å oppnå enighet mellom aktørene som er involvert i videoproduksjon. Denne første studien identifiserer ni temaer som beskriver hvordan videoredigerere håndterer usikkerhet og oppnår enighet, spesielt gjennom organisatoriske mekanismer, dokumentasjon og ikoniske referanser. Studien foreslår også tre ulike retninger for design av nye løsninger for å støtte samarbeid i videoredigering. Det andre studiet undersøker videoproduksjon fra et organisatorisk perspektiv, med fokus på en pågående overgang til distribuert arbeid og dets innvirkning på videoproduksjon. Den andre studien skisserer de kortsiktige og langsiktige implikasjonene av å innføre distribuerte arbeidsformer i TV-produksjonsorganisasjoner under COVID-19-pandemien. Den siste studien ser på samarbeid i videoredigering som et designproblem og presenterer designideer for hvordan man kan støtte et slikt samarbeid. I tillegg peker denne studien på utfordringer som kan være til hinder for innføringen av nye videoredigeringsverktøy som skal støtte samarbeid. Ved å sammenstille resultatene fra de tre studiene, samt analysere tidligere forskning og eksisterende videoredigeringsverktøy, identifiserer avhandlingen tre designtilnærminger for å støtte samarbeid i videoredigeringsprogramvare: holistisk, skreddersydd og konfigurerbar. Selv om disse tilnærmingene diskuteres med tanke på samarbeid i videoredigeringspraksiser, kan de tilby et bredere analytisk rammeverk for å vurdere utformingen av samarbeidsverktøy også for andre anvendelsesområder.This thesis addresses the following question: how can collaboration be supported in video editing? In many domains, such as writing and design, collaborative tools have become common and widespread. However, video-editing software is still predominantly designed for solo users. Nevertheless, video editing is a social activity that, in a professional setting, often involves various people working together. Based on interviews and design workshops, this thesis investigates the collaborative practices of video editors and explores the design space of collaborative video editing. In three studies, this thesis looks at video editing from three angles. First, it investigates the collaborative practices of video editors and identifies the strategies and social mechanisms they employ to reach agreements with various parties involved in the videoproduction process. The first study identifies nine themes that characterise the ways video editors manage uncertainties and reach agreements, particularly through organisational mechanisms, documentation, and iconic referencing. The study also suggests three design paths to explore further. Second, it examines video production from an organisational point of view, focusing on the recent shift towards remote work and its impact on video production. The second study delineates the short-term and long-term implications of adopting remote work in TV production organisations during the COVID-19 pandemic. Third, it approaches collaborative video editing as a design problem and offers design ideas to enhance collaboration. Additionally, it uncovers challenges that might impede the adoption of new collaborative video-editing tools. In synthesising the results of the three studies, as well as analysing previous research and existing video-editing tools, this thesis identifies three design approaches for supporting collaboration in video-editing software: holistic, tailored, and configurable. While discussed in the context of collaborative video editing, these approaches offer a broader analytical framework for considering the design of collaborative production tools.Doktorgradsavhandlin

A competencies framework of visual impairments for enabling shared understanding in design

Existing work in Human Computer Interaction and accessibility research has long sought to investigate the experiences of people with visual impairments in order to address their needs through technology design and integrate their participation into different stages of the design process. Yet challenges remain regarding how disabilities are framed in technology design and the extent of involvement of disabled people within it. Furthermore, accessibility is often considered a specialised job and misunderstandings or assumptions about visually impaired people’s experiences and needs occur outside dedicated fields. This thesis presents an ethnomethodology-informed design critique for supporting awareness and shared understanding of visual impairments and accessibility that centres on their experiences, abilities, and participation in early-stage design. This work is rooted in an in-depth empirical investigation of the interactional competencies that people with visual impairments exhibit through their use of technology, which informs and shapes the concept of a Competencies Framework of Visual Impairments. Although past research has established stances for considering the individual abilities of disabled people and other social and relational factors in technology design, by drawing on ethnomethodology and its interest in situated competence this thesis employs an interactional perspective to investigate the practical accomplishments of visually impaired people. Thus, this thesis frames visual impairments in terms of competencies to be considered in the design process, rather than a deficiency or problem to be fixed through technology. Accordingly, this work favours supporting awareness and reflection rather than the design of particular solutions, which are also strongly needed for advancing accessible design at large. This PhD thesis comprises two main empirical studies branched into three different investigations. The first and second investigations are based on a four-month ethnographic study with visually impaired participants examining their everyday technology practices. The third investigation comprises the design and implementation of a workshop study developed to include people with and without visual impairments in collaborative reflections about technology and accessibility. As such, each investigation informed the ones that followed, revisiting and refining concepts and design materials throughout the thesis. Although ethnomethodology is the overarching approach running through this PhD project, each investigation has a different focus of enquiry: • The first is focused on analysing participants’ technology practices and unearthing the interactional competencies enabling them. • The second is focused on analysing technology demonstrations, which were a pervasive phenomenon recorded during fieldwork, and the work of demonstrating as exhibited by visually impaired participants. • Lastly, the third investigation defines a workshop approach employing video demonstrations and a deck of reflective design cards as building blocks for enabling shared understanding among people with and without visual impairments from different technology backgrounds; that is, users, technologists, designers, and researchers. Overall, this thesis makes several contributions to audiences within and outside academia, such as the detailed accounts of some of the main technology practices of people with visual impairments and the methodological analysis of demonstrations in empirical Human Computer Interaction and accessibility research. Moreover, the main contribution lies in the conceptualisation of a Competencies Framework of Visual Impairments from the empirical analysis of interactional competencies and their practical exhibition through demonstrations, as well as the creation and use of a deck of cards that encapsulates the competencies and external elements involved in the everyday interactional accomplishments of people with visual impairments. All these contributions are lastly brought together in the implementation of the workshop approach that enabled participants to interact with and learn from each other. Thus, this thesis builds upon and advances contemporary strands of work in Human Computer Interaction that call for re-orienting how visual impairments and, overall, disabilities are framed in technology design, and ultimately for re-shaping the design practice itself