LiFT: A Scalable Framework for Measuring Fairness in ML Applications

Many internet applications are powered by machine learned models, which are usually trained on labeled datasets obtained through either implicit / explicit user feedback signals or human judgments. Since societal biases may be present in the generation of such datasets, it is possible for the trained models to be biased, thereby resulting in potential discrimination and harms for disadvantaged groups. Motivated by the need for understanding and addressing algorithmic bias in web-scale ML systems and the limitations of existing fairness toolkits, we present the LinkedIn Fairness Toolkit (LiFT), a framework for scalable computation of fairness metrics as part of large ML systems. We highlight the key requirements in deployed settings, and present the design of our fairness measurement system. We discuss the challenges encountered in incorporating fairness tools in practice and the lessons learned during deployment at LinkedIn. Finally, we provide open problems based on practical experience.Comment: Accepted for publication in CIKM 202

HydroShare – A Case Study of the Application of Modern Software Engineering to a Large Distributed Federally-Funded Scientific Software Development Project

HydroShare is an online collaborative system under development to support the open sharing of hydrologic data, analytical tools, and computer models. With HydroShare, scientists can easily discover, access, and analyze hydrologic data and thereby enhance the production and reproducibility of hydrologic scientific results. HydroShare also takes advantage of emerging social media functionality to enable users to enhance information about and collaboration around hydrologic data and models. HydroShare is being developed by an interdisciplinary collaborative team of domain scientists, university software developers, and professional software engineers from ten institutions located across the United States. While the combination of non–co-located, diverse stakeholders presents communication and management challenges, the interdisciplinary nature of the team is integral to the project’s goal of improving scientific software development and capabilities in academia. This chapter describes the challenges faced and lessons learned with the development of HydroShare, as well as the approach to software development that the HydroShare team adopted on the basis of the lessons learned. The chapter closes with recommendations for the application of modern software engineering techniques to large, collaborative, scientific software development projects, similar to the National Science Foundation (NSF)–funded HydroShare, in order to promote the successful application of the approach described herein by other teams for other projects

Analysis and Perspective from the Complex Aerospace Systems Exchange (CASE) 2013

NASA Langley Research Center embedded four rapporteurs at the Complex Aerospace Systems Exchange (CASE) held in August 2013 with the objective to capture the essence of the conference presentations and discussions. CASE was established to provide a discussion forum among chief engineers, program managers, and systems engineers on challenges in the engineering of complex aerospace systems. The meeting consists of invited presentations and panels from industry, academia, and government followed by discussions among attendees. This report presents the major and reoccurring themes captured throughout the meeting and provides analysis and insights to further the CASE mission

Evaluating Model Testing and Model Checking for Finding Requirements Violations in Simulink Models

Matlab/Simulink is a development and simulation language that is widely used by the Cyber-Physical System (CPS) industry to model dynamical systems. There are two mainstream approaches to verify CPS Simulink models: model testing that attempts to identify failures in models by executing them for a number of sampled test inputs, and model checking that attempts to exhaustively check the correctness of models against some given formal properties. In this paper, we present an industrial Simulink model benchmark, provide a categorization of different model types in the benchmark, describe the recurring logical patterns in the model requirements, and discuss the results of applying model checking and model testing approaches to identify requirements violations in the benchmarked models. Based on the results, we discuss the strengths and weaknesses of model testing and model checking. Our results further suggest that model checking and model testing are complementary and by combining them, we can significantly enhance the capabilities of each of these approaches individually. We conclude by providing guidelines as to how the two approaches can be best applied together.Comment: 10 pages + 2 page reference

Effects of regular use of scalable, technology enhanced solution for primary mathematics education

Mathematics is one of the key subjects in any school curriculum and most teachers agree that mathematical skills are important for students to master. There is an abundance of research in learning mathematics and a consensus exists among researchers that technology can enhance the learning process. However, many factors need to be taken into consideration when introducing technology into teaching mathematics. Developing a more natural collaboration between learning technology experts, teachers, and students ensures all stakeholders are considered. Involving teachers early on helps develop enduring commitment to innovations and practical solutions. Moreover, creating a culture of collaboration between experts in the field and teachers brings to bear the best of what both worlds have to offer. This thesis synthesizes six papers and offers additional findings that focus on how technology experts can collaborate with elementary teachers to improve student learning outcomes. We focus on managing educational change in ways that improve the sustainability of innovations. We also explore how technical and teaching experts co-create effective lesson plans. In one of the six papers we collected and reported teachers’ responses to survey questions covering typical usage patterns on a platform. Teachers’ direct feedback was collected and incorporated to improve technical solutions. Moreover, one study was conducted abroad to measure the effect of culture on the teaching and learning process. Evidence of effectiveness of technologically enhanced lessons and corresponding homework was based on multiple studies in grades 1 - 3, covering 379 students. The effectiveness of educational technology was measured based on two variables: student performance in mathematics, based on the learning objectives specified in the curriculum, and arithmetic fluency measured by how rapidly and accurately students solved basic arithmetic operations. Statistically significant findings show that educational technology can improve two target variables when comparing students who did not use educational technology to students who did. An additional effect size analysis was conducted to verify and compare results with previous research. Based on these results, platform use produced the same or better effect than previous studies. Based on teacher feedback and user growth on the platform, we managed to integrate technology into the regular school classroom in meaningful and sustainable ways. We were clearly able to support teachers in their practice in a manner that resulted in noticeable student achievement gains. A survey revealed a need to emphasize new features that were introduced to the platform in teacher training programs. Teachers also reported having a positive attitude towards the platform and the initiative gained wide acceptance among their peers.Matematiikka on yksi tärkeimmistä kouluaineista pelkästään tuntimääräisesti mitattunakin. Matematiikan osaamista ja oppimista pidetään yleisesti tärkeänä ja arvostettuna taitona. Matematiikan oppimisesta on valtavasti tutkimusta ja tutkijoiden keskuudessa vallitsee yhteisymmärrys tietotekniikan positiivisista mahdollisuuksista edistää matematiikan oppimista. Tietotekniikan ja oppimisen vuorovaikutus on kuitenkin monisyinen vyyhti ja sen onnistunut hyödyntäminen vaatii tutkijoiden, opettajien ja oppilaiden välistä tiivistä ja vuorovaikutteista yhteistyötä. Uusien innovaatioiden ja kokeilujen onnistumiselle ja niihin sitoutumiselle luodaan vahva pohja, kun opettajat otetaan mukaan kehitystyöhön ensimetreiltä lähtien. Tällaisen tiiviin yhteistyökulttuurin vaaliminen mahdollistaa käytännön työn ja teorian vahvuuksien hyödyntämisen. Tämä väitöstyö koostuu kuudesta artikkelista. Artikkelit kuvaavat, kuinka tutkijat ja opettajat työskentelivät yhdessä parantaakseen oppilaiden matematiikan oppimista. Tavoitteenamme oli muuttaa koulun käytänteitä pitkäjänteisesti ja kestävällä tavalla. Tutkimme kuinka tutkijat ja opettajat pystyivät yhdessä luomaan onnistuneita ja tehokkaita oppimiskokonaisuuksia. Opettajat olivat koko ajan kehitystyön keskiössä. Yhdessä kuudesta artikkelista tutkittiin kyselytutkimuksen avulla opettajien kokemuksia ja käyttötottumuksia. Näitä vastauksia hyödynnettiin teknisessä kehitystyössä ja hyvien käytänteiden hiomisessa. Yksi väitöskirjan tutkimuksista tehtiin ulkomailla opetus- ja oppimiskulttuureista vaikutusten huomioimiseksi. Sähköisten oppituntien ja kotitehtävien vaikuttavuuden arviointi perustuu useisiin 1.-3. luokilla tehtyihin tutkimuksiin ja kaikkiaan 379 oppilaan vastauksiin. Sähköisten oppituntien vaikuttavuutta arvioitiin kahden eri mittarin perusteella. Ensin matematiikan taitojen perusteella, eli kuinka hyvin kunkin luokka-asteen oppimistavoitteet olivat täyttyneet ja myöhemmin myös laskusujuvuuden perusteella, eli kuinka nopeasti ja tarkasti oppilaat pystyivät laskemaan peruslaskutoimituksia. Tulokset osoittavat, että opetusteknologian avulla pystytään parantamaan oppilaiden suoriutumista edellä mainittujen osa-alueiden osalta verrattuna oppilaisiin, jotka eivät käyttäneet opetusteknologiaa. Tulokset olivat tilastollisesti merkitseviä. Näiden tulosten varmistamiseksi laskettiin vaikuttavuuden suuruus ja sitä verrattiin aiempiin alan tutkimuksiin. Tulosten perusteella sähköisillä oppitunneilla oli sama tai parempi vaikuttavuus kuin aiemmissa tutkimuksissa. Opettajien palautteiden ja kasvavan käyttäjämäärän perusteella voidaan sanoa, että onnistuimme tavoitteessamme integroida opetusteknologiaa mielekkäällä tavalla osaksi koulutyötä. Onnistuimme myös tukemaan ja auttamaan opettajia opetustyössään ja samalla merkittävästi parantamaan oppilaiden suoriutumista. Kyselytutkimuksen perusteella huomasimme, että uusien ominaisuuksien kouluttamiseen tulee kiinnittää enemmän huomiota. Samassa tutkimuksessa opettajat raportoivat olevansa tyytyväisiä alustaan ja sähköiset oppitunnit näyttävät saaneen vankan jalansijan suomalaisessa opettajakunnassa

A Survey on ChatGPT: AI-Generated Contents, Challenges, and Solutions

With the widespread use of large artificial intelligence (AI) models such as ChatGPT, AI-generated content (AIGC) has garnered increasing attention and is leading a paradigm shift in content creation and knowledge representation. AIGC uses generative large AI algorithms to assist or replace humans in creating massive, high-quality, and human-like content at a faster pace and lower cost, based on user-provided prompts. Despite the recent significant progress in AIGC, security, privacy, ethical, and legal challenges still need to be addressed. This paper presents an in-depth survey of working principles, security and privacy threats, state-of-the-art solutions, and future challenges of the AIGC paradigm. Specifically, we first explore the enabling technologies, general architecture of AIGC, and discuss its working modes and key characteristics. Then, we investigate the taxonomy of security and privacy threats to AIGC and highlight the ethical and societal implications of GPT and AIGC technologies. Furthermore, we review the state-of-the-art AIGC watermarking approaches for regulatable AIGC paradigms regarding the AIGC model and its produced content. Finally, we identify future challenges and open research directions related to AIGC.Comment: 20 pages, 6 figures, 4 table

Engineering simulations for cancer systems biology

Computer simulation can be used to inform in vivo and in vitro experimentation, enabling rapid, low-cost hypothesis generation and directing experimental design in order to test those hypotheses. In this way, in silico models become a scientific instrument for investigation, and so should be developed to high standards, be carefully calibrated and their findings presented in such that they may be reproduced. Here, we outline a framework that supports developing simulations as scientific instruments, and we select cancer systems biology as an exemplar domain, with a particular focus on cellular signalling models. We consider the challenges of lack of data, incomplete knowledge and modelling in the context of a rapidly changing knowledge base. Our framework comprises a process to clearly separate scientific and engineering concerns in model and simulation development, and an argumentation approach to documenting models for rigorous way of recording assumptions and knowledge gaps. We propose interactive, dynamic visualisation tools to enable the biological community to interact with cellular signalling models directly for experimental design. There is a mismatch in scale between these cellular models and tissue structures that are affected by tumours, and bridging this gap requires substantial computational resource. We present concurrent programming as a technology to link scales without losing important details through model simplification. We discuss the value of combining this technology, interactive visualisation, argumentation and model separation to support development of multi-scale models that represent biologically plausible cells arranged in biologically plausible structures that model cell behaviour, interactions and response to therapeutic interventions