Improving fairness in machine learning systems: What do industry practitioners need?

The potential for machine learning (ML) systems to amplify social inequities and unfairness is receiving increasing popular and academic attention. A surge of recent work has focused on the development of algorithmic tools to assess and mitigate such unfairness. If these tools are to have a positive impact on industry practice, however, it is crucial that their design be informed by an understanding of real-world needs. Through 35 semi-structured interviews and an anonymous survey of 267 ML practitioners, we conduct the first systematic investigation of commercial product teams' challenges and needs for support in developing fairer ML systems. We identify areas of alignment and disconnect between the challenges faced by industry practitioners and solutions proposed in the fair ML research literature. Based on these findings, we highlight directions for future ML and HCI research that will better address industry practitioners' needs.Comment: To appear in the 2019 ACM CHI Conference on Human Factors in Computing Systems (CHI 2019

Supporting Students’ Research Writing in Psychology through Argument Diagramming

Arguing for the need for a scientific research study (i.e. writing an introduction to a research paper) poses significant challenges for students. When faced with these challenges, students often generate overly ‘safe’ studies, or replications, or in contrast include no strong support for their hypothesis. Additionally, instruction on argumentation has been slow to integrate into scientific education and discourse. This raises the question—how can we support novice scientists in generating and defending high quality hypotheses? A long history of research supports the affordances provided by spatial representations of complex information, particularly in the sciences. More recently, argument diagramming— the process of spatially representing an argument by its component parts and their relationships— has gained traction in instruction for philosophy, social studies, and law. However, its effectiveness for supporting students in science is relatively untested. Additionally, many of these studies have focused on basic contrasts between diagramming and no diagramming. The purpose of these studies was to test the effectiveness of argument diagrams for supporting students’ resea

Representational transformations : using maps to write essays

This research was supported by NSERC (The Natural Sciences and Engineering Research Council of Canada) RGPIN-2020-04401 and EPSRC (Engineering and Physical Sciences Research Council) EP/T518062/1.Essay-writing is a complex, cognitively demanding activity. Essay-writers must synthesise source texts and original ideas into a textual essay. Previous work found that writers produce better essays when they create effective intermediate representations. Diagrams, such as concept maps and argument maps, are particularly effective. However, there is insufficient knowledge about how people use these intermediate representations in their essay-writing workflow. Understanding these processes is critical to inform the design of tools to support workflows incorporating intermediate representations. We present the findings of a study, in which 20 students planned and wrote essays. Participants used a tool that we developed, Write Reason, which combines a free-form mapping interface with an essay-writing interface. This let us observe the types of intermediate representations participants built, and crucially, the process of how they used and moved between them. The key insight is that much of the important cognitive processing did not happen within a single representation, but instead in the processes that moved between multiple representations. We label these processes `representational transformations'. Our analysis characterises key properties of these transformations: cardinality, explicitness, and change in representation type. We also discuss research questions surfaced by the focus on transformations, and implications for tool designers.Publisher PDFPeer reviewe

Management implications of moving from a traditional structured systems development methodology to object-orientation

Thesis (M.S.) University of Alaska Fairbanks, 2003As software application systems become larger and more complex, many software employers and managers believe that the key to sustaining its competitive advantage in the computing technology market lies in its software engineering capabilities. Software crisis situation seems to be a common occurrence in the software development environment as systems become larger and more complex. Object Orientation (OO) has been proposed as a viable alternative to traditional approach (i.e., structured techniques), an approach that many hope will solve the current software crisis. 00 is a new paradigm, and it requires new types of knowledge, new specialists, and significant changes in the mindset, an entirely different way of thinking, representing and solving a problem. The transition of moving toward the 00 from the traditional approach may involve a high risk of failure if the managers do not understand the nature of paradigm shifts and do not anticipate the future. The problem of moving to 00 has become very important. An understanding of potential problems from migrating to the new paradigm helps managers make a smoother paradigm shift. The implications and challenges of the 00 paradigm are presented. The study suggests that Object-Oriented System Development (OOSD) requires more discipline, management and training than traditional software development does. Education and experience are keys for the success of any OOSD project

Legal knowledge-based systems: new directions in system design

This thesis examines and critiques the concept of 'legal knowledge-based’ systems. Work on legal knowledge-based systems is dominated by work in 'artificial intelligence and law’. It seeks to automate the application of law and to automate the solution of legal problems. Automation however, has proved elusive. In contrast to such automation, this thesis proposes the creation of legal knowledge-based systems based on the concept of augmentation of legal work. Focusing on systems that augment legal work opens new possibilities for system creation and use. To inform how systems might augment legal work, this thesis examines philosophy, psychology and legal theory for information they provide on how processes of legal reasoning operate. It is argued that, in contrast to conceptions of law adopted in artificial intelligence and law, 'sensemaking' provides a useful perspective with which to create systems. It is argued that visualisation, and particularly diagrams, are an important and under considered element of reasoning and that producing systems that support diagramming of processes of legal reasoning would provide useful support for legal work. This thesis reviews techniques for diagramming aspects of sensemaking. In particular this thesis examines standard methods for diagramming arguments and methods for diagramming reasoning. These techniques are applied in the diagramming of legal judgments. A review is conducted of systems that have been constructed to support the construction of diagrams of argument and reasoning. Drawing upon these examinations, this thesis highlights the necessity of appropriate representations for supporting reasoning. The literature examining diagramming for reasoning support provides little discussion of appropriate representations. This thesis examines theories of representation for insight they can provide into the design of appropriate representations. It is concluded that while the theories of representation that are examined do not determine what amounts to a good representation, guidelines for the design and choice of representations can be distilled. These guidelines cannot map the class of legal knowledge-based systems that augment legal sensemaking, they can however, be used to explore this class and to inform construction of systems

Improving Peer Feedback Prediction: The Sentence Level is Right

Recent research aims to automatically pre-dict whether peer feedback is of high qual-ity, e.g. suggests solutions to identified problems. While prior studies have fo-cused on peer review of papers, simi-lar issues arise when reviewing diagrams and other artifacts. In addition, previous studies have not carefully examined how the level of prediction granularity impacts both accuracy and educational utility. In this paper we develop models for predict-ing the quality of peer feedback regard-ing argument diagrams. We propose to perform prediction at the sentence level, even though the educational task is to la-bel feedback at a multi-sentential com-ment level. We first introduce a corpus annotated at a sentence level granularity, then build comment prediction models us-ing this corpus. Our results show that ag-gregating sentence prediction outputs to label comments not only outperforms ap-proaches that directly train on comment annotations, but also provides useful infor-mation for enhancing peer review systems with new functionality.

An investigation into computer support for cooperative work in software engineering groups

The research of this thesis relates to Computer Supported Cooperative Work (CSCW) in the context of software engineering, and in particular software engineering education. Whilst research into group working has tended to be directed towards CSCW, very little research has been undertaken on group working within software engineering. Linked with CSCW is groupware, which is the class of tools that supports and augments groupwork. This thesis represents an attempt to contribute to the understanding of the groupware needs of software engineers, and to identify and trial groupware that supports software engineering activities. An infrastructure has been developed providing virtual environments, for use by both collocated and geographically distributed software engineering students, to support their groupwork. This infrastructure comprises of synchronous and asynchronous groupware, in the form of desktop video conferencing, and a shared information workspace. This shared workspace has been tailored from the groupware tool, Basic Support for Cooperative Work (BSCW).Within this thesis, hypotheses have been formulated as to the student use of these virtual environments. These hypotheses concentrate on the areas of: organisation and coordination of tasks, the level of cooperation that occurs within the phases of the software lifecycle, the usage of the functions within a shared workspace, and what importance is placed on the role of synchronous communication within software engineering student groupwork. Through a series of case studies it was possible to determine the outcome of these hypotheses using various data collection methods. These methods include questionnaires, focus group meetings, observations, and automatic monitoring of workspace activities. The outcomes of this thesis are that the hypotheses regarding organisation and coordination, and, the role of synchronous communication within software engineering, have been proved. Whilst the determination of the level of cooperation during the phases of the software lifecycle has not been proved, the use of functions within the shared workspace has been partly proved

Sysinformatics & Systems Mimicry: New Fields Emerging from a “Science” of Systems Processes Engineering

AbstractThis paper gives an overview of the Systems Processes-Patterns/Systems Pathology (SP3) project of the INCOSE SSWG (Systems Science Working Group) because we see two new specialties arising from that research program. The project is based on research conducted over the last 300 years on natural systems in the seven natural sciences (astronomy, physics, chemistry, geology, biology, math and computers).1 It seeks to unify that knowledge with systems thinking from human and social systems research. The resulting integrated knowledge base (KB) is overwhelming, even when abstracted to only those 50 principles and pathologies that tell us something experimentally about how natural systems work or don’t work. The SP3 project collects facts in 25 categories (each described in this paper) and hundreds of Linkage Propositions (LPs) that explain how 55 systems processes influence each other to maintain systems health. We suggest a new approach called sysinformatics modeled after the bioinformatics courses that today prepare workers for genetics and systems biology. We need corresponding courses in sysinformatics to prepare workers in systems science, systems engineering, computer-based modeling, and new fields such as sustainability studies. The need for bioinformatics resulted from a range of advances in new experimental tools and techniques that led to terabytes of data that required meaningful analysis to bring from the lab bench to the hospital bed. Similarly, sysinformatics would attempt to make more meaningful the mountains of data and relations resulting from a new science of systems. It will require active participation of the computer sciences, engineering specialties, and deep mathematics. Overall, it would help in understanding of how systems processes work and don’t work. Sysinformatics would be a rigorously transdisciplinary field aimed at the discovery, storage, retrieval, organization, classification, and analysis of systems science data. It would stimulate discovery of new tools and algorithms for analysis of data, encourage invention of new ways to present and display data in ways more easily or intuitively meaningful to humans (like the Circos maps of genomics that interpret many different whole genome changes). Sysinformatics would add data mining, AI, simulation image processing, a range of algorithms, new companies and computerized tools to the toolboxes of practicing systems engineers. This paper describes initial similarities and differences between bioinformatics and sysinformatics. It also introduces “systems mimicry” as a potential new specialty and suggests similarities and differences between biomimicry and systems mimicry