The Role of Process and Outcome Accountability Claims for Shaping AI Developers’ Perceived Accountability

As accountability becomes increasingly important for developers of artificial intelligence (AI)-based systems, governance mechanisms such as AI principles or audits are often criticized for not sufficiently influencing AI developers. Therefore, we examine how visualized arguments in user interfaces (UIs) of integrated development environments (IDEs) can increase AI developers’ perceived accountability. Combining construal level theory and Toulmin’s model of argumentation, four UI design artifacts were developed, each containing a claim of process or outcome accountability with or without monitoring and evaluation tools that act as claim-supporting data. Results of an online experiment with 164 AI developers show that claiming process accountability increases AI developers’ perceived accountability more than claiming outcome accountability, both without supporting data. However, when supporting data are available, both claims increase AI developers’ perceived accountability comparably effectively. The study’s results highlight the theoretical and practical usefulness of visualized arguments in UIs of IDEs to promote AI developers’ accountability

Unleashing the Potential of Argument Mining for IS Research: A Systematic Review and Research Agenda

Argument mining (AM) represents the unique use of natural language processing (NLP) techniques to extract arguments from unstructured data automatically. Despite expanding on commonly used NLP techniques, such as sentiment analysis, AM has hardly been applied in information systems (IS) research yet. Consequentially, knowledge about the potentials for the usage of AM on IS use cases appears to be still limited. First, we introduce AM and its current usage in fields beyond IS. To address this research gap, we conducted a systematic literature review on IS literature to identify IS use cases that can potentially be extended with AM. We develop eleven text-based IS research topics that provide structure and context to the use cases and their AM potentials. Finally, we formulate a novel research agenda to guide both researchers and practitioners to design, compare and evaluate the use of AM for text-based applications and research streams in IS

Towards an authentic argumentation literacy test

A central goal of education is to improve argumentation literacy. How do we know how well this goal is achieved? Can we measure argumentation literacy? The present study is a preliminary step towards measuring the efficacy of education with regards to argumentation literacy. Tests currently in use to determine critical thinking skills are often similar to IQ-tests in that they predominantly measure logical and mathematical abilities. Thus, they may not measure the various other skills required in understanding authentic argumentation. To identify the elements of argumentation literacy, this exploratory study begins by surveying introductory textbooks within argumentation theory, critical thinking, and rhetoric. Eight main abilities have been identified. Then, the study outlines an Argumentation Literacy Test that would comprise these abilities suggested by the literature. Finally, the study presents results from a pilot of a version of such a test and discusses needs for further development

Using problem-centered learning to foster argumentation in introductory sociology

This mixed methods case study explored the efficacy of a problem-based learning environment and three different instructional methods designed to foster argumentation in an introductory sociology course. While no statistically significant differences were found in the overall assignment scores among the three treatment groups, there were significant differences in the effort they expended to achieve those scores, suggesting that a treatment in which students were instructed to generate counter-arguments to a provided model essay was most efficient while a treatment in which students were guided to construct an argument step-by-step encouraged more time-on-task

Examining Scholarly Influence: A Study in Hirsch Metrics and Social Network Analysis

This dissertation research is focused on how we, as researchers, ‘influence’ others researchers. In particular, I am concerned with the notion of what constitutes the ‘influence’ of a scholar and how ‘influence’ is conferred upon scholars. This research is concerned with the construct called ‘scholarly influence’. Scholarly influence is of interest because a clear “theory of scholarly influence” does not yet exist. Rather a number of surrogate measures or concepts that are variable are used to evaluate the value of one’s academic work. ‘Scholarly influence’ is broken down into ‘ideational influence’ or the influence that one has through publication and the uptake of the ideas presented in the publication, and ‘social influence’ or the influence that one has through working with other researchers. Finally through the use of the definition of ‘scholarly influence’ this dissertation tries to commence a definition of ‘quality’ in scholarly work

Myside Bias Shifting in the Written Arguments of First Year Composition Students

This dissertation reports on research conducted to better understand how college student writers learned to work against their own biases as they researched and wrote arguments. I conducted a review of former studies to design a curriculum that would help students avoid bias and increase their ability to write arguments tailored to specific readers in ways that accomplish their goals. This review also informed the kinds of data to be collected and analyzed in order to accomplish the research goal, which was to understand whether and how each of seven students enrolled in a composition course reduced their biases. I collected written arguments, drawings, and classroom discussions of these students and administered surveys, and participants underwent interviews, to study the effect of the curriculum and instruction. This dissertation reports findings on how each student writer’s bias shifted differently over the course of the semester, and the role identity played in bias shifting. Results include the observation that the curriculum was effective at reducing bias in student arguments, though to various degrees and for differing reasons, based on a variety of contextual factors. Unlike experimental studies of bias, this study provides rich details about seven individual students’ experiences in a course designed to reduce bias. Implications include researched evidence upon which teachers, administrators, curriculum designers, and policymakers may base future decisions upon regarding the teaching of argumentation

The impact of National Science Foundation investments in undergraduate engineering education research: A comparative, mixed methods study

The U.S. invests billions of taxpayers\u27 dollars in research tied to the national priorities that contribute to its competitiveness in a global economy. As the federal funding agency with an explicit focus on engineering education, the National Science Foundation (NSF) contains a portfolio of projects focused on improving the quantity of engineering graduates and the quality of engineering programs. Within the agency, the Division of Undergraduate Education invests approximately $190 million (FY 2012) annually on science, technology, engineering and mathematics (STEM) education projects. Although the DUE portfolio includes a suite a projects with different foci supporting national initiatives and Principal Investigators (PIs) report their results in annual reports and conferences, there is little consistency on how impact is defined, evaluated, and measured. ^ While many agree on the importance of investing in research, the stiff economic climate necessitates that the research that demonstrates impact is what will continue to be supported. However, the dearth of scholarship on impact contributes to the lack of understanding around this topic. This study links the fragmented literature on impact to form a unified starting point for continuing the conversation. While existing literature includes three dimensions of research impact (i.e., scientific, societal, and domain-specific impact), this study focuses on the domain-specific impacts of engineering education research using two guiding frameworks, namely, Toulmin\u27s Model (1958) and the Common Guidelines for Education Research and Development (Earle et al., 2013), and a multiphase mixed methods research design (Creswell & Plano Clark, 2011).^ The qualitative phase of this study explores how researchers on NSF-funded STEM education R&D projects talk about the impact of their work; the findings reveal eight themes that are commonly discussed when PIs articulate the impact of their research, and two themes related to how they support their claims. The findings also indicate that the STEM discipline associated with the study and the project focus have more to do with the types of impact PIs claim than the amount of funding awarded to the project. As a result of identifying the points of alignment between PIs\u27 perspectives on impact and existing literature, a preliminary description of what impact looks like in this context is proposed—using the three dimensions of research impact as an organizing framework. Although this study puts forth a preliminary description of the impact of STEM education research, extensions of this work are necessary before providing practitioners and policymakers with a valid, comprehensive framework characterizing what impact means in this context.^ Ideas supporting the types of claims PIs make when discussing the impact of their work were used to develop a survey that was distributed to a small sample of current and former NSF Program Officers (POs) in the second phase of this study. The survey results provide preliminary evidence on how PIs and NSF PO\u27 perspectives on research impact compare, and affirm that additional studies are needed. Consequently, implications for policy and practice and potential research directions are also discussed

The public sphere according to UK stem cell scientists

In this thesis the concept of social representations is made relevant to the study of the ‘public sphere’ according to scientists. This is elaborated by the re-examination of the notion of a ‘consensual’ and a ‘reified universe’ substantiating a more sociopsychological approach in the study of relevant phenomena. Two processes generate social representations of the public: anchoring and objectification. The empirical study investigates the scientists’ views of the public sphere, in relation to public perceptions, media coverage and the regulation of cloning technology. Elite media coverage of the stem cell debate and conversations with stem cell scientists are systematically analysed with multiple methods. Findings are based on 461 news articles that appeared in Nature and Science between 1997 and 2005 and on interviews with 18 U.K based stem cell researchers conducted between February and October 2005. The analysis compares the debate before and after the ‘stem cell war’ of 2002, and typifies a high tension in representing the public sphere, elaborated in metaphors and prevailing arguments. Central elements of the representation assume a strong disassociation of science from the public sphere; peripheral elements operate with a degree of blurring of those same boundaries, which recognises a common project. This representation, while being expressive of its context of production, constitutes a functional response to it

Content Selection for Effective Counter-Argument Generation

The information ecosystem of social media has resulted in an abundance of opinions on political topics and current events. In order to encourage better discussions, it is important to promote high-quality responses and relegate low-quality ones. We thus focus on automatically analyzing and generating counter-arguments in response to posts on social media with the goal of providing effective responses. This thesis is composed of three parts. In the first part, we conduct an analysis of arguments. Specifically, we first annotate discussions from Reddit for aspects of arguments and then analyze them for their persuasive impact. Then we present approaches to identify the argumentative structure of these discussions and predict the persuasiveness of an argument. We evaluate each component independently using automatic or manual evaluations and show significant improvement in each. In the second part, we leverage our discoveries from our analysis in the process of generating counter-arguments. We develop two approaches in the retrieve-and-edit framework, where we obtain content using methods created during our analysis of arguments, among others, and then modify the content using techniques from natural language generation. In the first approach, we develop an approach to retrieve counter-arguments by annotating a dataset for stance and building models for stance prediction. Then we use our approaches from our analysis of arguments to extract persuasive argumentative content before modifying non-content phrases for coherence. In contrast, in the second approach we create a dataset and models for modifying content -- making semantic edits to a claim to have a contrasting stance. We evaluate our approaches using intrinsic automatic evaluation of our predictive models and an overall human evaluation of our generated output. Finally, in the third part, we discuss the semantic challenges of argumentation that we need to solve in order to make progress in the understanding of arguments. To clarify, we develop new methods for identifying two types of semantic relations -- causality and veracity. For causality, we build a distant-labeled dataset of causal relations using lexical indicators and then we leverage features from those indicators to build predictive models. For veracity, we build new models to retrieve evidence given a claim and predict whether the claim is supported by that evidence. We also develop a new dataset for veracity to illuminate the areas that need progress. We evaluate these approaches using automated and manual techniques and obtain significant improvement over strong baselines. Finally, we apply these techniques to claims in the domain of household electricity consumption, mining claims using our methods for causal relations and then verifying their truthfulness