When Automated Assessment Meets Automated Content Generation: Examining Text Quality in the Era of GPTs

The use of machine learning (ML) models to assess and score textual data has become increasingly pervasive in an array of contexts including natural language processing, information retrieval, search and recommendation, and credibility assessment of online content. A significant disruption at the intersection of ML and text are text-generating large-language models such as generative pre-trained transformers (GPTs). We empirically assess the differences in how ML-based scoring models trained on human content assess the quality of content generated by humans versus GPTs. To do so, we propose an analysis framework that encompasses essay scoring ML-models, human and ML-generated essays, and a statistical model that parsimoniously considers the impact of type of respondent, prompt genre, and the ML model used for assessment model. A rich testbed is utilized that encompasses 18,460 human-generated and GPT-based essays. Results of our benchmark analysis reveal that transformer pretrained language models (PLMs) more accurately score human essay quality as compared to CNN/RNN and feature-based ML methods. Interestingly, we find that the transformer PLMs tend to score GPT-generated text 10-15\% higher on average, relative to human-authored documents. Conversely, traditional deep learning and feature-based ML models score human text considerably higher. Further analysis reveals that although the transformer PLMs are exclusively fine-tuned on human text, they more prominently attend to certain tokens appearing only in GPT-generated text, possibly due to familiarity/overlap in pre-training. Our framework and results have implications for text classification settings where automated scoring of text is likely to be disrupted by generative AI.Comment: Data available at: https://github.com/nd-hal/automated-ML-scoring-versus-generatio

Journal Self-Citation III: Exploring the Self-Citation Patterns in MIS Journals

Self-citation is a common practice in the research community. It includes citing one’s own papers and one’s target journal. A recent action by a publisher requesting each author of its journal to cite at least five papers published by the journal calls for a study of the self-citation patterns in MIS journals. This study intends to examine the cited table and the citing table in the database of Journal Citation Reports and identify the self-cited and self-citing patterns of MIS journals included in this database. Through a descriptive analysis, influential as well as problematic journals are identified and the implications for journal stakeholders are discussed

Calendar.help: Designing a Workflow-Based Scheduling Agent with Humans in the Loop

Although information workers may complain about meetings, they are an essential part of their work life. Consequently, busy people spend a significant amount of time scheduling meetings. We present Calendar.help, a system that provides fast, efficient scheduling through structured workflows. Users interact with the system via email, delegating their scheduling needs to the system as if it were a human personal assistant. Common scheduling scenarios are broken down using well-defined workflows and completed as a series of microtasks that are automated when possible and executed by a human otherwise. Unusual scenarios fall back to a trained human assistant who executes them as unstructured macrotasks. We describe the iterative approach we used to develop Calendar.help, and share the lessons learned from scheduling thousands of meetings during a year of real-world deployments. Our findings provide insight into how complex information tasks can be broken down into repeatable components that can be executed efficiently to improve productivity.Comment: 10 page

Preserving Communication Context. Virtual workspace and interpersonal space in Japanese CSCW.

The past decade has seen the development of a perspective\ud holding that technology is socially constructed (Mackenzie and Wacjman, 1985; Bijker, Hughes and Pinch, 1987; Bijker and Law, 1992). This paper examines the social construction of one group of technologies, systems for computer supported cooperative work (CSCW). It describes the design of CSCW in Japan, with particular attention to the influence of culture on the design process. Two case studies are presented to illustrate the argument that culture is an important factor in technology design, despite commonly held assumptions about the neutrality and objectivity of science and technology. The paper further argues that, by looking at\ud CSCW systems as texts which reflect the context of their production and the society from which they come, we may be better able to understand the transformations that operate when these texts are “read” in the contexts of their implementation

As naturezas linguísticas dos sistemas computacionais

Livro de homenagem à professora Maria Emília Ricardo MarquesA partir do momento em que os sistemas computacionais, vulgo computadores e respectivos programas, desenvolveram a capacidade de interacção com os seus utilizadores humanos, generalizou-se a adopção de modelos linguísticos como ponto de partida para a concepção (design) desses sistemas. O desenvolvimento das modalidades de interacção enriqueceu o leque de referências conceptuais com princípios semióticos. A integração de sistemas computacionais em espaços de colaboração interpessoal sugeriu uma ligação forte às perspectivas da teoria da linguagem que a colocam no espaço da acção. Linguagem, acção, comunicação e interpretação são dimensões centrais de toda a concepção dos actuais e futuros sistemas informáticos

Information scraps: how and why information eludes our personal information management tools

In this paper we describe information scraps -- a class of personal information whose content is scribbled on Post-it notes, scrawled on corners of random sheets of paper, buried inside the bodies of e-mail messages sent to ourselves, or typed haphazardly into text files. Information scraps hold our great ideas, sketches, notes, reminders, driving directions, and even our poetry. We define information scraps to be the body of personal information that is held outside of its natural or We have much still to learn about these loose forms of information capture. Why are they so often held outside of our traditional PIM locations and instead on Post-its or in text files? Why must we sometimes go around our traditional PIM applications to hold on to our scraps, such as by e-mailing ourselves? What are information scraps' role in the larger space of personal information management, and what do they uniquely offer that we find so appealing? If these unorganized bits truly indicate the failure of our PIM tools, how might we begin to build better tools? We have pursued these questions by undertaking a study of 27 knowledge workers. In our findings we describe information scraps from several angles: their content, their location, and the factors that lead to their use, which we identify as ease of capture, flexibility of content and organization, and avilability at the time of need. We also consider the personal emotive responses around scrap management. We present a set of design considerations that we have derived from the analysis of our study results. We present our work on an application platform, jourknow, to test some of these design and usability findings