25 research outputs found
Learnersourcing Personalized Hints
Personalized support for students is a gold standard in education, but it scales poorly with the number of students. Prior work on learnersourcing presented an approach for learners to engage in human computation tasks while trying to learn a new skill. Our key insight is that students, through their own experience struggling with a particular problem, can become experts on the particular optimizations they implement or bugs they resolve. These students can then generate hints for fellow students based on their new expertise. We present workflows that harvest and organize studentsâ collective knowledge and advice for helping fellow novices through design problems in engineering. Systems embodying each workflow were evaluated in the context of a college-level computer architecture class with an enrollment of more than two hundred students each semester. We show that, given our design choices, students can create helpful hints for their peers that augment or even replace teachersâ personalized assistance, when that assistance is not available
Leveraging Learners for Teaching Programming and Hardware Design at Scale
In a massive open online course (MOOC), a single pro-gramming or digital hardware design exercise may yield thousands of student solutions that vary in many ways, some superï¬ cial and some fundamental. Understanding large-scale variation in student solutions is a hard but important problem. For teachers, this variation can be a source of pedagogically valuable examples and expose corner cases not yet covered by autograding. For students, the variation in a large class means that other students may have struggled along a similar solution path, hit the same bugs, and can offer hints based on that earned expertise. We developed three systems to take advantage of the solu-tion variation in large classes, using program analysis and learnersourcing. All three systems have been evaluated using data or live deployments in on-campus or edX courses with thousands of students
Robosourcing Educational Resources -- Leveraging Large Language Models for Learnersourcing
In this article, we introduce and evaluate the concept of robosourcing for
creating educational content. Robosourcing lies in the intersection of
crowdsourcing and large language models, where instead of a crowd of humans,
requests to large language models replace some of the work traditionally
performed by the crowd. Robosourcing includes a human-in-the-loop to provide
priming (input) as well as to evaluate and potentially adjust the generated
artefacts; these evaluations could also be used to improve the large language
models. We propose a system to outline the robosourcing process. We further
study the feasibility of robosourcing in the context of education by conducting
an evaluation of robosourced and programming exercises, generated using OpenAI
Codex. Our results suggest that robosourcing could significantly reduce human
effort in creating diverse educational content while maintaining quality
similar to human-created content
Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book
Advances in natural language processing have resulted in large language
models (LLMs) that are capable of generating understandable and sensible
written text. Recent versions of these models, such as OpenAI Codex and GPT-3,
can generate code and code explanations. However, it is unclear whether and how
students might engage with such explanations. In this paper, we report on our
experiences generating multiple code explanation types using LLMs and
integrating them into an interactive e-book on web software development. We
modified the e-book to make LLM-generated code explanations accessible through
buttons next to code snippets in the materials, which allowed us to track the
use of the explanations as well as to ask for feedback on their utility. Three
different types of explanations were available for students for each
explainable code snippet; a line-by-line explanation, a list of important
concepts, and a high-level summary of the code. Our preliminary results show
that all varieties of explanations were viewed by students and that the
majority of students perceived the code explanations as helpful to them.
However, student engagement appeared to vary by code snippet complexity,
explanation type, and code snippet length. Drawing on our experiences, we
discuss future directions for integrating explanations generated by LLMs into
existing computer science classrooms
Automatically Generating CS Learning Materials with Large Language Models
Recent breakthroughs in Large Language Models (LLMs), such as GPT-3 and
Codex, now enable software developers to generate code based on a natural
language prompt. Within computer science education, researchers are exploring
the potential for LLMs to generate code explanations and programming
assignments using carefully crafted prompts. These advances may enable students
to interact with code in new ways while helping instructors scale their
learning materials. However, LLMs also introduce new implications for academic
integrity, curriculum design, and software engineering careers. This workshop
will demonstrate the capabilities of LLMs to help attendees evaluate whether
and how LLMs might be integrated into their pedagogy and research. We will also
engage attendees in brainstorming to consider how LLMs will impact our field.Comment: In Proceedings of the 54th ACM Technical Symposium on Computing
Science Educatio
Creating Systems and Applying Large-Scale Methods to Improve Student Remediation in Online Tutoring Systems in Real-time and at Scale
A common problem shared amongst online tutoring systems is the time-consuming nature of content creation. It has been estimated that an hour of online instruction can take up to 100-300 hours to create. Several systems have created tools to expedite content creation, such as the Cognitive Tutors Authoring Tool (CTAT) and the ASSISTments builder. Although these tools make content creation more efficient, they all still depend on the efforts of a content creator and/or past historical. These tools do not take full advantage of the power of the crowd. These issues and challenges faced by online tutoring systems provide an ideal environment to implement a solution using crowdsourcing. I created the PeerASSIST system to provide a solution to the challenges faced with tutoring content creation. PeerASSIST crowdsources the work students have done on problems inside the ASSISTments online tutoring system and redistributes that work as a form of tutoring to their peers, who are in need of assistance. Multi-objective multi-armed bandit algorithms are used to distribute student work, which balance exploring which work is good and exploiting the best currently known work. These policies are customized to run in a real-world environment with multiple asynchronous reward functions and an infinite number of actions. Inspired by major companies such as Google, Facebook, and Bing, PeerASSIST is also designed as a platform for simultaneous online experimentation in real-time and at scale. Currently over 600 teachers (grades K-12) are requiring students to show their work. Over 300,000 instances of student work have been collected from over 18,000 students across 28,000 problems. From the student work collected, 2,000 instances have been redistributed to over 550 students who needed help over the past few months. I conducted a randomized controlled experiment to evaluate the effectiveness of PeerASSIST on student performance. Other contributions include representing learning maps as Bayesian networks to model student performance, creating a machine-learning algorithm to derive student incorrect processes from their incorrect answer and the inputs of the problem, and applying Bayesian hypothesis testing to A/B experiments. We showed that learning maps can be simplified without practical loss of accuracy and that time series data is necessary to simplify learning maps if the static data is highly correlated. I also created several interventions to evaluate the effectiveness of the buggy messages generated from the machine-learned incorrect processes. The null results of these experiments demonstrate the difficulty of creating a successful tutoring and suggest that other methods of tutoring content creation (i.e. PeerASSIST) should be explored
A Data-Driven Framework for Team Formation for Maintenance Tasks
Even as maintenance evolves with new technologies, it is still a heavily human-driven domain; multiple steps in the maintenance workflow still require human expertise and intervention. Various maintenance activities require multiple maintainers, all with different skill sets and expertise, and from various positions and levels within the organization. Responding to maintenance requests, training exercises, or executing larger maintenance projects all can require maintenance teams. Having the correct assortment of individuals both in terms of skills and management experience can help improve the efficiency of these maintenance tasks. This paper presents a workflow for creating teams of maintainers by adapting accepted practices from the human-computer interaction (HCI) community. These steps provide a low-cost solution to help account for the needs of maintainers and their management, while matching skills of the maintainers with the needs of the activity
Impact of Guidance and Interaction Strategies for LLM Use on Learner Performance and Perception
Personalized chatbot-based teaching assistants can be crucial in addressing
increasing classroom sizes, especially where direct teacher presence is
limited. Large language models (LLMs) offer a promising avenue, with increasing
research exploring their educational utility. However, the challenge lies not
only in establishing the efficacy of LLMs but also in discerning the nuances of
interaction between learners and these models, which impact learners'
engagement and results. We conducted a formative study in an undergraduate
computer science classroom (N=145) and a controlled experiment on Prolific
(N=356) to explore the impact of four pedagogically informed guidance
strategies and the interaction between student approaches and LLM responses.
Direct LLM answers marginally improved performance, while refining student
solutions fostered trust. Our findings suggest a nuanced relationship between
the guidance provided and LLM's role in either answering or refining student
input. Based on our findings, we provide design recommendations for optimizing
learner-LLM interactions