Simulating student mistakes to evaluate the fairness of automated grading

The use of autograding to assess programming students may lead to unfairness if an autograder is incorrectly configured. Mutation analysis offers a potential solution to this problem. By simulating student coding mistakes, an automated technique can evaluate the fairness and completeness of an autograding configuration. In this paper, we introduce a set of mutation operators to be used in such a technique, derived from a mistake classification of real student solutions for two introductory programming tasks

Generative AI and Its Educational Implications

We discuss the implications of generative AI on education across four critical sections: the historical development of AI in education, its contemporary applications in learning, societal repercussions, and strategic recommendations for researchers. We propose ways in which generative AI can transform the educational landscape, primarily via its ability to conduct assessment of complex cognitive performances and create personalized content. We also address the challenges of effective educational tool deployment, data bias, design transparency, and accurate output verification. Acknowledging the societal impact, we emphasize the need for updating curricula, redefining communicative trust, and adjusting to transformed social norms. We end by outlining the ways in which educational stakeholders can actively engage with generative AI, develop fluency with its capacities and limitations, and apply these insights to steer educational practices in a rapidly advancing digital landscape.Comment: This is a preprint version of an edited book chapter to appear in Kourkoulou, D., O. Tzirides, B. Cope, M. Kalantzis, (eds) (2024). Trust and Inclusion in AI-Mediated Education: Where Human Learning Meets Learning Machines, Springe

Evaluating Large Language Models on the GMAT: Implications for the Future of Business Education

The rapid evolution of artificial intelligence (AI), especially in the domain of Large Language Models (LLMs) and generative AI, has opened new avenues for application across various fields, yet its role in business education remains underexplored. This study introduces the first benchmark to assess the performance of seven major LLMs, OpenAI's models (GPT-3.5 Turbo, GPT-4, and GPT-4 Turbo), Google's models (PaLM 2, Gemini 1.0 Pro), and Anthropic's models (Claude 2 and Claude 2.1), on the GMAT, which is a key exam in the admission process for graduate business programs. Our analysis shows that most LLMs outperform human candidates, with GPT-4 Turbo not only outperforming the other models but also surpassing the average scores of graduate students at top business schools. Through a case study, this research examines GPT-4 Turbo's ability to explain answers, evaluate responses, identify errors, tailor instructions, and generate alternative scenarios. The latest LLM versions, GPT-4 Turbo, Claude 2.1, and Gemini 1.0 Pro, show marked improvements in reasoning tasks compared to their predecessors, underscoring their potential for complex problem-solving. While AI's promise in education, assessment, and tutoring is clear, challenges remain. Our study not only sheds light on LLMs' academic potential but also emphasizes the need for careful development and application of AI in education. As AI technology advances, it is imperative to establish frameworks and protocols for AI interaction, verify the accuracy of AI-generated content, ensure worldwide access for diverse learners, and create an educational environment where AI supports human expertise. This research sets the stage for further exploration into the responsible use of AI to enrich educational experiences and improve exam preparation and assessment methods

AI ethics and higher education : good practice and guidance for educators, learners, and institutions

Artificial intelligence (AI) is exerting unprecedented pressure on the global higher educational landscape in transforming recruitment processes, subverting traditional pedagogy, and creating new research and institutional opportunities. These technologies require contextual and global ethical analysis so that they may be developed and deployed in higher education in just and responsible ways. To-date, these efforts have been largely focused on small parts of the educational environments leaving most of the world out of an essential contribution. This volume acts as a corrective to this and contributes to the building of competencies in ethics education and to broader, global debates about how AI will transform various facets of our lives, not the least of which is higher education

Using Blockchain to Ensure Reputation Credibility in Decentralized Review Management

In recent years, there have been incidents which decreased people's trust in some organizations and authorities responsible for ratings and accreditation. For a few prominent examples, there was a security breach at Equifax (2017), misconduct was found in the Standard & Poor's Ratings Services (2015), and the Accrediting Council for Independent Colleges and Schools (2022) validated some of the low-performing schools as delivering higher standards than they actually were. A natural solution to these types of issues is to decentralize the relevant trust management processes using blockchain technologies. The research problems which are tackled in this thesis consider the issue of trust in reputation for assessment and review credibility at different angles, in the context of blockchain applications. We first explored the following questions. How can we trust courses in one college to provide students with the type and level of knowledge which is needed in a specific workplace? Micro-accreditation on a blockchain was our solution, including using a peer-review system to determine the rigor of a course (through a consensus). Rigor is the level of difficulty in regard to a student's expected level of knowledge. Currently, we make assumptions about the quality and rigor of what is learned, but this is prone to human bias and misunderstandings. We present a decentralized approach that tracks student records throughout the academic progress at a school and helps to match employers' requirements to students' knowledge. We do this by applying micro-accredited topics and Knowledge Units (KU) defined by NSA's Center of Academic Excellence to courses and assignments. We demonstrate that the system was successful in increasing accuracy of hires through simulated datasets, and that it is efficient, as well as scalable. Another problem is how can we trust that the peer reviews are honest and reflect an accurate rigor score? Assigning reputation to peers is a natural method to ensure correctness of these assessments. The reputation of the peers providing rigor scores needs to be taken into account for an overall rigor of a course, its topics, and its tasks. Specifically, those with a higher reputation should have more influence on the total score. Hence, we focused on how a peer's reputation is managed. We explored decentralized reputation management for the peers, choosing a decentralized marketplace as a sample application. We presented an approach to ensuring review credibility, which is a particular aspect of trust in reviews and reputation of the parties who provide them. We use a Proof-of-Stake based Algorand system as a base of our implementation, since this system is open-source, and it has a rich community support. Specifically, we directly map reputation to stake, which allows us to deploy Algorand at the blockchain layer. Reviews are analyzed by the proposed evaluation component using Natural Language Processing (NLP). In our system, NLP gauges the positivity of the written review, compares that value to a scaled numerical rating given, and determines adjustments to a peer's reputation from that result. We demonstrate that this architecture ensures credible and trustworthy assessments. It also efficiently manages the reputation of the peers, while keeping reasonable consensus times. We then turned our focus on ensuring that a peer's reputation is credible. This led us to introducing a new type of consensus called "Proof-of-Review". Our proposed implementation is again based on Algorand, since its modular architecture allows for easy modifications, such as adding extra components, but this time, we modified the engine. The proposed model then provides a trust in evaluations (review and assessment credibility) and in those who provide them (reputation credibility) using a blockchain. We introduce a blacklisting component, which prevents malicious nodes from participating in the protocol, and a minimum-reputation component, which limits the influence of under-performing users. Our results showed that the proposed blockchain system maintains liveliness and completeness. Specifically, blacklisting and the minimum-reputation requirement (when properly tuned) do not affect these properties. We note that the Proof-of-Review concept can be deployed in other types of applications with similar needs of trust in assessments and the players providing them, such as sensor arrays, autonomous car groups (caravans), marketplaces, and more

Developing an online support tool to assist students in higher education with project proposals

The research presented in this thesis investigates ways to assist students with writing their project proposals. There is limited literature on the problems students have when writing project proposals in Higher Education. Particularly most of the literature has concentrated on the writing aspects, rhetorical aspects and structure of a scientific article. Even though various studies on assessment of undergraduate individual and group project works have been done, the project proposal has not been given much attention. Therefore assessment of the proposal stage of the undergraduate final year project becomes the focus of this study, conducted over three years. This three-phase study directly involved three main stakeholders (students, supervisors and coordinators) in the overall process. In Phase 1, the existence of the proposal problems was investigated and identified from the perceptions of the students and supervisors. Possible solutions to the proposal problems were identified. Next Phase 2, I acknowledged the requirements of the stakeholders, which provided the framework and initiated the design and development of an eGuide, a self-paced online guide. The implementation and evaluation of the eGuide were then conducted in this phase. Finally Phase 3, the study emphasised improvement to practice focusing on the Degree final year project by utilizing the cyclic approach of an action research. Questionnaires and focus groups were used to gather information from students and supervisors, both to identify the problems they perceived with the student project proposal process and the effectiveness of the online support tool, eGuide. In the development of the eGuide, it proved necessary to design and pilot a robust rubric for students and supervisors to structure the project proposal process. The eGuide was evaluated for its effectiveness by the various users and followed by an action research approach to make further improvements to the Degree final year project curriculum. The assessment criteria evolved further to become a marking template with a very effective feedback tool. The study has a stimulating effect on the practices of how supervision of project proposal was shaped and how the project proposal was being assessed. Practical outcome of the study ultimately benefits not only the students who were the focus in the first place but also the supervisors and the coordinators. The study provides further avenues for research opportunities in this area to take place in the future