4 research outputs found
Kattis vs ChatGPT: Assessment and Evaluation of Programming Tasks in the Age of Artificial Intelligence
AI-powered education technologies can support students and teachers in computer science education. However, with the recent developments in generative AI, and especially the increasingly emerging popularity of ChatGPT, the effectiveness of using large language models for solving programming tasks has been underexplored. The present study examines ChatGPTâs ability to generate code solutions at different difficulty levels for introductory programming courses. We conducted an experiment where ChatGPT was tested on 127 randomly selected programming problems provided by Kattis, an automatic software grading tool for computer science programs, often used in higher education. The results showed that ChatGPT independently could solve 19 out of 127 programming tasks generated and assessed by Kattis. Further, ChatGPT was found to be able to generate accurate code solutions for simple problems but encountered difficulties with more complex programming tasks. The results contribute to the ongoing debate on the utility of AI-powered tools in programming education
Can chatbots solve coding assignments assessed by automatic grading tools in higher education? : A case of ChatGPT
The present study examines ChatGPT-3's ability to generate code solutions for introductory programming courses in computer science and the potential implications for academic integrity. An experiment was conducted where ChatGPT was tested on programming problems from Kattis, an automatic software grading tool for computer programs, used in higher education. The results showed that ChatGPT independently could solve 19 out of 127 programming tasks assessed by Kattis. The studyâs results also show that ChatGPT could generate accurate code solutions for simple problems on Kattis but encounters difficulties with more complex programming tasks. A qualitative follow up investigation was also carried out. To provide comments on methodology and discuss cheating in higher education concerning programming courses the two teachers were interviewed. The Kattis system is considered to have useful features for preventing cheating, such as hidden test cases, but it also has limitations in detecting AI-generated code. The report concludes by discussing the implications for various stakeholders, including teachers, students, and researchers.Studien undersöker ChatGPT-3:s förmĂ„ga att generera kodlösningar för grundlĂ€ggande programmeringskurser inom datavetenskap och de potentiella konsekvenserna för akademisk integritet. Ett experiment utfördes dĂ€r ChatGPT testades med programmeringsproblem frĂ„n Kattis, ett automatiskt rĂ€ttningsverktyg för datorprogram som anvĂ€nds inom högre utbildning. Resultaten visade att ChatGPT sjĂ€lvstĂ€ndigt löste 19 av 127 programmeringsuppgifter som bedömdes av Kattis. Studien konstaterar att ChatGPT kan generera korrekta kodlösningar för problem med lĂ„g svĂ„righetsgrad enligt Kattis, men stöter pĂ„ svĂ„righeter med mer komplexa programmeringsuppgifter. En kvalitativ uppföljningsundersökning genomfördes Ă€ven dĂ€r tvĂ„ lĂ€rare frĂ„n KTH intervjuades för att ge sina kommentarer om metodvalet och diskutera fusket inom högre utbildning nĂ€r det gĂ€ller programmeringskurser. Kattis-systemet anses ha anvĂ€ndbara funktioner för att förhindra fusk, sĂ„som dolda testfall, men har ocksĂ„ begrĂ€nsningar nĂ€r det gĂ€ller att upptĂ€cka AI-genererad kod. Rapporten avslutas med att diskutera implikationerna för olika intressenter, inklusive lĂ€rare, studenter och forskare
Can chatbots solve coding assignments assessed by automatic grading tools in higher education? : A case of ChatGPT
The present study examines ChatGPT-3's ability to generate code solutions for introductory programming courses in computer science and the potential implications for academic integrity. An experiment was conducted where ChatGPT was tested on programming problems from Kattis, an automatic software grading tool for computer programs, used in higher education. The results showed that ChatGPT independently could solve 19 out of 127 programming tasks assessed by Kattis. The studyâs results also show that ChatGPT could generate accurate code solutions for simple problems on Kattis but encounters difficulties with more complex programming tasks. A qualitative follow up investigation was also carried out. To provide comments on methodology and discuss cheating in higher education concerning programming courses the two teachers were interviewed. The Kattis system is considered to have useful features for preventing cheating, such as hidden test cases, but it also has limitations in detecting AI-generated code. The report concludes by discussing the implications for various stakeholders, including teachers, students, and researchers.Studien undersöker ChatGPT-3:s förmĂ„ga att generera kodlösningar för grundlĂ€ggande programmeringskurser inom datavetenskap och de potentiella konsekvenserna för akademisk integritet. Ett experiment utfördes dĂ€r ChatGPT testades med programmeringsproblem frĂ„n Kattis, ett automatiskt rĂ€ttningsverktyg för datorprogram som anvĂ€nds inom högre utbildning. Resultaten visade att ChatGPT sjĂ€lvstĂ€ndigt löste 19 av 127 programmeringsuppgifter som bedömdes av Kattis. Studien konstaterar att ChatGPT kan generera korrekta kodlösningar för problem med lĂ„g svĂ„righetsgrad enligt Kattis, men stöter pĂ„ svĂ„righeter med mer komplexa programmeringsuppgifter. En kvalitativ uppföljningsundersökning genomfördes Ă€ven dĂ€r tvĂ„ lĂ€rare frĂ„n KTH intervjuades för att ge sina kommentarer om metodvalet och diskutera fusket inom högre utbildning nĂ€r det gĂ€ller programmeringskurser. Kattis-systemet anses ha anvĂ€ndbara funktioner för att förhindra fusk, sĂ„som dolda testfall, men har ocksĂ„ begrĂ€nsningar nĂ€r det gĂ€ller att upptĂ€cka AI-genererad kod. Rapporten avslutas med att diskutera implikationerna för olika intressenter, inklusive lĂ€rare, studenter och forskare