Introductory programming: a systematic literature review

As computing becomes a mainstream discipline embedded in the school curriculum and acts as an enabler for an increasing range of academic disciplines in higher education, the literature on introductory programming is growing. Although there have been several reviews that focus on specific aspects of introductory programming, there has been no broad overview of the literature exploring recent trends across the breadth of introductory programming. This paper is the report of an ITiCSE working group that conducted a systematic review in order to gain an overview of the introductory programming literature. Partitioning the literature into papers addressing the student, teaching, the curriculum, and assessment, we explore trends, highlight advances in knowledge over the past 15 years, and indicate possible directions for future research

Does the Performance on Principles of Economics Courses Affect the Overall Academic Success of Undergraduate Business Majors?

Using a sample of 1,339 graduates from an accredited business school and the maximum likelihood technique, this paper explores the relationship between overall academic success and performance on the Principles of Economics courses. The estimated model, which also includes some demographic variables, shows that the rank of professors teaching the course, age of students, and the number of credits earned do not influence business majors' overall academic success. However, the grades earned on the Principles of Economics courses, gender, ethnicity, the major in which the student is enrolled, the number of years the student takes to graduate, as well as whether or not the student is completing a minor significantly affect the overall academic success or the final GPA of business majors.

Predictive models as early warning systems for student academic performance in introductory programming

Computer programming is fundamental to Computer Science and IT curricula. At the novice level it covers programming concepts that are essential for subsequent advanced programming courses. However, introductory programming courses are among the most challenging courses for novices and high failure and attrition rates continue even as computer science education has seen improvements in pedagogy. Consequently, the quest to identify factors that affect student learning and academic performance in introductory computer programming courses has been a long-standing activity. Specifically, weak novice learners of programming need to be identified and assisted early in the semester in order to alleviate any potential risk of failing or withdrawing from their course. Hence, it is essential to identify at-risk programming students early, in order to plan (early) interventions. The goal of this thesis was to develop a validated, predictive model(s) with suitable predictors of student academic performance in introductory programming courses. The proposed model utilises the Naïve Bayes classification machine learning algorithm to analyse student performance data, based on the principle of parsimony. Furthermore, an additional objective was to propose this validated predictive model as an early warning system (EWS), to predict at-risk students early in the semester and, in turn, to potentially inform instructors (and students) for early interventions. We obtained data from two introductory programming courses in our study to develop and test the predictive models. The models were built with student presage and in progress-data for which instructors may easily collect or access despite the nature of pedagogy of educational settings. In addition, our work analysed the predictability of selected data sources and looked for the combination of predictors, which yields the highest prediction accuracy to predict student academic performance. The prediction accuracies of the models were computed by using confusion matrix data including overall model prediction accuracy, prediction accuracy sensitivity and specificity, balanced accuracy and the area under the ROC curve (AUC) score for generalisation. On average, the models developed with formative assessment tasks, which were partially assisted by the instructor in the classroom, returned higher at-risk prediction accuracies than the models developed with take-home assessment task only as predictors. The unknown data test results of this study showed that it is possible to predict 83% of students that need support as early as Week 3 in a 12-week introductory programming course. The ensemble method-based results suggest that it is possible to improve overall at-risk prediction performance with low false positives and to incorporate this in early warning systems to identify students that need support, in order to provide early intervention before they reach critical stages (at-risk of failing). The proposed model(s) of this study were developed on the basis of the principle of parsimony as well as previous research findings, which accounted for variations in academic settings, such as academic environment, and student demography. The predictive model could potentially provide early warning indicators to facilitate early warning intervention strategies for at-risk students in programming that allow for early interventions. The main contribution of this thesis is a model that may be applied to other programming and non-programming courses, which have both continuous formative and a final exam summative assessment, to predict final student performance early in the semester.Ohjelmointi on informaatioteknologian ja tietojenkäsittelytieteen opinto-ohjelmien olennainen osa. Aloittelijatasolla opetus kattaa jatkokurssien kannalta keskeisiä ohjelmoinnin käsitteitä. Tästä huolimatta ohjelmoinnin peruskurssit ovat eräitä haasteellisimmista kursseista aloittelijoille. Korkea keskeyttämisprosentti ja opiskelijoiden asteittainen pois jättäytyminen ovat vieläkin tunnusomaisia piirteitä näille kursseille, vaikka ohjelmoinnin opetuksen pedagogiikka onkin kehittynyt. Näin ollen vaikuttavia syitä opiskelijoiden heikkoon suoriutumiseen on etsitty jo pitkään. Erityisesti heikot, aloittelevat ohjelmoijat tulisi tunnistaa mahdollisimman pian, jotta heille voitaisiin tarjota tukea ja pienentää opiskelijan riskiä epäonnistua kurssin läpäimisessä ja riskiä jättää kurssi kesken. Heikkojen opiskelijoiden tunnistaminen on tärkeää, jotta voidaan suunnitella aikainen väliintulo. Tämän väitöskirjatyön tarkoituksena oli kehittää todennettu, ennustava malli tai malleja sopivilla ennnustusfunktioilla koskien opiskelijan akateemista suoriutumista ohjelmoinnin peruskursseilla. Kehitetty malli käyttää koneoppivaa naiivia bayesilaista luokittelualgoritmia analysoimaan opiskelijoiden suoriutumisesta kertynyttä aineistoa. Lähestymistapa perustuu yksinkertaisimpien mahdollisten selittävien mallien periaatteeseen. Lisäksi, tavoitteena oli ehdottaa tätä validoitua ennustavaa mallia varhaiseksi varoitusjärjestelmäksi, jolla ennustetaan putoamisvaarassa olevat opiskelijat opintojakson alkuvaiheessa sekä informoidaan ohjaajia (ja opiskelijaa) aikaisen väliintulon tarpeellisuudesta. Keräsimme aineistoa kahdelta ohjelmoinnin peruskurssilta, jonka pohjalta ennustavaa mallia kehitettiin ja testattiin. Mallit on rakennettu opiskelijoiden ennakkotietojen ja kurssin kestäessä kerättyjen suoriutumistietojen perusteella, joita ohjaajat voivat helposti kerätä tai joihin he voivat päästä käsiksi oppilaitoksesta tai muusta ympäristöstä huolimatta. Lisäksi väitöskirjatyö analysoi valittujen datalähteiden ennustettavuutta ja sitä, mitkä mallien muuttujista ja niiden kombinaatioista tuottivat kannaltamme korkeimman ennustetarkkuuden opiskelijoiden akateemisessa suoriutumisessa. Mallien ennustusten tarkkuuksia laskettiin käyttämällä sekaannusmatriisia, josta saadaan laskettua ennusteen tarkkuus, ennusteen spesifisyys, sensitiivisyys, tasapainotettu tarkkuus sekä luokitteluvastekäyriä (receiver operating characteristics (ROC)) ja näiden luokitteluvastepinta-ala (area under curve (AUC)) Mallit, jotka kehitettiin formatiivisilla tehtävillä, ja joissa ohjaaja saattoi osittain auttaa luokkahuonetilanteessa, antoivat keskimäärin tarkemman ennustuksen putoamisvaarassa olevista opiskelijoista kuin mallit, joissa käytettiin kotiin vietäviä tehtäviä ainoina ennusteina. Tuntemattomalla testiaineistolla tehdyt mallinnukset osoittavat, että voimme tunnistaa jo 3. viikon kohdalla 83% niistä opiskelijoista, jotka tarvitsevat lisätukea 12 viikkoa kestävällä ohjelmoinnin kurssilla. Tulosten perusteella vaikuttaisi, että yhdistämällä metodeja voidaan saavuttaa parempi yleinen ennustettavuus putoamisvaarassa olevien opiskelijoiden suhteen pienemmällä määrällä väärin luokiteltuja epätositapauksia. Tulokset viittaavat myös siihen, että on mahdollista sisällyttää yhdistelmämalli varoitusjärjestelmiin, jotta voidaan tunnistaa avuntarpeessa olevia opiskelijoita ja tarjota täten varhaisessa vaiheessa tukea ennen kuin on liian myöhäistä. Tässä tutkimuksessa esitellyt mallit on kehitetty nojautuen yksinkertaisimman selittävän mallin periaatteeseen ja myös aiempiin tutkimustuloksiin, joissa huomioidaan erilaiset akateemiset ympäristöt ja opiskelijoiden tausta. Ennustava malli voi tarjota indikaattoreita, jotka voivat mahdollisesti toimia pohjana väliintulostrategioihin kurssilta putoamisvaarassa olevien opiskelijoiden tukemiseksi. Tämän tutkimuksen keskeisin anti on malli, jolla opiskelijoiden suoriutumista voidaan arvioida muilla ohjelmointia ja muita aihepiirejä käsittelevillä kursseilla, jotka sisältävät sekä jatkuvaa arviointia että loppukokeen. Malli ennustaisi näillä kursseilla lopullisen opiskelijan suoritustason opetusjakson alkuvaiheessa

School-leavers' Transition to Tertiary Study: a Literature Review.

The theoretical and empirical literature relating to factors and problems in the transition of students from secondary to tertiary level education is reviewed here. Studies on persistence and attrition, and on the analysis and prediction of academic performance of students, generally and in particular discipline areas, are included.Transition to university; student performance.

Do Constructed-Response and Multiple-Choice Questions Measure the Same Thing?

Our study empirically investigates the relationship between constructed-response (CR) and multiple-choice (MC) questions using a unique data set compiled from several years of university introductory economics classes. We conclude that CR and MC questions do not measure the same thing. Our main contribution is that we show that CR questions contain independent information that is related to student learning. Specifically, we find that the component of CR scores that cannot be explained by MC responses is positively and significantly related to (i) performance on a subsequent exam in the same economics course, and (ii) academic performance in other courses. Further, we present evidence that CR questions provide information that could not be obtained by expanding the set of MC questions. A final contribution of our study is that we demonstrate that empirical approaches that rely on factor analyses or Walstad-Becker (1994)-type regressions are unreliable in the following sense: It is possible for these empirical procedures to lead to the conclusion that CR and MC questions measure the same thing, even when the underlying data contain strong, contrary evidence. JEL Classifications: A22Principles of Economics Assessment; Multiple Choice; Constructed Response; Free Response; Essay

Scholarly insight Autumn 2017:a Data wrangler perspective

As the OU is going through several fundamental changes, it is important that strategic decisions made by Faculties and senior management are informed by evidence-based research and insights. One way how Data Wranglers provide insights of longitudinal development and performance of OU modules is the Key Metric Report 2017. A particular new element is that data can now also be unpacked and visualised on a Nation-level. As evidenced by the Nation-level reporting, there are substantial variations of success across the four Nations, and we hope that our interactive dashboards allow OU staff to unpack the underlying data. The second way Data Wranglers provide insight to Faculties and Units is through the Scholarly insight report series. Building on the previous two reports whereby we reported on substantial variation and inconsistencies in learning designs and assessment practices within qualifications across the OU, in this Scholarly insight Autumn 2017 report we address four big pedagogical questions that were framed and co-constructed together with the Faculties and LTI units. Many Faculties and colleagues have reacted positively on our Scholarly insight Spring 2017 report, whereby for the first time we were able to show empirically that students experienced substantial variations in success within 12 large OU qualifications. As evidenced in our previous report, 55% of variation in students’ success over time was explained by OU institutional factors (i.e., how students were assessed within their respective module; how students were able to effectively transition from one learning design of one module to the next one), rather than students’ characteristics, engagement and behaviour. We have received several queries and questions from Faculties and Units about how to better understand these students’ journeys, and how qualifications and module designs could be better aligned within their respective qualification(s). As these are complex conceptual and Big Pedagogy questions, in Chapter 1 we continued these complex analyses by looking at the transitional processes of the first two modules that OU students take, and how well aligned these modules and qualification paths are. In Chapter 2, we explored the more fine-grained, qualitative, and lived experiences of 19 students across a range of qualifications to understand how OU grading practices and (in)consistencies of assessment and feedback influenced their affect, behaviour, and cognition. In addition to building on previous topics, we introduced two new Scholarly insights in Chapter 3 and Chapter 4. As the OU is increasingly using learning analytics to support our staff and students, in Chapter 3 we analysed the impact of giving Predictive Learning Analytics to over 500 Associate Lecturers across 31 modules on student retention. Finally, in Chapter 4 we explored the impact of first presentations of new modules on pass rates and satisfaction, whereby we were able to bust another myth that may have profound implications for Student First Transformation. Working organically in various Faculty sub-group meetings and LTI Units and in a google doc with various key stakeholders in the Faculties , we hope that our Scholarly insights can help to inform our staff, but also spark some ideas how to further improve our module designs and qualification pathways. Of course we are keen to hear what other topics require Scholarly insight

More Evidence on the Use of Constructed-Response Questions in Principles of Economics Classes

This study provides evidence that constructed response (CR) questions contribute information about student knowledge and understanding that is not contained in multiple choice questions (MC). We use an extensive data set of individual assessment results from Introductory Macro- and Microeconomics classes at a large, public university. We find that (i) CR scores contain information not contained in MC questions, (ii) this information is correlated with a measure of student knowledge and understanding of course material, and (iii) CR questions are better able to “explain” academic achievement in other courses than additional MC questions. There is some evidence to suggest that this greater explanatory power has to do with the ability of CR questions to measure higher-level learning as characterized by Bloom’s taxonomy (Bloom, 1956). Both (i) the generalisability of our results to other principles of economics classes, and (ii) the practical significance (in terms of students’ grades) of our findings, remain to be determined.Principles of Economics Assessment; Multiple Choice; Constructed Response; Free Response; Essay

From Gatekeeping to Engagement: A Multicontextual, Mixed Method Study of Student Academic Engagement in Introductory STEM Courses.

The lack of academic engagement in introductory science courses is considered by some to be a primary reason why students switch out of science majors. This study employed a sequential, explanatory mixed methods approach to provide a richer understanding of the relationship between student engagement and introductory science instruction. Quantitative survey data were drawn from 2,873 students within 73 introductory science, technology, engineering, and mathematics (STEM) courses across 15 colleges and universities, and qualitative data were collected from 41 student focus groups at eight of these institutions. The findings indicate that students tended to be more engaged in courses where the instructor consistently signaled an openness to student questions and recognizes her/his role in helping students succeed. Likewise, students who reported feeling comfortable asking questions in class, seeking out tutoring, attending supplemental instruction sessions, and collaborating with other students in the course were also more likely to be engaged. Instructional implications for improving students' levels of academic engagement are discussed