Providing computer-assisted, two-way feedback in formative assessment: an innovation supporting best educational practice

This thesis presents the design and development of an educational ICT innovation called the Quality Assessment System (QAS), intended to: increase the speed of providing useful, legible and consistent feedback, enhance student engagement in the analysis and improvement of their own work, and provide an easily-accessible, cumulative history of completed tasks and feedback. The QAS has been developed to a proof-of-concept stage as a Microsoft Word add-in, which can be used on digital or handwritten work, and has functions to administer resubmissions. The prototype system was evaluated at a tertiary institution in the field of English for Speakers of Other Languages. I used observations, interview methods, and a Wizard-of-Oz experiment to simulate full use of the software. The research found that: - the QAS could foster the rapid provision of consistent, clear feedback; - the facility to provide digital feedback on handwritten work safeguarded the desire of some students to continue writing their tasks by hand; - the handling of resubmitted tasks and the comparison of feedback on the first and second submissions (or any other pair of user-selected tasks) was considered very useful; - some students were emotional attached to handwritten feedback and believed that feedback mediated by computer showed a lack of teacher care for the students; - administrators believed the QAS would be useful for resolving student-teacher disputes, and as a tool to enhance the robustness of the quality self-assessment system the faculty adhered to. While I acknowledge the need for caution in interpreting the fieldwork results of small samples, this research places systemisation tools such as the QAS firmly on the agenda for closer investigation

Optimising dominant intraprostatic lesion outlining for prostate radiotherapy

The dominant intraprostatic lesion (DIL) is the most common source of local recurrence following radiotherapy for localised prostate cancer. Recent trial findings suggest dose escalation to the DIL results in a low chance of prostate cancer recurrence at five years and confers minimal excess toxicity compared to standard prostate radiotherapy dose and fractionation schedules. Defining the DIL to outline for radiotherapy planning requires mpMRI interpretation skills and the ability to cognitively transfer the mpMRI defined DIL onto the CT planning scan. Outlining variation amongst radiotherapy outliners across all tumour sites is a well-recognised limitation in the radiotherapy planning pathway. Strategies to minimise inter-observer outlining variation include implementation of tumour sitespecific outlining guidelines and educational outlining workshops. This thesis explores the incidence of outlining variation of the DIL amongst UK prostate oncologists using data from the pre-accrual benchmark case submissions of the PIVOTALboost trial; a phase III randomised control trial of prostate and pelvis versus prostate alone radiotherapy, with or without DIL dose-escalation. Having established a high rate of case resubmission, predominantly due to unacceptable DIL outlining variation, the thesis then explores the role of conformity indices as a semiautomated tool in the assessment of unacceptable versus acceptable outlining variation in the PIVOTALboost pre-accrual benchmark case submissions. Thereafter, this thesis evaluates the impact of an outlining workshop on DIL outlining performance of UK prostate oncologists. Finally, this thesis investigates which step within the DIL outlining process is the cause of outlining variation; i.e. mpMRI interpretation or cognitive transfer, to inform future work aimed at minimising DIL outlining variation

Language Learning Tasks and Automatic Analysis of Learner Language: Connecting FLTL and NLP design of ICALL materials supporting use in real-life instruction

This thesis studies the application of Natural Language Processing to Foreign Language Teaching and Learning, within the research area of Intelligent Computer- Assisted Language Learning (ICALL). In particular, we investigate the design, the implementation, and the use of ICALL materials to provide learners of foreign languages, particularly English, with automated feedback. We argue that the successful integration of ICALL materials demands a design process considering both pedagogical and computational requirements as equally important. Our investigation pursues two goals. The first one is to integrate into task design insights from Second Language Acquisition and Foreign Language Teaching and Learning with insights from computational linguistic modelling. The second goal is to facilitate the integration of ICALL materials in real-world instruction settings, as opposed to research or lab-oriented instruction settings, by empowering teachers with the methodology and the technology to autonomously author such materials. To achieve the first goal, we propose an ICALL material design process that combines basic principles of Task-Based Language Instruction and Task-Based Test Design with the specification requirements of Natural Language Processing. The relation between pedagogical and computational requirements is elucidated by exploring (i) the formal features of foreign language learning activities, (ii) the complexity and variability of learner language, and (iii) the feasibility of applying computational techniques for the automatic analysis and evaluation of learner responses. To achieve the second goal, we propose an automatic feedback generation strategy that enables teachers to customise the computational resources required to automatically correct ICALL activities without the need for programming skills. This proposal is instantiated and evaluated in real world-instruction settings involving teachers and learners in secondary education. Our work contributes methodologically and empirically to the ICALL field, with a novel approach to the design of materials that highlights the cross-disciplinary and iterative nature of the task. Our findings reveal the strength of characterising tasks both from the perspective of Foreign Language Teaching and Learning and from the perspective of Computational Linguistics as a means to clarify the nature of learning activities. Such a characterisation allows us to identify ICALL materials which are both pedagogically meaningful and computationally feasible. Our results show that teachers can characterise, author and employ ICALL mate- rials as part of their instruction programme, and that the underlying computational machinery can provide the required automatic processing with sufficient efficiency. The authoring tool and the accompanying methodology become a crucial instrument for ICALL research and practice: Teachers are able to design activities for their students to carry out without relying on an expert in Natural Language Processing. Last but not least, our results show that teachers are value the experience very positively as means to engage in technology integration, but also as a means to better apprehend the nature of their instruction task. Moreover, our results show that learners are motivated by the opportunity of using a technology that enhances their learning experience

En eksempel-studie av automatisk tilbakemelding for programmeringsfag i høyere utdanning.

Denne studien er gjennomført i forbindelse med eksamen i et førsteårs programmeringsemne ved et universitet. Vi har undersøkt hvordan vi ved bruk av et egenutviklet verktøy for sensurering som består av kvantitativ vekting og som samtidig genererer kvalitative tilbakemeldinger til studentene forbedrer kvaliteten på sensur og på eksamensoppgavene, og ikke minst støtter studentenes videre læring. Gjennom analyse av kvantitative data generert av systemet, samt kvalitative data fra involverte parter fremkommer tydelig forbedringer i validitet og reliabilitet i sensur, samt positive erfaringer fra sensorer og faglærere og ikke minst fra studentene som opplever at det å få tilbakemelding på eksamen bidrar til deres læring og utvikling. Vi vil i denne artikkelen beskrive prosessen med å utviklingen dette systemet for sensur og automatisk tilbakemelding, og presentere resultatene sett fra faglærere, sensorenes og studentenes perspektiv

Empirical Research for Public Policy: With Examples from Family Law

Perhaps more than in any other field, legal scholarship has aimed directly at influencing public policy. Hence, it is not surprising that empirical scholarship on law related issues often seems to have an agenda that extends beyond the common social science goals of adding to our knowledge base and understanding of human behavior to suggesting to policy makers and practitioners legal and administrative changes that will ameliorate problems they confront and, by the researcher’s lights, make this a better world in which to live

Supporting Development Decisions with Software Analytics

Software practitioners make technical and business decisions based on the understanding they have of their software systems. This understanding is grounded in their own experiences, but can be augmented by studying various kinds of development artifacts, including source code, bug reports, version control meta-data, test cases, usage logs, etc. Unfortunately, the information contained in these artifacts is typically not organized in the way that is immediately useful to developers’ everyday decision making needs. To handle the large volumes of data, many practitioners and researchers have turned to analytics — that is, the use of analysis, data, and systematic reasoning for making decisions. The thesis of this dissertation is that by employing software analytics to various development tasks and activities, we can provide software practitioners better insights into their processes, systems, products, and users, to help them make more informed data-driven decisions. While quantitative analytics can help project managers understand the big picture of their systems, plan for its future, and monitor trends, qualitative analytics can enable developers to perform their daily tasks and activities more quickly by helping them better manage high volumes of information. To support this thesis, we provide three different examples of employing software analytics. First, we show how analysis of real-world usage data can be used to assess user dynamic behaviour and adoption trends of a software system by revealing valuable information on how software systems are used in practice. Second, we have created a lifecycle model that synthesizes knowledge from software development artifacts, such as reported issues, source code, discussions, community contributions, etc. Lifecycle models capture the dynamic nature of how various development artifacts change over time in an annotated graphical form that can be easily understood and communicated. We demonstrate how lifecycle models can be generated and present industrial case studies where we apply these models to assess the code review process of three different projects. Third, we present a developer-centric approach to issue tracking that aims to reduce information overload and improve developers’ situational awareness. Our approach is motivated by a grounded theory study of developer interviews, which suggests that customized views of a project’s repositories that are tailored to developer-specific tasks can help developers better track their progress and understand the surrounding technical context of their working environments. We have created a model of the kinds of information elements that developers feel are essential in completing their daily tasks, and from this model we have developed a prototype tool organized around developer-specific customized dashboards. The results of these three studies show that software analytics can inform evidence-based decisions related to user adoption of a software project, code review processes, and improved developers’ awareness on their daily tasks and activities

Promoting Programming Learning. Engagement, Automatic Assessment with Immediate Feedback in Visualizations

The skill of programming is a key asset for every computer science student. Many studies have shown that this is a hard skill to learn and the outcomes of programming courses have often been substandard. Thus, a range of methods and tools have been developed to assist students’ learning processes. One of the biggest fields in computer science education is the use of visualizations as a learning aid and many visualization based tools have been developed to aid the learning process during last few decades. Studies conducted in this thesis focus on two different visualizationbased tools TRAKLA2 and ViLLE. This thesis includes results from multiple empirical studies about what kind of effects the introduction and usage of these tools have on students’ opinions and performance, and what kind of implications there are from a teacher’s point of view. The results from studies in this thesis show that students preferred to do web-based exercises, and felt that those exercises contributed to their learning. The usage of the tool motivated students to work harder during their course, which was shown in overall course performance and drop-out statistics. We have also shown that visualization-based tools can be used to enhance the learning process, and one of the key factors is the higher and active level of engagement (see. Engagement Taxonomy by Naps et al., 2002). The automatic grading accompanied with immediate feedback helps students to overcome obstacles during the learning process, and to grasp the key element in the learning task. These kinds of tools can help us to cope with the fact that many programming courses are overcrowded with limited teaching resources. These tools allows us to tackle this problem by utilizing automatic assessment in exercises that are most suitable to be done in the web (like tracing and simulation) since its supports students’ independent learning regardless of time and place. In summary, we can use our course’s resources more efficiently to increase the quality of the learning experience of the students and the teaching experience of the teacher, and even increase performance of the students. There are also methodological results from this thesis which contribute to developing insight into the conduct of empirical evaluations of new tools or techniques. When we evaluate a new tool, especially one accompanied with visualization, we need to give a proper introduction to it and to the graphical notation used by tool. The standard procedure should also include capturing the screen with audio to confirm that the participants of the experiment are doing what they are supposed to do. By taken such measures in the study of the learning impact of visualization support for learning, we can avoid drawing false conclusion from our experiments. As computer science educators, we face two important challenges. Firstly, we need to start to deliver the message in our own institution and all over the world about the new – scientifically proven – innovations in teaching like TRAKLA2 and ViLLE. Secondly, we have the relevant experience of conducting teaching related experiment, and thus we can support our colleagues to learn essential know-how of the research based improvement of their teaching. This change can transform academic teaching into publications and by utilizing this approach we can significantly increase the adoption of the new tools and techniques, and overall increase the knowledge of best-practices. In future, we need to combine our forces and tackle these universal and common problems together by creating multi-national and multiinstitutional research projects. We need to create a community and a platform in which we can share these best practices and at the same time conduct multi-national research projects easily.Siirretty Doriast

Utilizing educational technology in computer science and programming courses : theory and practice

There is one thing the Computer Science Education researchers seem to agree: programming is a difficult skill to learn. Educational technology can potentially solve a number of difficulties associated with programming and computer science education by automating assessment, providing immediate feedback and by gamifying the learning process. Still, there are two very important issues to solve regarding the use of technology: what tools to use, and how to apply them? In this thesis, I present a model for successfully adapting educational technology to computer science and programming courses. The model is based on several years of studies conducted while developing and utilizing an exercise-based educational tool in various courses. The focus of the model is in improving student performance, measured by two easily quantifiable factors: the pass rate of the course and the average grade obtained from the course. The final model consists of five features that need to be considered in order to adapt technology effectively into a computer science course: active learning and continuous assessment, heterogeneous exercise types, electronic examination, tutorial-based learning, and continuous feedback cycle. Additionally, I recommend that student mentoring is provided and cognitive load of adapting the tools considered when applying the model. The features are classified as core components, supportive components or evaluation components based on their role in the complete model. Based on the results, it seems that adapting the complete model can increase the pass rate statistically significantly and provide higher grades when compared with a “traditional” programming course. The results also indicate that although adapting the model partially can create some improvements to the performance, all features are required for the full effect to take place. Naturally, there are some limits in the model. First, I do not consider it as the only possible model for adapting educational technology into programming or computer science courses. Second, there are various other factors in addition to students’ performance for creating a satisfying learning experience that need to be considered when refactoring courses. Still, the model presented can provide significantly better results, and as such, it works as a base for future improvements in computer science education.Ohjelmoinnin oppimisen vaikeus on yksi harvoja asioita, joista lähes kaikki tietojenkäsittelyn opetuksen tutkijat ovat jokseenkin yksimielisiä. Opetusteknologian avulla on mahdollista ratkaista useita ohjelmoinnin oppimiseen liittyviä ongelmia esimerkiksi hyödyntämällä automaattista arviointia, välitöntä palautetta ja pelillisyyttä. Teknologiaan liittyy kuitenkin kaksi olennaista kysymystä: mitä työkaluja käyttää ja miten ottaa ne kursseilla tehokkaasti käyttöön? Tässä väitöskirjassa esitellään malli opetusteknologian tehokkaaseen hyödyntämiseen tietojenkäsittelyn ja ohjelmoinnin kursseilla. Malli perustuu tehtäväpohjaisen oppimisjärjestelmän runsaan vuosikymmenen pituiseen kehitys- ja tutkimusprosessiin. Mallin painopiste on opiskelijoiden suoriutumisen parantamisessa. Tätä arvioidaan kahdella kvantitatiivisella mittarilla: kurssin läpäisyprosentilla ja arvosanojen keskiarvolla. Malli koostuu viidestä tekijästä, jotka on otettava huomioon tuotaessa opetusteknologiaa ohjelmoinnin kursseille. Näitä ovat aktiivinen oppiminen ja jatkuva arviointi, heterogeeniset tehtävätyypit, sähköinen tentti, tutoriaalipohjainen oppiminen sekä jatkuva palautesykli. Lisäksi opiskelijamentoroinnin järjestäminen kursseilla ja järjestelmän käyttöönottoon liittyvän kognitiivisen kuorman arviointi tukevat mallin käyttöä. Malliin liittyvät tekijät on tässä työssä lajiteltu kolmeen kategoriaan: ydinkomponentteihin, tukikomponentteihin ja arviontiin liittyviin komponentteihin. Tulosten perusteella vaikuttaa siltä, että mallin käyttöönotto parantaa kurssien läpäisyprosenttia tilastollisesti merkittävästi ja nostaa arvosanojen keskiarvoa ”perinteiseen” kurssimalliin verrattuna. Vaikka mallin yksittäistenkin ominaisuuksien käyttöönotto voi sinällään parantaa kurssin tuloksia, väitöskirjaan kuuluvien tutkimusten perusteella näyttää siltä, että parhaat tulokset saavutetaan ottamalla malli käyttöön kokonaisuudessaan. On selvää, että malli ei ratkaise kaikkia opetusteknologian käyttöönottoon liittyviä kysymyksiä. Ensinnäkään esitetyn mallin ei ole tarkoituskaan olla ainoa mahdollinen tapa hyödyntää opetusteknologiaa ohjelmoinnin ja tietojenkäsittelyn kursseilla. Toiseksi tyydyttävään oppimiskokemukseen liittyy opiskelijoiden suoriutumisen lisäksi paljon muitakin tekijöitä, jotka tulee huomioida kurssien uudelleensuunnittelussa. Esitetty malli mahdollistaa kuitenkin merkittävästi parempien tulosten saavuttamisen kursseilla ja tarjoaa sellaisena perustan entistä parempaan opetukseen

Interventions using digital tools to improve students’ engagement and learning outcomes in higher business education

The papers of this thesis are not available in Munin. Paper 1: Bertheussen, B. A.: "Cultivating spreadsheet usage in a finance course through learning and assessment innovations". Available in International Journal of Innovation in Education 2015, 3(1). Paper 2: Bertheussen, B. A., Myrland, Ø.: "Relation between academic performance and students’ engagement in digital learning activities". Available in Journal of Education for Business 2016, 91(3), 1–7. Paper 3: Bertheussen, B. A.: "Er handelshøyskolene innelåst i historiske pedagogiske spor?". Available in Magma 2013, 16(5),40–48. Paper 4: Bertheussen, B. A. "Ruteark eller regneark. Kognitive utfordringer med å løse finansoppgaver på papier og PC". Available in Uniped 2012, 35(3):87–101. Paper 5: Bertheussen, B. A.: "Validating a Digital Assessment Practice". (Manuscript). Paper 6: Bertheussen, B. A. "Power to business professors. Automatic grading of problem-solving tasks". Available in Journal of Accounting Education 2014, 32(1):76–87. Paper 7: Bertheussen, B. A.: "Automatisk formativ feedback kan gi god motivasjon og læring". Available in Uniped 2014, 37(4):59–71. Paper 8: Bertheussen, B. A. "Revitalizing plenary finance lectures". Available in Beta 2013, 27(1):78–92. The purpose of the present study was to develop interventions using digital tools to improve student engagement and learning outcomes. The empirical context was an undergraduate finance course wherein digital learning and assessment interventions were important features of the course design. When designing the interventions, the development activities were underpinned by pedagogical principles based on cognitive and sociocultural learning perspectives. Special emphasis was placed on integrating spreadsheet usage into all learning and assessment activities and constructively aligning course targets, assessment tasks and learning activities with the overall goal to foster an active and engaging learning environment. In addition, rooted in a pragmatic research paradigm, the methodology utilised includes many similarities with interventionist action research, which has gained a foothold in qualitative management accounting research. This interventionist research project includes two main contributions. The first is its impact on practice by designing and developing interventions to solve complex problems in an authentic classroom setting. Consequently, six practical educational interventions are discussed in this dissertation. The second contribution is theory building, which advances our knowledge regarding the characteristics of the interventions and the process of designing and developing them. Consequently, a total of eight refereed scientific articles have been produced during this research and development project. As outlined in this study, the development of the digital formative feedback intervention, is in line with research stating that, in higher education, traditional paper-based feedback is being supplemented with and in some cases replaced by innovative use of ICT. Moreover, software algorithms can effectively provide detailed and helpful individual formative feedback to students regarding their learning processes and outcomes. This study strongly supports the claim that it is problematic to use technology to enhance learning without recognition through assessments. The digital summative assessment intervention reported is regarded as a precondition for establishing a spreadsheet user-culture in the subject, especially as it served as an ‘icebreaker’ for other learning interventions that were integrated into the course design. The intervention processes discussed have been through several iterations and their stepwise development and implementation have emerged through negotiating, compromising and resolving tension between the practitioner researcher, students and institution. The resulting compromises resolved tensions which sometimes resulted from limited physical resources. As the students valued the outcome from engaging in the digital learning and assessment interventions, they had a flexible attitude and deployed their private infrastructure (laptops) within the learning environment. Consequently, a vital part of the institution’s infrastructure was transformed from a fixed asset (number of PCs available in a data lab) to a flexible asset in the theatres. This compromise that was negotiated between the institution, the practitioner researcher and the students was essential for the digital educational interventions to work and progress. The overall theoretical research findings from this study are presented in the form of a tentative framework, which can help bridge the gap between the intervention practice and theory. A central conjecture in the framework is that tool usage that is integrated into interventions can be influential on learning activity and engagement and consequently on students’ learning outcomes. Moreover, the framework supports the notion of ICT as a mediating cultural tool that provides a new type of affordance that can extend the mind and promote an active and engaging learning environment. In particular, integrating a spreadsheet tool in learning of management accounting subjects can offer opportunities for learners to rapidly construct financial models, enable simulations using the completed models and stimulate subject reflections based on the functions of the models and their results. The practical outcome of this study has been emphasised through the development of artefacts that aim to support practitioners intending to integrate spreadsheet usage within their subject teaching and learning. By publishing and sharing the artefacts, the current research project is capable of informing future development and implementation decisions by guiding practitioners in similar pedagogical contexts

Automated assessment of programming assignments : visual feedback, assignment mobility, and assessment of students' testing skills

The main objective of this thesis is to improve the automated assessment of programming assignments from the perspective of assessment tool developers. We have developed visual feedback on functionality of students' programs and explored methods to control the level of detail in visual feedback. We have found that visual feedback does not require major changes to existing assessment platforms. Most modern platforms are web based, creating an opportunity to describe visualizations in JavaScript and HTML embedded into textual feedback. Our preliminary results on the effectiveness of automatic visual feedback indicate that students perform equally well with visual and textual feedback. However, visual feedback based on automatically extracted object graphs can take less time to prepare than textual feedback of good quality. We have also developed programming assignments that are easier to port from one server environment to another by performing assessment on the client-side. This not only makes it easier to use the same assignments in different server environments but also removes the need for sandboxing the execution of students' programs. The approach will likely become more important in the future together with interactive study materials becoming more popular. Client-side assessment is more suitable for self-studying material than for grading because assessment results sent by a client are often too easy to falsify. Testing is an important part of programming and automated assessment should also cover students' self-written tests. We have analyzed how students behave when they are rewarded for structural test coverage (e.g. line coverage) and found that this can lead students to write tests with good coverage but with poor ability to detect faulty programs. Mutation analysis, where a large number of (faulty) programs are automatically derived from the program under test, turns out to be an effective way to detect tests otherwise fooling our assessment systems. Applying mutation analysis directly for grading is problematic because some of the derived programs are equivalent with the original and some assignments or solution strategies generate more equivalent mutants than others