Information Visualization of Metacognitive Skills During the Software Development Process Based on an Adapted Engineering Design Metacognitive Questionnaire

In software development, either alone or in a team, there are many aspects that determine the success in developing the software, including each developer\u27s skills. Studies show that the application of metacognition can increase the effectiveness and efficiency of software development. To measure a metacognition skill, there need to be a metacognition measurement tools. One example of this measurement method is adapted engineering design metacognitive questionnaire. However, the respondents feel that existing tools still have not given them any benefits. This research is conducted to develop an information visualization tools for the metacognition measurement from an adapted engineering design metacognitive questionnaire. The research was performed using qualitative method adapted from the user-centered design approach, which is user requirement analysis, design alternatives, prototyping, and evaluation. The finding suggests that with information visualization, the students as the respondents feel the benefits of filling the EDMQ questionnaire. However, from the design standpoint, there are still numerous things that can be improved to make the visualization more informative

An Exploration of High School Students' Self-Confidence while Analysing Iterative Code

A number of studies on novice programming report that loops and conditionals can be potential sources of errors and misconceptions. We then felt the need to engage in a more systematic and in-depth investigation about the teaching and learning of iteration in some representative high schools of our regional area. As a medium-term outcome of this endeavour we expect to get fine-grained insights about the nature of students' difficulties, on the one hand, as well as to identify possible pedagogical approaches to be adopted by teachers, on the other. As a step of this project, we designed and administered a survey composed of a set of small tasks, addressing students’ understanding of iteration in terms of code reading abilities. After summarising the motivations underlying the choice of the tasklets and the overall structure of the instrument, in this paper we will focus on a particular aspect which has not yet received extensive attention in the computer science education literature. Specifically, we will consider students' perception of self-confidence, in connection with their actual performance in each task, the specific program features, the cognitive demands (procedural vs. higher-level thinking skills), and the use of code vs. flow-charts. A noteworthy result of this analysis is that students’ perception of self-confidence is poorly correlated to actual performance in the task at hand. The main implications of our study are twofold, pertaining our understanding of less conspicuous facets of the learning of iteration as well as possible pedagogical strategies to strengthen metacognitive skills

An Exploratory Investigation on High-School Students' Understanding of Threads

Students' difficulties to learn concurrent programming are well known amongst Computer Science instructors. While in the International Computing Education community it is still up to debate the extent to which such topic should be included in pre-university curricula, based on our country's Ministerial guidelines for technical high schools with a specialization in Computer Science, students are expected to acquire key concurrent programming skills. With the aim of getting insights about the nature of students' difficulties, as well as to identify possible pedagogical approaches to be adopted by teachers, we have undertaken an investigation on students' perception, proficiency and self-confidence when dealing with concurrency and synchronization tasks. We then present the results of a preliminary study carried out by submitting a survey in a couple of representative high schools of our area. The survey includes subjective perception questions as well as small program comprehension tasks addressing students' understanding of thread synchronization. Moreover, we also analyze students' self-confidence in connection with their actual performance in such tasks. A total of 68 high school students were engaged in the survey. Our findings indicate that students' perception of self-confidence tends to weakly correlate to their actual performance, although more in general they express a low self-confidence level in relation to the topic. In particular, the results clearly show that the concept of thread synchronization is especially difficult to master for a large majority of them

Beyond Automated Assessment: Building Metacognitive Awareness in Novice Programmers in CS1

The primary task of learning to program in introductory computer science courses (CS1) cognitively overloads novices and must be better supported. Several recent studies have attempted to address this problem by understanding the role of metacognitive awareness in novices learning programming. These studies have focused on teaching metacognitive awareness to students by helping them understand the six stages of learning so students can know where they are in the problem-solving process, but these approaches are not scalable. One way to address scalability is to implement features in an automated assessment tool (AAT) that build metacognitive awareness in novice programmers. Currently, AATs that provide feedback messages to students can be said to implement the fifth and sixth learning stages integral to metacognitive awareness: implement solution (compilation) and evaluate implemented solution (test cases). The computer science education (CSed) community is actively engaged in research on the efficacy of compile error messages (CEMs) and how best to enhance them to maximize student learning and it is currently heavily disputed whether or not enhanced compile error messages (ECEMs) in AATs actually improve student learning. The discussion on the effectiveness of ECEMs in AATs remains focused on only one learning stage critical to metacognitive awareness in novices: implement solution. This research carries out an ethnomethodologically-informed study of CS1 students via think-aloud studies and interviews in order to propose a framework for designing an AAT that builds metacognitive awareness by supporting novices through all six stages of learning. The results of this study provide two important contributions. The first is the confirmation that ECEMs that are designed from a human-factors approach are more helpful for students than standard compiler error messages. The second important contribution is that the results from the observations and post-assessment interviews revealed the difficulties novice programmers often face to developing metacognitive awareness when using an AAT. Understanding these barriers revealed concrete ways to help novice programmers through all six stages of the problem-solving process. This was presented above as a framework of features, which when implemented properly, provides a scalable way to implicitly produce metacognitive awareness in novice programmers

Attitudes, behaviors, and learning outcomes from using classtranscribe, a UDL-featured video-based online learning platform with learnersourced text-searchable captions

This thesis consisted of a series of three studies on students' attitudes, behaviors, and learning outcomes from using ClassTranscribe, a Universal Design for Learning (UDL) featured video-based online learning platform. ClassTranscribe provided accurate accessible transcriptions and captioning plus a custom text-searchable interface to rapidly find relevant video moments from the entire course. Users could edit the machine-generated captions in a crowdsourcing way. The system logged student viewing, searching, and editing behaviors as fine-grained web browser interaction events including full-screen-switching, loss-of-focus, caption searching and editing events, and continued-video-watching events with the latter at 15-second granularity. In Study I, lecture material of a sophomore large-enrollment (N=271) system programming 15-week class in Spring 2019 was delivered solely online using a new video-based web platform - ClassTranscribe. Student learning behaviors and findings from four research questions were presented using individual-level performance and interaction data. Firstly, we reported on learning outcomes from alternative learning paths that arose from the course's application of Universal Design for Learning principles. Secondly, final exam performance was equal or better to prior semesters that utilized traditional in-person live lectures. Thirdly, learning outcomes of low and high performing students were analyzed independently by grouping students into four quartiles based on their non-final-exam course performance of programming assignments and quizzes. We introduced and justified an empirically-defined qualification threshold for sufficient video minutes viewed for each group. In all quartiles, students who watched an above-threshold of video minutes improved their in-group final exam performance (ranging from +6% to +14%) with the largest gain for the lowest-performing quartile. The improvement was similar in magnitude for all groups when expressed as a fraction of unrewarded final exam points. Finally, we found that using ClassTranscribe caption-based video search significantly predicted improvement in final exam scores. Overall, the study presented and evaluated how learner use of online video using ClassTranscribe predicted course performance and positive learning outcomes. In Study II, we further explored learner's searching behavior, which was shown to be correlated with improved final exam scores in the first study. From Fall 2019 to Summer 2020, engineering students used ClassTranscribe in engineering courses to view course videos and search for video content. The tool collected detailed timestamped student behavioral data from 1,894 students across 25 engineering courses that included what individual students searched for and when. As the first study showed that using ClassTranscribe caption search significantly predicted improvement in final exam scores in a computer science course, in this study, we presented how students used the search functionality based on a more detailed analysis of the log data. The search functionality of ClassTranscribe used the timestamped caption data to find specific video moments both within the current video or across the entire course. The number of search activities per person ranged from zero to 186 events. An in-depth analysis of the students (N=167) who performed 1,022 searches was conducted to gain insight into student search needs and behaviors. Based on the total number of searches performed, students were grouped into “Infrequent Searcher” (< 18 searches) and “Frequent Searcher” (18 to 110 searches) using clustering algorithms. The search queries used by each group were found to follow the Zipf’s Law and were categorized into STEM-related terms, course logistics and others. Our study reported on students’ search context, behaviors, strategies, and optimizations. Using Universal Design for Learning as a foundation, we discussed the implications for educators, designers, and developers who are interested in providing new learning pathways to support and enhance video-based learning environments. In Study III, we investigated students' attitudes towards learnersourced captioning for lecture videos. We deployed ClassTranscribe in a large (N=387) text retrieval and mining course where 58 learners participated in editing captions of 89 lecture videos, and each lecture video was edited by two editors sequentially. In the following semester, 18 editors participated in follow-up interviews to discuss their experience of using and editing captions in the class. Our study showed how students use captions to learn, and shed light on students' attitudes, motivations, and strategies in collaborating with other learners to fix captions in a learnersourced way

The Design and Evaluation of an Educational Software Development Process for First Year Computing Undergraduates

First year, undergraduate computing students experience a series of well-known challenges when learning how to design and develop software solutions. These challenges, which include a failure to engage effectively with planning solutions prior to implementation ultimately impact upon the students’ competency and their retention beyond the first year of their studies. In the software industry, software development processes systematically guide the development of software solutions through iterations of analysis, design, implementation and testing. Industry-standard processes are, however, unsuitable for novice programmers as they require prior programming knowledge. This study investigates how a researcher-designed educational software development process could be created for novice undergraduate learners, and the impact of this process on their competence in learning how to develop software solutions. Based on an Action Research methodology that ran over three cycles, this research demonstrates how an educational software development methodology (termed FRESH) and its operationalised process (termed CADET which is a concrete implementation of the FRESH methodology), was designed and implemented as an educational tool for enhancing student engagement and competency in software development. Through CADET, students were reframed as software developers who understand the value in planning and developing software solutions, and not as programmers who prematurely try to implement solutions. While there remain opportunities to further enhance the technical sophistication of the process as it is implemented in practice, CADET enabled the software development steps of analysis and design to be explicit elements of developing software solutions, rather than their more typically implicit inclusion in introductory CS courses. The research contributes to the field of computing education by exploring the possibilities of – and by concretely generating – an appropriate scaffolded methodology and process; by illustrating the use of computational thinking and threshold concepts in software development; and by providing a novel evaluation framework (termed AKM-SOLO) to aid in the continuous improvement of educational processes and courses by measuring student learning experiences and competencies