Using reflective writing and textual explanations to evaluate students' conceptual knowledge

BACKGROUND OR CONTEXT - Writing is one method used to prompt students to reflect on their own thought processes. Eliciting students’ explanations in the form of text, or writing, also provides lecturers with information about students’ thinking (Goncher, Boles, Jayalath, 2014; Boles, Goncher, Jayalath, 2015). Often in engineering courses, students adopt algorithmic problem-solving approaches without demonstrating conceptual reasoning. Adding a written, or explanatory component, to problems or questions is one approach that can elicit conceptual reasoning. PURPOSE OR GOAL - The purpose of this study was to identify and compare affordances of using students’ written explanations based the type of problem and response. This comparative study sought to answer two research questions, 1) How were the students’ textual answers different for the type of problem and requested explanations? and 2) What does the type and organization of the text of students’ explanations reveal about their conceptual knowledge? APPROACH - We analyzed students’ explanations for procedurally based problems in the statics discipline and conceptually based problems in the signal processing discipline. The first method used “process problems” that required students to explain, using only words, the process that they used to solve a statics homework problem. The second method utilized the Signals and Systems Concept Inventory items, and required students to provide a written explanation for their multiple-choice selection to each item. We categorized responses by the type of problem and structure of the written explanations to evaluate conceptual knowledge. DISCUSSION - We found that the structure of the text and type of problem provided different insights into students’ reasoning. The results showed that students approach learning in statics with varying emphasis placed on procedural and conceptual knowledge, and some students had difficulties explaining underlying concepts in signal processing and reverted to procedural explanations. Regardless of the type of problem, students that are able to get feedback on their thought processes can use the feedback to formatively evaluate their own understanding. RECOMMENDATIONS/IMPLICATIONS/CONCLUSION - Educators who incorporate or require students to reflect on their thinking through textual explanations can promote the revision of incorrect and/or inconsistent knowledge, leading to improved conceptual knowledge development. Assignments or activities that include more incidental writing will engage students in more freethinking and reflection (Essig et al., 2014;Hawkins, Coney, & Bystrom, 1996), and can lead to a richer understanding of technical concepts

Task-related models for teaching and assessing iteration learning in high school

A number of studies report about students’ difficulties with basic flow-control constructs, and specifically with iteration. Although such issues are less explored in the context of pre-tertiary education, this seems to be especially the case for high-school programming learning, where the difficulties concern both the “mechanical” features of the notional machine as well as the logical aspects connected with the constructs, ranging from the implications of loop conditions to a more abstract grasp of the underlying algorithms. For these reasons, the aim of this work is to: i) identifying methodological tools to enhance a comprehensive understanding of the iteration constructs, ii) suggest strategies to teach iterations. We interviewed 20 experienced upper secondary teachers of introductory programming in different kinds of schools. The interviews were mainly aimed at ascertaining teachers’ beliefs about major sources of issues for basic programming concepts and their approach to the teaching and learning of iteration constructs. Once teachers’ perception of students’ difficulties have been identified, we have submitted, to a sample of 164 students, a survey which included both questions on their subjective perception of difficulty and simple tasks probing their understanding of iteration. Data collected from teachers and students confirm that iteration is a central programming concept and indicate that the treatment of conditions and nested constructs are major sources of students’ difficulties with iteration. The interviews allowed us to identify a list of problems that are typically presented by teachers to explain the iterations. Hence, a catalogue of significant program examples has been built to support students’ learning, tasks with characteristics different from those typically presented in class. Based on the outcome of previous steps, a survey to collect related information and good practices from a larger sample of teachers has been designed. Data collected have been analysed distinguishing an orientation towards more conceptual objectives, and one towards more practical objectives. Furthermore, regarding evaluation, a orientation focused on process-based assessment and another on product-based assessment. Finally, based on the outcome of previous students’ survey and drawing from the proposed examples catalogue, we have designed and submitted a new students’ survey, composed of a set of small tasks, or tasklets, to investigate in more depth on high-school students’ understanding of iteration in terms of code reading abilities. The chosen tasklets covered the different topics: technical program feature, correlation between tracing effort and abstraction, the role of flow-charts, students’ perception of self-confidence concerning high-level thinking skills.A number of studies report about students’ difficulties with basic flow-control constructs, and specifically with iteration. Although such issues are less explored in the context of pre-tertiary education, this seems to be especially the case for high-school programming learning, where the difficulties concern both the “mechanical” features of the notional machine as well as the logical aspects connected with the constructs, ranging from the implications of loop conditions to a more abstract grasp of the underlying algorithms. For these reasons, the aim of this work is to: i) identifying methodological tools to enhance a comprehensive understanding of the iteration constructs, ii) suggest strategies to teach iterations. We interviewed 20 experienced upper secondary teachers of introductory programming in different kinds of schools. The interviews were mainly aimed at ascertaining teachers’ beliefs about major sources of issues for basic programming concepts and their approach to the teaching and learning of iteration constructs. Once teachers’ perception of students’ difficulties have been identified, we have submitted, to a sample of 164 students, a survey which included both questions on their subjective perception of difficulty and simple tasks probing their understanding of iteration. Data collected from teachers and students confirm that iteration is a central programming concept and indicate that the treatment of conditions and nested constructs are major sources of students’ difficulties with iteration. The interviews allowed us to identify a list of problems that are typically presented by teachers to explain the iterations. Hence, a catalogue of significant program examples has been built to support students’ learning, tasks with characteristics different from those typically presented in class. Based on the outcome of previous steps, a survey to collect related information and good practices from a larger sample of teachers has been designed. Data collected have been analysed distinguishing an orientation towards more conceptual objectives, and one towards more practical objectives. Furthermore, regarding evaluation, a orientation focused on process-based assessment and another on product-based assessment. Finally, based on the outcome of previous students’ survey and drawing from the proposed examples catalogue, we have designed and submitted a new students’ survey, composed of a set of small tasks, or tasklets, to investigate in more depth on high-school students’ understanding of iteration in terms of code reading abilities. The chosen tasklets covered the different topics: technical program feature, correlation between tracing effort and abstraction, the role of flow-charts, students’ perception of self-confidence concerning high-level thinking skills

An Assessment of Conceptual Understanding of Metabolic Pathways

Researchers have previously studied misconceptions of biochemistry topics such as photosynthesis, protein structure, and ATP-production. However, no studies have reported on students’ misconceptions regarding major metabolic pathways. Since learning metabolism builds on students’ prior knowledge, new material being learned will be affected by the presence of any misconceptions. Some of these misconceptions will be robust and thus hard to be replaced by the correct concepts. Thus, if students are to learn new material, these misconceptions must be diminished. This dissertation focused on the origins of these misconceptions, investigated what misconceptions students had on metabolism, and why students developed misconceptions instead of proper scientific conceptions. The ultimate goal of this study was to help students improve their conceptual understanding of general chemistry concepts that impact metabolic pathways by developing a video that targeted most of their misconceptions. This video depicted some metabolic reactions at the molecular level. Students’ misconceptions were first identified; based on them, an instructional intervention was designed to help students develop better, well-constructed conceptual schema (posttest). Evidence presented in the sample suggested the use of multimedia in helping students understand biochemistry was effective. An exploratory mixed methods design was used as a first stage to pilot any misconceptions among biochemistry students. A case study was conducted to investigate if graduate chemistry major students had any misconceptions on metabolism (n = 6). Based on the misconceptions found, the video was created to help undergraduate students develop a proper scientific understanding of the main concept targeted by this dissertation, which was chemical potential energy. The first phase of this research included quantitative validation of the instrument used (n = 45) and the second phase involved a phenomenological study where 11 graduate non-chemistry major students volunteered to participate. Many misconceptions were revealed by this study and most of them seemed to be prevalent and quite persistent

Understanding and Addressing Misconceptions in Introductory Programming: A Data-Driven Approach

With the expansion of computer science (CS) education, CS teachers in K-12 schools should be cognizant of student misconceptions and be prepared to help students establish accurate understanding of computer science and programming. This exploratory design-based research (DBR) study implemented a data-driven approach to identify secondary school students’ misconceptions using both their compilation and test errors and provide targeted feedback to promote students’ conceptual change in introductory programming. Research subjects were two groups of high school students enrolled in two sections of a Java-based programming course in a 2017 summer residential program for gifted and talented students. This study consisted of two stages. In the first stage, students of group 1 took the introductory programming class and used an automated learning system, Mulberry, which collected data on student problem-solving attempts. Data analysis was conducted to identify common programming errors students demonstrated in their programs and relevant misconceptions. In the second stage, targeted feedback to address these misconceptions was designed using principles from conceptual change and feedback theories and added to Mulberry. When students of group 2 took the same introductory programming class and solved programming problems in Mulberry, they received the targeted feedback to address their misconceptions. Data analysis was conducted to assess how the feedback affected the evolution of students’ (mis)conceptions. Using students’ erroneous solutions, 55 distinct compilation errors were identified, and 15 of them were categorized as common ones. The 15 common compilation errors accounted for 92% of all compilation errors. Based on the 15 common compilation errors, three underlying student misconceptions were identified, including deficient knowledge of fundamental Java program structure, misunderstandings of Java expressions, and confusion about Java variables. In addition, 10 common test errors were identified based on nine difficult problems. The results showed that 54% of all test errors were related to the difficult problems, and the 10 common test errors accounted for 39% of all test errors of the difficult problems. Four common student misconceptions were identified based on the 10 common test errors, including misunderstandings of Java input, misunderstandings of Java output, confusion about Java operators, and forgetting to consider special cases. Both quantitative and qualitative data analysis were conducted to see whether and how the targeted feedback affected students’ solutions. Quantitative analysis indicated that targeted feedback messages enhanced students’ rates of improving erroneous solutions. Group 2 students showed significantly higher improvement rates in all erroneous solutions and solutions with common errors compared to group 1 students. Within group 2, solutions with targeted feedback messages resulted in significantly higher improvement rates compared to solutions without targeted feedback messages. Results suggest that with targeted feedback messages students were more likely to correct errors in their code. Qualitative analysis of students’ solutions of four selected cases determined that students of group 2, when improving their code, made fewer intermediate incorrect solutions than students in group 1. The targeted feedback messages appear to have helped to promote conceptual change. The results of this study suggest that a data-driven approach to understanding and addressing student misconceptions, which is using student data in automated assessment systems, has the potential to improve students’ learning of programming and may help teachers build better understanding of their students’ common misconceptions and develop their pedagogical content knowledge (PCK). The use of automated assessment systems with misconception identification components may be helpful in pre-college introductory programming courses and so is encouraged as K-12 CS education expands. Researchers and developers of automated assessment systems should develop components that support identifying common student misconceptions using both compilation and non-compilation errors. Future research should continue to investigate the use of targeted feedback in automated assessment systems to address students’ misconceptions and promote conceptual change in computer science education

openHTML: Assessing Barriers and Designing Tools for Learning Web Development

In this dissertation, I argue that society increasingly recognizes the value of widespread computational literacy and that one of the most common ways that people are exposed to creative computing today is through web development. Prior research has investigated how beginners learn a wide range of programming languages in a variety of domains, from computer science majors taking introductory programming courses to end-user developers maintaining spreadsheets. Yet, surprisingly little is known about the experiences people have learning web development. What barriers do beginners face when authoring their first web pages? What mistakes do they commonly make when writing HTML and CSS? What are the computational skills and concepts with which they engage? How can tools and practices be designed to support these activities? Through a series of studies, interleaved with the iterative design of an experimental web editor for novices called openHTML, this dissertation aims to fill this gap in the literature and address these questions. In drawing connections between my findings and the existing computing education literature, my goal is to attain a deeper understanding of the skills and concepts at play when beginners learn web development, and to broaden notions about how people can develop computational literacy. This dissertation makes the following contributions: * An account of the barriers students face in an introductory web development course, contextualizing difficulties with learning to read and write code within the broad activity of web development. * The implementation of a web editor called openHTML, which has been designed to support learners by mitigating non-coding aspects of web development so that they can attend to learning HTML and CSS. * A detailed taxonomy of errors people make when writing HTML and CSS to construct simple web pages, derived from an intention-based analysis. * A fine-grained analysis of HTML and CSS syntax errors students make in the initial weeks of a web development course, how they resolve them, and the role validation plays in these outcomes. * Evidence for basic web development as a rich activity involving numerous skills and concepts that can support foundational computational literacy.Ph.D., Information Studies -- Drexel University, 201

The Statistics Concept Inventory: Development and analysis of a cognitive assessment instrument in statistics.

The Statistics Concept Inventory (SCI) is a multiple choice test designed to assess students' conceptual understanding of topics typically encountered in an introductory statistics course. This dissertation documents the development of the SCI from Fall 2002 up to Spring 2006. The first phase of the project essentially sought to answer the question: "Can you write a test to assess topics typically encountered in introductory statistics?" Book One presents the results utilized in answering this question in the affirmative. The bulk of the results present the development and evolution of the items, primarily relying on objective metrics to gauge effectiveness but also incorporating student feedback. The second phase boils down to: "Now that you have the test, what else can you do with it?" This includes an exploration of Cronbach's alpha, the most commonly-used measure of test reliability in the literature. An online version of the SCI was designed, and its equivalency to the paper version is assessed. Adding an extra wrinkle to the online SCI, subjects rated their answer confidence. These results show a general positive trend between confidence and correct responses. However, some items buck this trend, revealing potential sources of misunderstandings, with comparisons offered to the extant statistics and probability educational research. The third phase is a re-assessment of the SCI: "Are you sure?" A factor analytic study favored a uni-dimensional structure for the SCI, although maintaining the likelihood of a deeper structure if more items can be written to tap similar topics. A shortened version of the instrument is proposed, demonstrated to be able to maintain a reliability nearly identical to that of the full instrument. Incorporating student feedback and a faculty topics survey, improvements to the items and recommendations for further research are proposed. The state of the concept inventory movement is assessed, to offer a comparison to the work presented on the SCI. Finally, the dissertation concludes with a summary of the four years' progress, acknowledging that work is never complete but that the results thus far place the SCI in a strong position to grow for years to come

Adaptive Learning Module for Introduction to Materials Science

There is a growing need for individualized instructional designs in 4-year institutions due to: 1) growing undergraduate enrollment trends across the U.S., 2) broader diversity in students backgrounds, and 3) the growing prevalence of asynchronous remote learning. With the growth in computer technology in education, tools such as adaptive intelligent tutoring systems have been developed to respond to this need. However, these commercially available tools come with high costs and typically lack the emphasis on conceptual learning, since activities and assessments that promote conceptual understanding are difficult to make. As a result, this dissertation addresses this limitation by expanding the research in conceptual learning with the development of the Crystallography Adaptive Learning Module (CALM). This novel tool incorporates activities and assessments that promote engagement and conceptual learning. The content is focused on the topic of crystallography as it was found to be challenging for many undergraduate students in materials science introductory courses, but the design is general and could be applied to any conceptually challenging STEM topic. As various factors can affect conceptual learning, the first part of this work compared conceptual gains in students who enrolled in different modalities – in-person face-to-face (F2F) versus asynchronous online. The results showed the conceptual learning gains were significantly lower for the online, asynchronous mode and confirmed that the different modality can affect conceptual learning in students. Next, as part of the development of an adaptive conceptually based assessment in the CALM, the study examined experts’ ability to predict the difficulty level of a concept question by comparing student actual performance with instructor predictions. No clear correlation was found, suggesting actual student performance data is needed to make such conceptually driven learning adaptive tools. Lastly, the role of adaptivity was assessed in terms of conceptual learning performance and student engagement with learning materials. In a quasi-experimental, randomized study were either guided students through the learning materials with adaptive feedback (via CALM) or were allowed to self-select the amount and content of the learning materials they received. Within this comparison, student overall academic performance (cumulative grade point average, GPA) was also considered. Results showed that the level of engagement with interactive learning content was higher for the CALM instructional design, however, there was no statistically significantly difference in performance between the two instructional designs and among subgroups of the same cumulative GPA groups. There was some evidence that suggests the adaptive individualized feedback pedagogy might improve student conceptual understanding for the middle cumulative GPA group. In summary, this dissertation confirms of various factors to consider that can affect student conceptual understanding; course modality, pedagogical design, and student overall academic performance, The CALM individualized computer tool developed was found to be a promising tool that improve student participation and conceptual understanding in certain group of students and should be continued in upgrading and expanding research toward other topics. Findings from this dissertation can benefit educators, researchers, and software developers who focus on enhancing student conceptual learning in challenging STEM topics

Using visual representations to improve instructional materials for distance education computing students

Understanding how to develop instructional materials for distance education students is a challenging problem, but it is exacerbated when a domain is complex to teach, such as computer science. Visual representations have a history of use in computing as a means to alleviate the difficulties of learning abstract concepts. However, it is not clear whether improvements observed are as a result of improvements in the visual representations used in instructional materials or due to individual differences in students. This research examines the two themes of individual differences and visual representation in order to investigate how they collectively impact on improving instructional materials for distance education students studying computer science. It investigates the impact of different representations on learning while additionally investigating the relationship between individual differences and student learning.The research in this thesis shows that visual representations are important in designing instructional materials. In particular, texts with visual representations have the power to cue students to perceive instructional materials as easier to process and more engaging.Investigation into the impact of concrete high-imagery versus abstract low-imagery visual representations illustrated that concrete visual representations incurred fewer cognitive overheads for computer science students and were able to ameliorate the challenges of learning computing.The research in this thesis into individual differences demonstrated that Imagers did benefit more from studying instructional materials containing text with visual components. However the research indicates that appropriate selection of individual difference tests is dependent upon the application, i.e., whether the results are to be used to assess generalised tendencies or episodes in learning and whether the tests examine underlying approaches to cognition or practices in education.An underlying question was whether students studying instructional materials containing low-imagery visual representations would cope as well as those studying high-imagery ones. Accomplished learners demonstrated that they could perform as well as with those receiving high-imagery visual representations. However, studying and recalling these materials did incur more cognitive processing.This thesis argues that improving instructional materials by including appropriate visual representations is a useful basis for improving learning for distance education computer science students

Elementary and secondary science teachers negotiation of controversial science content: The relationships among prior conception appropriation, thinking disposition, and learning about geologic time

A major component of the values people place on science and their attitude toward it is their openness to new ideas or overall open-mindedness. An individual’s values and attitudes become integrally connected to their prior knowledge and conceptions regarding science and science content. Sometimes the nature of a natural phenomenon and the scientific explanation for the phenomenon is controversial. A controversial scientific concept is one that evokes emotion and forces individuals to assess the values associated with this content and make assessments of their attitudes toward it. This is especially true during learning. The purpose of this study was to provide evidence on how prior knowledge and existing conceptions are related to open-mindedness when learning science content that is regarded as controversial. The participants for this study consisted of 7 elementary science teachers and 8 secondary science teachers. Data collected for the study included the determination of how individuals assessed and used their prior/existing conceptions when learning controversial science content based on individual interviews, an individual’s level of open-mindedness as measured by the Actively Open-minded Thinking scale (AOT) and determined through the interviews, and the assessment of the change in an individual’s level of knowledge regarding geologic time as measured by the Geoscience Concept Inventory (GCI). The investigation consisted of multiple case studies analyzed within cases and across cases. The teachers’ use of their prior conceptions was determined through the coding of interviews based on the four appropriation modes of Integration, Differentiation, Exchange, and Bridging. Results from the interview data showed that 53% of the teachers differentiated their existing conceptions from new geologic time conceptions, while 47% integrated new conceptions with their prior conceptions. In addition, 40% of the teachers exhibited a bimodal appropriation of their existing conceptions. Bridging and exchange were the secondary appropriation modes observed among bimodal appropriators. No relationships were found between the teachers’ thinking disposition (open-mindedness) and their level of geologic time knowledge, nor where there any relationships found between the teachers’ prior conception appropriation and their geologic time knowledge or their appropriation and thinking disposition