Measuring programming competence by assessing chunk structures in a code transcription task

In a simple transcription task in which sections of Java program code are copied by freehand writing, it is demonstrated that chunk related temporal signals are sufficiently robust to permit the measurement of programming competence. An experiment with 24 participants revealed that the number of views of the stimulus per trial and the duration of writing per stimulus view are both strongly correlated with independent measures of Java competence

Applying temporal chunk signals analysis to measure programming competence by the transcription of Java program code

This thesis investigates the basis for a novel method of quickly and efficiently assessing programming comprehension. It investigates the feasibility of assessing learners’ mental chunk structures, and their temporal chunk signals, as a way of measuring their competence. The focus is on the Java programming language. The thesis investigates the feasibility of chunk-based measures in two different simple transcription tasks: view display, where stimulus is visible at all times; and hide and show, where the stimulus is only made visible when a participant presses a special button. University computer science students and faculty are the target group. Chunking theory is utilised to define three chunking measures of competence and to anticipate how they would vary across participants with different degrees of Java competence. The measures are as follows: (1) the number of characters transcribed per view (or the number of views) of the Java program code; (2) the time spent writing between the views; and (3) the duration of pauses before writing each written character. Ninety-six participants participated in the three experiments, transcribing on graphics tablets in experimental settings, and evidence of chunking’s essential role in transcription tasks was revealed. Significant relationships were discovered between the chunking measures of competence and independent measures of Java competence (Java familiarity scores and students’ final test marks (for the third experiment)). The third experiment included a longitudinal post-test component spanning three months of learning, in which changes to the mean scores in characters per view, writing-times, and pauses reflected the students’ amount of learning

Competence assessment by stimulus matching: an application of GOMS to assess chunks in memory

It has been shown that in hand-written transcription tasks temporal micro-behavioral chunk signals hold promise as measures of competence in various domains (e.g., Cheng, 2014). But data capture under that an approach requires the use of graphics tablets which are relatively uncommon. In this paper we propose and explore an alternative method – Competence Assessment by Stimulus Matching (CASM). This new method uses simple mouse-driven interfaces to produce temporal chunk signals as measures of learner’s ability. However, it is not obvious what features of CASM will produce effective competence measures and the design space of CASM tasks is large. Thus, this paper uses GOMS modelling in order to explore the design space to find factors that will maximize the discrimination of chunk measures of competence. Results of a pilot experiment show that CASM has potential in using chunk signals to measure competence in the domain of English language

Modeling second language learners' interlanguage and its variability: a computer-based dynamic assessment approach to distinguishing between errors and mistakes

Despite a long history, interlanguage variability research is a debatable topic as most paradigms do not distinguish between competence and performance. While interlanguage performance has been proven to be variable, determining whether interlanguage competence is exposed to random and/or systematic variations is complex, given the fact that distinction between competence-dependent errors and performance-related mistakes should be established to best represent the interlanguage competence. This thesis suggests a dynamic assessment model grounded in sociocultural theory to distinguish between errors and mistakes in texts written by learners of French, to then investigate the extent to which interlanguage competence varies across time, text types, and students. The key outcomes include: 1. An expanded model based on dynamic assessment principles to distinguish between errors and mistakes, which also provides the structure to create and observe learners’ zone of proximal development; 2. A method to increase the accuracy of the part-of-speech tagging procedure whose reliability correlates with the number of incorrect words contained in learners’ texts; 3. A sociocultural insight into interlanguage variability research. Results demonstrate that interlanguage competence is as variable as performance. The main finding shows that knowledge over time is subject to not only systematic, but also unsystematic variations

Exploring A New Path For School Climate & Safety Assessment

The passage of the No Child Left Behind (NCLB) Act in 2001, ushered in a new era of federal oversight in state educational accountability systems. While the act may have succeeded in identifying schools in need of support and creating data systems to help inform parents and assist educators in establishing clear and consistent goals, the state accountability systems created under this law were widely criticized for their narrow academic focus and failure to include the holistic and multifaceted nature of school quality. In response, the federal government replaced NCLB in 2015 with The Every Student Succeeds Act (ESSA). The new law included a provision widely referred to as the “Fifth Indicator” of student success, which was to be non-academic in nature. To address this indicator in state accountability plans, the federal government specifically recommended several strategies, to include measures of school climate and safety. Despite this recommendation and despite decades of research to support its inclusion, only a handful of states have adopted such measures. As a result, schools and districts in most states are left to develop their own systems of school climate assessment and improvement. The primary purpose of this qualitative case study was to explore the efforts of a single Florida school district in their quest to develop such a system. The Sunshine School District (SSD) had begun looking at innovative ways to assess and improve school climate as the ESSA was passed. An instrument had been developed that was loosely based on research and had not been tested in school settings. This instrumental case study provided an in-depth examination of the literature and an analysis of archival data to help refine the instrument and prepare it for a pilot test in nine district schools. Through this study, a follow-up focus group with pilot participants was conducted to determine whether the instrument held promise as a means to assess school climate and safety, as well as drive improvement. Analysis of the data revealed participants found the instrument to be flexible, useful, and effective – particularly as it pertains to the assessment of school climate and safety practices and establishing improvement goals. The analysis additionally revealed the instrument can be lengthy and may not be equally applicable to all schools and grade levels. Although the instrument needs further refinement, pilot participants reported it to still be effective and beneficial as an informal assessment and improvement tool

Knowledge restructing and the development of expertise in computer programming

This thesis reports a number of empirical studies exploring the development of expertise in computer programming. Experiments 1 and 2 are concerned with the way in which the possession of design experience can influence the perception and use of cues to various program structures. Experiment 3 examines how violations to standard conventions for constructing programs can affect the comprehension of expert, intermediate and novice subjects. Experiment 4 looks at the differences in strategy that are exhibited by subjects of varying skill level when constructing programs in different languages. Experiment 5 takes these ideas further to examine the temporal distribution of different forms of strategy during a program generation task. Experiment 6 provides evidence for salient cognitive structures derived from reaction time and error data in the context of a recognition task. Experiments 7 and 8 are concerned with the role of working memory in program generation and suggest that one aspect of expertise in the programming domain involves the acquisition of strategies for utilising display-based information. The final chapter attempts to bring these experimental findings together in terms of a model of knowledge organisation that stresses the importance of knowledge restructuring processes in the development of expertise. This is contrasted with existing models which have tended to place emphasis upon schemata acquisition and generalisation as the fundamental modes of learning associated with skill development. The work reported here suggests that a fine-grained restructuring of individual schemata takes places during the later stages of skill development. It is argued that those mechanisms currently thought to be associated with the development of expertise may not fully account for the strategic changes and the types of error typically found in the transition between novice, intermediate and expert problem solvers. This work has a number of implications for existing theories of skill acquisition. In particular, it questions the ability of such theories to account for subtle changes in the various manifestations of skilled performance that are associated with increasing expertise. Secondly, the work reported in this thesis attempts to show how specific forms of training might give rise to the knowledge restructuring process that is proposed. Finally, the thesis stresses the important role of display-based problem solving in complex tasks such as programming and highlights the role of programming language notation as a mediating factor in the development and acquisition of problem solving strategies

Measuring and assessing typing skills in writing research

In keyboard writing, typing skills are considered an important prerequisite of proficient text production. We describe the design, implementation, and application of a standardized copy-typing task in order to measure and assess individual typing fluency. A test-retest analysis indicates the instrument’s reliability. While the task has been developed across eleven different languages and the related keyboard layouts, we here refer to a corpus of Dutch copy tasks (n = 1682). Analyses show that copying speed non-linearly varies with age. Bayesian analyses reveal differences in the typing performance and the underlying distributions of inter-key intervals between the different task components (e.g., lexical vs. non-lexical materials; high-frequent vs. lowfrequent bigrams). Based on these findings it is strongly recommended to include copy-task measures in the analysis of keystroke logging data in writing studies. This supports a better comparability and interpretability of keystroke data from more complex or communicatively-embedded writing tasks across individuals. Further potential applications of the copy task for writing research are explained and discussed

Data-driven machine translation for sign languages

This thesis explores the application of data-driven machine translation (MT) to sign languages (SLs). The provision of an SL MT system can facilitate communication between Deaf and hearing people by translating information into the native and preferred language of the individual. We begin with an introduction to SLs, focussing on Irish Sign Language - the native language of the Deaf in Ireland. We describe their linguistics and mechanics including similarities and differences with spoken languages. Given the lack of a formalised written form of these languages, an outline of annotation formats is discussed as well as the issue of data collection. We summarise previous approaches to SL MT, highlighting the pros and cons of each approach. Initial experiments in the novel area of example-based MT for SLs are discussed and an overview of the problems that arise when automatically translating these manual-visual languages is given. Following this we detail our data-driven approach, examining the MT system used and modifications made for the treatment of SLs and their annotation. Through sets of automatically evaluated experiments in both language directions, we consider the merits of data-driven MT for SLs and outline the mainstream evaluation metrics used. To complete the translation into SLs, we discuss the addition and manual evaluation of a signing avatar for real SL output

Japanese EFL Learners’ Pragmatic Development in the Production of Speech Acts Drawing on ACT-R Model and Skill Acquisition Theory

This mixed-methods quasi-experimental study explored the development of pragmatic competence of lower-proficiency EFL learners in their university English classes in Japan. Although pragmatic competence has emerged as a key topic within the field of interlanguage pragmatics (ILP), almost all studies have examined L2 learners’ language use rather than pragmatic development focusing on learning outcomes than process. This study investigates both learners’ language use and development, in order to draw a more comprehensive picture of pragmatic development. It also attempts to identify the mechanisms that drive this development by employing a framework of Adaptive Control Thought-Rational (ACT-R) theory in tandem with skill acquisition theory, which is a promising but underexplored framework in the L2 pragmatic development context. As such, this study aims to fill a gap in the research literature and make a theoretical contribution by showing the potential of the framework to account for learners’ pragmatic development. For this study, I recruited 120 Japanese EFL learners making up four intact classes to examine the development over one term (14 weeks) of their skills for producing speech acts after receiving pragmatic instruction. The development was examined both in terms of knowledge and processing ability with more focus on the latter to produce speech acts. Four types of speech acts were chosen for this experiment: request and refusal speech acts, for which specific instruction was provided; and complaint and disagreement speech acts, for which no instruction was provided. Request and refusal were selected as they were most widely studied, and complaints and disagreements were selected as they are relatively similar in nature to request and refusal speech acts and a good candidate to examine learners’ ability of knowledge extension. Quantitative and qualitative analyses were employed to see how much learners improved their production of request and refusal speech acts - in light of the effectiveness of instruction they had received. A similar analysis was carried out on the uninstructed speech acts of complaint and disagreement to assess their capability to extend their learned knowledge from request and refusal making to the production of new speech acts, namely to assess their processing capability. The results showed that the participants in the treatment groups (TGs) improved in the production of both instructed and uninstructed speech acts by developing their knowledge and processing ability. The development of such knowledge was assessed by measuring the TGs’ improvement in the use of politeness strategies, which are associated with declarative knowledge. As for the development of their processing ability, this was assessed in two ways: in terms of their ability to select contextually appropriate strategies and to apply their learned knowledge sufficiently to produce uninstructed speech acts, these being associated with procedural knowledge. Since the application of the learned speech act schema enables learners to produce ostensibly ‘new’ speech acts with relative ease, not from scratches. This frees up most of the working memory to be available for other purposes, such as planning what to say next, and looking for more sophisticated expressions. This was reflected in the results of this experiment that showed, following instruction, the use of a wider range of strategies and more sophisticated lexical and syntactic expressions. However, the results did show that the participants were still in an early stage of proceduralisation and needed further practice to improve their processing ability to move toward automatisation. This study has pedagogical, theoretical, and methodological implications. Pedagogically, there are several implications afforded by a clearer understanding of learning processes that can be used to revise the EFL curriculum. Theoretically, by showing how pragmatic competence develops in an EFL classroom, this study shows the potential of the ACT-R model, partially revised to apply to this study, to elucidate the operational mechanism of pragmatic ability. Methodologically, this study shows how the application of the revised model I formulated through adaptation and clarification of a range of interpretations of the ACT-R model can better account for proceduralisation in pragmatic development, raising implications for allowing related research to move forward in an otherwise muddled ongoing discussion in the field