Using latent-semantic analysis and network analysis for monitoring conceptual development

This paper describes and evaluates CONSPECT (from concept inspection), an application that analyses states in a learner’s conceptual development. It was designed to help online learners and their tutors monitor conceptual development and also to help reduce the workload of tutors monitoring a learner’s conceptual development. CONSPECT combines two technologies - Latent Semantic Analysis (LSA) and Network Analysis (NA) into a technique called Meaningful Interaction Analysis (MIA). LSA analyses the meaning in the textual digital traces left behind by learners in their learning journey; NA provides the analytic instrument to investigate (visually) the semantic structures identified by LSA. This paper describes the validation activities undertaken to show how well LSA matches first year medical students in 1) grouping similar concepts and 2) annotating text

Monitoring conceptual development with text mining technologies: CONSPECT

This paper evaluates CONSPECT, a service that analyses states in a learner’s conceptual development. It combines two technologies – Latent Semantic Analysis to analyse text and Network Analysis (NA) to provide visualisations – into a technique called Meaningful Interaction Analysis (MIA). CONSPECT was designed to help both online learners and their tutors monitor their conceptual development. This paper reports on the validation experiments undertaken to determine how well LSA matches first year medical students in clustering concepts and in annotating text. The validation used several techniques, including card sorting and Likert scales. CONSPECT produces almost ‘peer’ quality results and what remains to be tested is whether it improves with more advanced learners. One of the experiments showed an average 0.7 correlation between humans and CONSPECT

Culturel difference in structure of categories in Denmark and China

There is a difference in how Danish and Chinese people group object, method and concepts into categories. Difference in these points affect the information structure in applications, which involve menus, links and directories. This study involves groups from Chinese and Danish cultures and investigates how these two cultures group cards into different categories and how their cultural backgrounds affect the structure of their categories. Card Sort, Information Structure, Cultural Difference and Usability

Visual Literacy of Molecular Biology Revealed through a Card-Sorting Task

Visual literacy, which is the ability to effectively identify, interpret, evaluate, use, and create images and visual media, is an important aspect of science literacy. As molecular processes are not directly observable, researchers and educators rely on visual representations (e.g., drawings) to communicate ideas in biology. How learners interpret and organize those numerous diagrams is related to their underlying knowledge about biology and their skills in visual literacy. Furthermore, it is not always obvious how and why learners interpret diagrams in the way they do (especially if their interpretations are unexpected), as it is not possible to “see” inside the minds of learners and directly observe the inner workings of their brains. Hence, tools that allow for the investigation of visual literacy are needed. Here, we present a novel card-sorting task based on visual literacy skills to investigate how learners interpret and think about DNA-based concepts. We quantified differences in performance between groups of varying expertise and in pre- and postcourse settings using percentages of expected card pairings and edit distance to a perfect sort. Overall, we found that biology experts organized the visual representations based on deep conceptual features, while biology learners (novices) more often organized based on surface features, such as color and style. We also found that students performed better on the task after a course in which molecular biology concepts were taught, suggesting the activity is a useful and valid tool for measuring knowledge. We have provided the cards to the community for use as a classroom activity, as an assessment instrument, and/or as a useful research tool to probe student ideas about molecular biology

Extracting Build Changes with BUILDDIFF

Build systems are an essential part of modern software engineering projects. As software projects change continuously, it is crucial to understand how the build system changes because neglecting its maintenance can lead to expensive build breakage. Recent studies have investigated the (co-)evolution of build configurations and reasons for build breakage, but they did this only on a coarse grained level. In this paper, we present BUILDDIFF, an approach to extract detailed build changes from MAVEN build files and classify them into 95 change types. In a manual evaluation of 400 build changing commits, we show that BUILDDIFF can extract and classify build changes with an average precision and recall of 0.96 and 0.98, respectively. We then present two studies using the build changes extracted from 30 open source Java projects to study the frequency and time of build changes. The results show that the top 10 most frequent change types account for 73% of the build changes. Among them, changes to version numbers and changes to dependencies of the projects occur most frequently. Furthermore, our results show that build changes occur frequently around releases. With these results, we provide the basis for further research, such as for analyzing the (co-)evolution of build files with other artifacts or improving effort estimation approaches. Furthermore, our detailed change information enables improvements of refactoring approaches for build configurations and improvements of models to identify error-prone build files.Comment: Accepted at the International Conference of Mining Software Repositories (MSR), 201

Using knowledge elicitation techniques to establish a baseline of quantitative measures of computational thinking skill acquisition among university computer science students.

The purpose of this study was to establish a baseline of quantitative measures of computational thinking skill acquisition as an aid in evaluating student outcomes for programming competency. Proxy measures for the desired skill levels were identified that reliably differentiate the conceptual representations of computer science students most likely, from those least likely, to have attained the desired level of programming skill. Insights about the development of computational thinking skills across the degree program were gained by analyzing variances between these proxy measures and the conceptual representations of cross-sections of participating students partitioned by levels of coursework attainment, programming experience, and academic performance. Going forward, similar measures can provide a basis for quantitative assessment of individual attainment of the desired learning outcome. The voluntary participants for this study were students enrolled in selected undergraduate computer science courses at the University. Their conceptual representations regarding programming concepts were elicited with a repeated, open card sort task and stimuli set as used for prior studies of computer science education. A total of 135 students participated, with 124 of these providing 296 card sorts. Differences between card sorts were quantified with the edit distance metric which provided a basis for statistical analysis. Card sorts from cross-sections of participants were compared and contrasted using graph theory algorithms to calculate measures of average segment length of minimum spanning trees (orthogonality), to identify clusters of highly similar card sorts, and to reduce clusters down to individual exemplar card sorts. Variances in distance between the card sorts of cross-sections of participants and the identified exemplars were analyzed with one-way ANOVAs to evaluate differences in development of conceptual representations relative to coursework attainment and programming experience. Findings Collections of structurally similar card sorts were found to align with categorizations identified in earlier studies of computer science education. A logistic regression identified two exemplar sorts representing deep factor categorizations that reliably predicted those participants most, and least likely to have attained the desired level of programming skill. Measures of proximal distance between participants' card sorts and these two exemplars were found to decrease, indicating greater similarity, as students attained progressive coursework milestones. This finding suggests that proximal distances to exemplars of common categorizations for this stimuli set can effectively differentiate conceptual development levels of students between, as well as within, cross-sections selected by achievement of coursework milestones. Measures of proximal distances to one exemplar of deep factor categorization were found to increase, indicating less similarity, as participants’ levels of programming experience increased. This finding was contrary to the theoretical framework for skill acquisition. Further analysis found that variances in experience level as captured by the study instrument were not equally distributed among the cross-sections. The preponderance of participants reporting greater levels of experience were degree majors not required to enroll in the courses most likely to develop that specific conceptualization. Therefore, for this deep factor categorization, instruction was found to have a greater influence on conceptual development than programming experience. However, it is possible that other categorizations, such as those related to software engineering technology, may be found to be more influenced by experience. The orthogonality of participant card sorts was found to increase with each category of increase in academic performance, as in keeping with prior studies. Orthogonality also increased with greater levels of programming experience as expected by the theoretical framework. However, since experience was not equally distributed across categories of coursework achievement, the relationship between the orthogonality of participant card sorts and milestones of coursework achievement was not found to be statistically significant overall. Based on the findings, the researcher concludes that a baseline of quantitative measures of computational thinking skills can be constructed based upon categorizations of elicited conceptual representations and associated exemplar card sorts. Eleven categorizations identified in a prior study of computer science seniors appear to represent reasonable expectations for deep factor categorizations. Follow up research is recommended (a) to identify for each categorization the exemplar card sorts that may be specific to different degree majors, and (b) to identify which categorizations may be more influenced by programming experience than by instruction. Given an elicitation tool that prompts for the specific categorizations and a set of exemplar representations as proposed above, instructional programs can establish expected ranges of proximal distance measures to specific exemplars. These exemplars should be selected according to particular categorizations, degree majors, and coursework milestones. These differentiated measures will serve as evidence that students are meeting the instructional program learning objective for developing competency in the design and implementation of computer-based solutions

Word co-occurrence graphs for design and manufacture knowledge mapping

Design & Manufacture Knowledge Mapping is a critical activity in medium-to-large organisations supporting many organisational activities. However, techniques for effective mapping of knowledge often employ interviews, consultations and appraisals. Although invaluable in providing expert insight, the application of such methods is inherently intrusive and resource intensive. This paper presents word co-occurrence graphs as a means to automatically generate knowledge maps from technical documents and validates against expert generated knowledge maps