An Evaluation Of Learning Employing Natural Language Processing And Cognitive Load Assessment

One of the key goals of Pedagogy is to assess learning. Various paradigms exist and one of this is Cognitivism. It essentially sees a human learner as an information processor and the mind as a black box with limited capacity that should be understood and studied. With respect to this, an approach is to employ the construct of cognitive load to assess a learner\u27s experience and in turn design instructions better aligned to the human mind. However, cognitive load assessment is not an easy activity, especially in a traditional classroom setting. This research proposes a novel method for evaluating learning both employing subjective cognitive load assessment and natural language processing. It makes use of primary, empirical and deductive methods. In details, on one hand, cognitive load assessment is performed using well-known self-reporting instruments, borrowed from Human Factors, namely the Nasa Task Load Index and the Workload Profile. On the other hand, Natural Language Processing techniques, borrowed from Artificial Intelligence, are employed to calculate semantic similarity of textual information, provided by learners after attending a typical third-level class, and the content of the class itself. Subsequently, an investigation of the relationship of cognitive load assessment and textual similarity is performed to assess learning

Bridging the Gulf of Envisioning: Cognitive Design Challenges in LLM Interfaces

Large language models (LLMs) exhibit dynamic capabilities and appear to comprehend complex and ambiguous natural language prompts. However, calibrating LLM interactions is challenging for interface designers and end-users alike. A central issue is our limited grasp of how human cognitive processes begin with a goal and form intentions for executing actions, a blindspot even in established interaction models such as Norman's gulfs of execution and evaluation. To address this gap, we theorize how end-users 'envision' translating their goals into clear intentions and craft prompts to obtain the desired LLM response. We define a process of Envisioning by highlighting three misalignments: (1) knowing whether LLMs can accomplish the task, (2) how to instruct the LLM to do the task, and (3) how to evaluate the success of the LLM's output in meeting the goal. Finally, we make recommendations to narrow the envisioning gulf in human-LLM interactions

Establishing knowledge and skill in a novel system-supervisory task: an application to automated mail sorting

This thesis aims to establish methods for identifying and training the knowledge and skills of operating a novel automated system still undergoing final design and construction. The absence of operating experience requires the characteristics of the system to be examined so that the future tasks of supervisors can be anticipated in order to address human factors design. This work is carried out in the context of an 'Integrated Mail Processor' (IMP)—a highly automated letter sorting machine being developed by Royal Mail. [Continues.

A semi-automatic computer-aided assessment framework for primary mathematics

Assessment and feedback processes shape students behaviour, learning and skill development. Computer-aided assessments are increasingly being used to support problem-solving, marking and feedback activities. However, many computer-aided assessment environments only replicate traditional pencil-and-paper tasks. Attention is on grading and providing feedback on the final product of assessment tasks rather than the processes of problem solving. Focusing on steps and problem-solving processes can help teachers to diagnose strengths and weaknesses, discover problem-solving strategies, and to provide appropriate feedback to students. This thesis presents a semi-automatic framework for capturing and marking students solution steps in the context of elementary school mathematics. The first focus is on providing an interactive touch-based tool called MuTAT to facilitate interactive problem solving for students. The second focus is on providing a marking tool named Marking Assistant which utilises the case-based reasoning artificial intelligence methodology to carry out marking and feedback activities more efficiently and consistently. Results from studies carried out with students showed that the MuTAT prototype tool was usable, and performance scores on it were comparable to those obtained when paper-and-pencil was used. More importantly, the MuTAT provided more explicit information on the problem-solving process, showing the students thinking. The captured data allowed for the detection of arithmetic strategies used by the students. Exploratory studies conducted using the Marking Assistant prototype showed that 26% savings in marking time can be achieved compared to traditional paper-and-pencil marking and feedback. The broad feedback capabilities the research tools provided can enable teachers to evaluate whether intended learning outcomes are being achieved and so decide on required pedagogical interventions. The implications of these results are that innovative CAA environments can enable more direct and engaging assessments which can reduce staff workloads while improving the quality of assessment and feedback for students

Practice, principles, and theory in the design of instructional text

This study is concerned with an analysis of the research arising from three quite different perspectives on instructional text - the `physical characteristics' research (legibility, layout, and readability), the `improvement of text' research (visual illustrations, adjunct aids, and typographical cueing), and the `learning theories' research (representation of knowledge, human memory, and quality of learning). From this analysis there is synthesised principles for the design of instructional text against which heuristic practice in text design is evaluated and from which a nascent theory of instructional text design is evolved. The principles derived from the various research perspectives provide a basis for the manipulation of text design elements in order to ensure that (a) existing knowledge in the reader can be activated, and (b) new knowledge can be assimilated in a manner facilitative of comprehension by (i) presentation in a structured and organised way, and (ii) appropriately highlighted through verbal and typographic cueing supported, as required, by verbal illustration and organisation. The emerging theory of instructional text design suggests: a topical analysis to determine the heirarchic relationship of ideas within the topic and the desired learning outcomes or objectives; a consideration of the linguistic aspects of the text; a consideration of the role of visual illustrations; and a consideration of the physical parameters of the text. These activities are concerned, respectively, with the design areas of structure and organisation, readability, visual illustration, and legibility, and are summed up in the acronym SORVIL

Expertise Development in Commercial Property Valuation Practice

Ever since the issue of inaccuracy and variance in commercial property valuation was first documented in the mid-80s by Brown (1985) and Hager and Lord (1985), many researchers have investigated the complex factors involved in effective problem solving in the valuation domain, focusing on the valuer and the valuation process. Several behavioural issues, including heuristics, have been noted to affect valuation outcomes. There is a growing literature on understanding the concept of expertise, especially using the field of cognitive psychology, and the present research explores valuer’s cognitions in a commercial valuation context. The study aimed to determine how the role of valuers’ cognitions and cognitive structures are crucial in furthering our understanding of effective valuation problem solving, as well as improving valuer training efforts. The research was undertaken from a ‘Critical Realist’ perspective, and used a knowledge elicitation method called ‘Cognitive Task Analysis’. Data were collected through a ‘Verbal Protocol Analysis’ (VPA) of a simulated commercial valuation exercise based on a real building, using semi-structured interviews. Six subjects (comprising two expert valuers, two intermediate valuers and two novice valuers) participated in the simulated valuation and in the follow-up interviews. Two further experts were interviewed to validate the findings. Content and event-sequence analysis were performed on the data collected from the simulated valuation to yield the knowledge states, problem-solving techniques (‘operators’) and strategies used by valuers. Mapping of thought processes revealed that expert and intermediate valuers had better and well-structured patterns of thought which demonstrate greater degrees of cohesiveness and interrelatedness between problem-solving operators. Centred on data interpretation and meta-reasoning activities, expert and intermediate valuers used the problem-solving operators initially to schedule valuation analysis or establish valuation strategies, and to re-interpret and diagnose previously acquired information to update the outcome of their past valuations. Novice valuers’ structured processes of solving the valuation problem show fewer linkages between problem-solving operators, which may suggest underdeveloped cognitive structure or quick disengagement from task. The results also show that where available data is inadequate, valuers solve an overall valuation problem by dividing the problem into a number of sub-problems that are solved by engaging in two main types of thinking: analytical and creative. These two levels of thinking enable the valuer to integrate available data with his/her existing knowledge through forward and retrospective (‘backwards’) reasoning. However, there were effects associated with level of expertise in the way these cognitive processes are used, with the expert and intermediate valuers being more fluid, thorough and comprehensive than the novice valuers. This enabled the expert and intermediate valuers to develop a greater number of more-sophisticated solutions to challenging valuation problems, and these were more likely to be immediately followed by meta-reasoning related activities or further exploration of data to justify the solutions generated. Novice valuers could not generate such well-developed solutions indicating that they were much more superficial in their valuation problem solving. These processes are discussed and synthesised into a descriptive model of expert-valuer cognitive structure for undertaking valuation of a commercial property, in order to show an understanding of how valuers integrate the various cognitive processes to determine the value of a property based on available information. The research concludes with an assessment of the implications for valuation training and education

Empirical Studies of Novices Learning Programming

This thesis is concerned with the problems that novices have in learning to program: in particular it is concerned with the difficulties experienced by novices learning at a distance, using instructional materials which have been designed especially for novices. One of the major problems for novices is how to link the new information which they encounter with their existing knowledge. Du Boulay, O'Shea and Monk (1981) suggest helping novices to bridge the gap between their existing knowledge and new information by teaching via a conceptual model, which serves to explain the new information in familiar terms.In this thesis the difficulties which novices have when learning to program with the help of a conceptual model were investigated. The curricula and conceptual models of four different programming languages are examined, all of which were designed to teach novices. Du Boulay, O'Shea and Monk (1981) have suggested criteria for analysing conceptual models. It is argued that these criteria, however, do not address the presentation of the conceptual model, and so are insufficient to evaluate them. An additional form of analysis was proposed and used, in addition to the criteria offered by Du Boulay et al. This is a way of describing the conceptual model which distinguishes three views of the conceptual model: state, procedure and function, and which highlights the different aspects which are important for the novice learner by identifying the different kinds of knowledge which are necessary to understand the conceptual model. This analysis of the conceptual models showed that the environments are not as exemplary as the du Boulay et al's criteria suggest, and indicated that three of the environments, SOLO, PT501 and DESMOND, lack a functional representation, and that the fourth, Open Logo, has other different problems.An empirical study was carried out to study the transfer effects of learning two of the languages, a high level and a low level language, sequentially. There was no evidence for such transfer effects. The difficulties novices have in learning the four different languages were also investigated. These studies show that even though the novices were studying environments designed for novices learning at a distance, they did not develop good levels of competence, and the problems they had fall into two main categories: programming and pedagogical.Although the different languages had different aims and curricula, novices had some problems which were common to all or most of the languages. These included understanding flow of control, developing and using programming plans, developing accurate mental models, and in the high level languages, understanding recursion. It is argued that some of these problems are related to the conceptual models. In particular, the difficulties novices had in developing and using plan knowledge, which is one of their main problems, can be explained by the lack of an appropriate functional description in the languages.The subjects' pedagogical problems arose from the relationship between the style and structure of the curriculum, its content, and the subjects themselves. In all the four texts the teaching material is very carefully structured and it is suggested that this may encourage the learner to adopt an over-dependent attitude towards the text, and in some cases, to work at an inappropriate syntactic level.The relationship between the distance learning situation and the novice programmer is discussed, and recommendations are made for improving the curricula used for teaching novices programming