TELMA Cross Experiment Guidelines

Cerulli, M., Pedemonte, B., Robotti, E. (eds.). Internal Report, R.I. 01/07, I.T.D. - C.N.R., GenovaThis document contains the guidelines developed by members of TELMA as a means for planning, conducting, and analysing a cross experiment aimed at contributing to the construction of a shared research perspective among TELMA teams . This is the product of the PhD students and young researchers that brought forward the whole activity. The actual experimental phase was proceeded by a reflective phase in which an agreement was achieved on what research questions to address during the experiment. On this basis the first version of the guidelines document was built, containing all the research questions to be addressed, but also the experimental plans for each team. This included the employed didactical functionalities of the considered ICT tools, indications of the experimental settings, and the methods of data collection and analysis. During the whole experimental phase, the document was constantly updated, and shared among the involved persons which were periodically required to compare the different activities and reflections brought forward by all the teams

New measurement paradigms

This collection of New Measurement Paradigms papers represents a snapshot of the variety of measurement methods in use at the time of writing across several projects funded by the National Science Foundation (US) through its REESE and DR K–12 programs. All of the projects are developing and testing intelligent learning environments that seek to carefully measure and promote student learning, and the purpose of this collection of papers is to describe and illustrate the use of several measurement methods employed to achieve this. The papers are deliberately short because they are designed to introduce the methods in use and not to be a textbook chapter on each method. The New Measurement Paradigms collection is designed to serve as a reference point for researchers who are working in projects that are creating e-learning environments in which there is a need to make judgments about students’ levels of knowledge and skills, or for those interested in this but who have not yet delved into these methods

Improving Dynamic Decision Making Through Training and Self-Reflection

The modern business environment requires managers to make effective decisions in a dynamic and uncertain world. How can such dynamic decision making (DDM) improve? The current study investigated the effects of brief training aimed at improving DDM skills in a virtual DDM task. The training addressed the DDM process, stressed the importance of self-reflection in DDM, and provided 3 self-reflective questions to guide participants during the task. Additionally, we explored whether participants low or high in self-reflection would perform better in the task and whether participants low or high in self-reflection would benefit more from the training. The study also explored possible strategic differences between participants related to training and self-reflection. Participants were 68 graduate business students. They individually managed a computer-simulated chocolate production company called CHOCO FINE and answered surveys to assess self-reflection and demographics. Training in DDM led to better performance, including the ability to solve initial problems more successfully and to make appropriate adjustments to market changes. Participants’ self-reflection scores also predicted performance in this virtual business company. High self-reflection was also related to more consistency in planning and decision making. Participants low in self-reflection benefited the most from training. Organizations could use DDM training to establish and promote a culture that values self-reflective decision making

Cognitive modelling of complex problem solving behaviour

In the universe of problems humans face every day there is subset characterized by a salient dynamic component. The FireChief task (Omodei & Wearing 1995) is a fire-fighting computer simulation that can be characterized as the acquisition of interactive skills involving fast-paced actions cued by external information. This research describes the process followed to create a cognitive model of this complex dynamic task where full experimental control is not available. The cognitive model provides a detailed description of how cognition and perception interplay to produce the interactive skill of fighting the fire. Several artefacts were produced by this effort including a dynamic task fully compatible with ACT-R, a tool for analysing the data, and a cognitive model whose features enable the replication of several aspects of the empirical data. A key finding is that good performance is linked to an effective combination of strategic control with attention to changing task demands, reflecting time and care taken in informing and effecting action. The contributions of this work towards our understanding of complex problem solving are the methodological approach to the creation of the model, the design patterns embedded in the model (which are a reflection of the cognitive demands imposed by the nature of the task) and mainly an explanation of how skill, described in terms of strategy use, is acquired in complex scenarios. This study also provides a deeper understanding of the interactions observed in the Cañas et al. (2005) dataset, including a computational realisation of how cognitive inflexibility occurs

Assessment of Problem Solving Skills by means of Multiple Complex Systems – Validity of Finite Automata and Linear Dynamic Systems

The assessment of highly domain-general problem solving skills is increasingly important as problem solving is increasingly demanded by modern workplaces (e.g., Autor, Levy, & Murnane, 2003) and increasingly present in international large-scale assessments such as the Programme for International Student Assessment (PISA, e.g., OECD, 2014). This thesis is about the computer-based assessment of problem solving skills based on Multiple Complex Systems (MCS, Greiff, Fischer, Stadler, & Wüstenberg, 2014): The main idea of the MCS approach is to present multiple computer-simulations of “minimally complex” problems (Greiff, 2012) in order to reliably assess certain problem solving skills. In each simulation, the problem solver has to interact with a problem in order to find out (a) how to adequately represent the problem, and (b) how to solve the problem. Up to now, two instances of the MCS approach have been proposed: (1) the MicroDYN approach (based on simulations of linear equation systems) and – more recently, in the second paper of this thesis – (2) the MicroFIN approach (based on simulations of finite state machines). In the current thesis I will elaborate on three research questions regarding the validity (cf. Bühner, 2006) of the MCS approach: (1) its content validity with regard to the concept of complex problem solving; (2) the convergent validity of different instances of the MCS approach; (3) the discriminant validity of the interactive problems of the MCS approach with regard to traditional static measures of reasoning and analytic problem solving skills. Each research question will be addressed in one corresponding paper: In a first paper (Fischer, Greiff, & Funke, 2012) complex problem solving is defined as the goal-oriented control of systems that contain multiple highly interrelated elements. After reviewing some of the major strands of research on complex problem solving (e.g., research on strategy selection, information reduction, intelligence, or on the interplay of implicit and explicit knowledge in the process of complex problem solving) a theoretical framework outlining the most important cognitive processes involved in solving complex problems is derived. The theoretical framework highlights both interactive knowledge acquisition (problem representation) and interactive knowledge application (problem solution) as the two major phases in the process of complex problem solving. Both phases are represented in all current instances of the MCS approach. In a second paper (Greiff, Fischer et al., 2013) the convergent validity of MicroDYN and MicroFIN is investigated (thereby introducing MicroFIN as an alternative to MicroDYN) in order to demonstrate that both instances address the same kind of problem solving skills. Based on a multitrait-multimethod analysis of a sample of university students (N = 339) it is demonstrated that – in addition to method-specific skills – both instances assess a common set of skills (method-general traits) related to (1) representing and (2) solving different kinds of interactive problems. In a regression of science grades on reasoning and the skills assessed by the instances of the MCS approach it is demonstrated that only the method-general representation trait and reasoning have substantial unique contributions. Thus, MicroDYN and MicroFIN seem to address a common set of skills and this set of skills is relevant for explaining school grades in science classes even beyond reasoning. In a third paper (Fischer et al., in press) the discriminant validity of the interactive MicroDYN test is investigated by relating it to reasoning and traditional static measures of Analytic Problem Solving skills (APS) as they were applied in PISA 2003 (OECD, 2004). Besides a common core of problem solving skills addressed by both kinds of tasks (e.g., analyzing complex information about the information given at a certain moment in time) Fischer et al. (in press) expected to find evidence for additional skills that were related to interactive problems only (e.g., systematically generating information and interactively testing hypotheses). Results indicate that MicroDYN shares a lot of variance with APS even after controlling for reasoning in a sample of high-school students (N = 577) and the university student sample (see above). With regard to the explanation of school grades MicroDYN had an incremental value compared to reasoning and APS in the high-school student sample but not significantly so in the university student sample (whereas APS had an incremental value in both samples). Basically these findings highlight both potential and limitations of the MicroDYN approach in its current form. Current instances of the MCS approach address a small set of problem solving skills reliably, but it takes more than these skills to competently solve complex problems. Implications for future research on the assessment of problem solving skills are discussed

Learning middle school mathematics through student designed and constructed video games

Mathematics achievement is an area in which American precollege students are faltering. Emerging research suggests that making mathematics instruction relevant and applicable in the lives of youth may impact math achievement, especially when it capitalizes on high-interest technologies such as video games. Employing a quasi-experimental and descriptive approach, this study examined the mathematics (i.e., numbers and operations, algebra, geometry, measurement, and probability) that middle school students employed during their design and construction of video games. First, it examined the mathematics content learned by 19 sixth and seventh graders during their analysis, synthesis, and programming of three video game projects over 7 months. Second, it measured the ability of the student programmers to laterally transfer mathematics content from the technology context of game production to the traditional context of paper-and-pencil tests. Third, it evaluated student attitudes toward mathematics prior to and following video game design and construction. The performance of student programmers was compared with that of a control group of nonprogrammers on measures of transfer and affect. Results indicated that middle grade students successfully identified the events defining game play (e.g., motion, collisions, and scoring) of three, simple video game models. They successfully represented video game events in both mathematical and programming forms by writing and coding (a) boundary conditions using inequalities, (b) coordinate locations and identification of coordinate convergence, (c) directional headings, (d) uniform linear motion, (e) variable changes, and (f) probability-based consequences. They were also successful in writing programming code for their own functional video games, with a high percentage of relevant mathematics content incorporated therein. However, while treatment students transferred mathematical knowledge from the technology to the traditional context, it appeared that, without explicit bridging, the transfer was no better than comparison students. Treatment students also demonstrated no significant changes in attitude associated with designing and constructing video games. This study demonstrated that video game design and construction can be a viable - although not significantly different - method, cognitively and affectively, of instructing age-appropriate, standards-based mathematics content