Measuring performance in dynamic decision making. Reliabiblity and validity of the Tailorshop simulation

The Tailorshop simulation is a computer-based dynamic decision-making task in which participants lead a fictional company for 12 simulated months. The present study investigated whether the performance measure in the Tailorshop simulation is reliable and valid. The participants were 158 employees from different companies. Structural equation models were used to test τ-equivalent measurement models. The results indicate that the trends of the company value between the second and the twelfth month are reliable variables. Furthermore, this measure predicted real-life job performance ratings by supervisors and was associated with the performance in another dynamic decision-making task. Thus, the trend of the company value provides a reliable and valid performance indicator for the Tailorshop simulation

The challenge of complexity for cognitive systems

Complex cognition addresses research on (a) high-level cognitive processes – mainly problem solving, reasoning, and decision making – and their interaction with more basic processes such as perception, learning, motivation and emotion and (b) cognitive processes which take place in a complex, typically dynamic, environment. Our focus is on AI systems and cognitive models dealing with complexity and on psychological findings which can inspire or challenge cognitive systems research. In this overview we first motivate why we have to go beyond models for rather simple cognitive processes and reductionist experiments. Afterwards, we give a characterization of complexity from our perspective. We introduce the triad of cognitive science methods – analytical, empirical, and engineering methods – which in our opinion have all to be utilized to tackle complex cognition. Afterwards we highlight three aspects of complex cognition – complex problem solving, dynamic decision making, and learning of concepts, skills and strategies. We conclude with some reflections about and challenges for future research