Evaluating an instrument to measure mental load and mental effort using Item Response Theory

Measurement of cognitive load (CL) is seen as a problematic issue since no consensus about appropriate instruments has been reached. In this study, a rating scale instrument to measure mental load (ML; 6 items) and mental effort (ME; 6 items) is evaluated using Item Response Theory. N=506 students self-reported their amount of ML and ME after working on a standardised multiple choice-test. The findings propose to separately measure ML and ME instead of CL in general. Furthermore, the 7-point rating scale had to be reduced post-hoc to a 3-point scale in order to reach consistent information. Finally, there was a significant (negative) correlation between ML and test performance, but not between ME and test performance.Peer Reviewe

Assessing Pre-service Teachers’ Views of Scientists, Their Activities, and Locations: the VoSAL Instrument

In science education, learners’ conceptions of scientists and their work are often assessed by the Draw-A-Scientist Test (DAST). Due to validity concerns, methodical literature demands the development of alternative instruments to measure learners’ conceptions validly and efficiently. This study presents an instrument with 29 rating scale items to assess pre-service teachers’ (PSTs) Views of Scientists, their Activities, and Locations (VoSAL). The items were developed based on theoretical considerations, previous findings, and repeated discussions by biology education experts. After several steps of test development, PSTs filled out the questionnaire (N = 1,098). Exploratory factor analyses and reliability measurements mostly confirm the proposed structure. Groups comparisons were performed regarding the results from pre-service biology teachers of three different study stages (nfreshmen = 114; nsecond and third years = 124; ngraduates = 107). Analyses of variance and corresponding post-hoc tests showed that undergraduates (freshmen, second and third years) differ significantly from graduates regarding the scales stereotypical appearance, inquiry location, and scientific activity, with undergraduates having more stereotypical conceptions than graduates. In sum, the VoSAL can be utilized to gain valid data of PSTs’ conceptions about scientists and their work. Also, the VoSAL can be considered efficient since the test time is between 5 and 10 min. Thus, the questionnaire is valuable in studies that aim to introduce and expose PSTs to realistic science images

Preservice Science Teachers’ Strategies in Scientific Reasoning: the Case of Modeling

The development of scientific reasoning competencies is a key goal of science education. To better understand the complex construct of scientific reasoning, which includes modeling as one style of reasoning, thorough investigations of the underlying processes are needed. Therefore, in this study, a typology of preservice science teachers’ modeling strategies was developed. Thirty-two preservice science teachers were videotaped while engaging in the modeling task of investigating a black box. Following a qualitative content analysis, sequences of modeling activities were identified. By transforming these sequences of modeling activities into state transition graphs, six types of modeling strategies were derived, differing in the homogeneity and complexity of their modeling processes. The preservice science teachers engaged in activities of (1) exploration only; (2a) exploration and development with a focus on development; (2b) exploration and development with a focus on exploration; (2c) exploration and development, balanced; (3a) exploration, development, and drawing predictions from a model once; or (3b) exploration, development, and repeatedly drawing predictions from a model. Finally, this typology is discussed regarding the process of its development and its potential to inform and guide further research as well as the development of interventions aiming to foster competencies in scientific modeling

Scientific Reasoning in Science Education: From Global Measures to Fine-Grained Descriptions of Students’ Competencies

[No abstract available

Investigating the dimensions of modeling competence among preservice science teachers: Meta-modeling knowledge, modeling practice, and modeling product

Worldwide, teachers are expected to engage their students in authentic practices, like scientific modeling. Research suggests that teachers experience challenges when integrating modeling in their classroom instruction, with one explanation that teachers themselves lack the necessary modeling competence. Currently, theoretical conceptualizations structure the modeling competence into three dimensions: meta-modeling knowledge, modeling practice, and modeling products. While each of these dimensions is well researched on its own and the three dimensions are commonly expected to be highly positively related, studies investigating their specific relationships are widely lacking. Aiming to fill this gap, the present study investigated the meta-modeling knowledge, modeling practice, and modeling products of 35 secondary preservice biology teachers engaging in a black box modeling task. Data were collected with an established pen-and-paper questionnaire consisting of five constructed response items assessing meta-modeling knowledge and by videotaping the participants engaging in the black box modeling task. Herein, the three dimensions of modeling competence were operationalized as five variables including decontextualized and contextualized meta-modeling knowledge, complexity, and homogeneity of the modeling processes and a modeling product score. In contrast to our expectations and common assumptions in the literature, significant relationships between the five variables were widely lacking. Only the complexity of the modeling processes correlated significantly with the quality of the modeling products. To investigate this relationship further, a qualitative in-depth analysis of two cases is presented. Implications for biology teacher education are discussed

Maschinelles Lernen mit Aussagen zur Modellkompetenz

Verfahren des maschinellen Lernens können dazu beitragen, Aussagen in Aufgaben im offenen Format in großen Stichproben zu analysieren. Am Beispiel von Aussagen von Biologielehrkräften, Biologie-Lehramtsstudierenden und Fachdidaktiker*innen zu den fünf Teilkompetenzen von Modellkompetenz (NTraining = 456; NKlassifikation = 260) wird die Qualität maschinellen Lernens mit vier Algorithmen (naïve Bayes, logistic regression, support vector machines und decision trees) untersucht. Evidenz für die Validität der Interpretation der Kodierungen einzelner Algorithmen liegt mit zufriedenstellender bis guter Übereinstimmung zwischen menschlicher und computerbasierter Kodierung beim Training (345–607 Aussagen je nach Teilkompetenz) vor, bei der Klassifikation (157–260 Aussagen je nach Teilkompetenz) reduziert sich dies auf eine moderate Übereinstimmung. Positive Korrelationen zwischen dem kodierten Niveau und dem externen Kriterium Antwortlänge weisen darauf hin, dass die Kodierung mit naïve Bayes keine gültigen Ergebnisse liefert. Bedeutsame Attribute, die die Algorithmen bei der Klassifikation nutzen, entsprechen relevanten Begriffen der Niveaufestlegungen im zugrunde liegenden Kodierleitfaden. Abschließend wird diskutiert, inwieweit maschinelles Lernen mit den eingesetzten Algorithmen bei Aussagen zur Modellkompetenz die Qualität einer menschlichen Kodierung erreicht und damit für Zweitkodierungen oder in Vermittlungssituationen genutzt werden könnte

Qualitative Content Analysis in Science Education Research Under the Consideration of Quality Criteria: a Review

Im Rahmen ihrer Entwicklung und Etablierung als empirisch arbeitende Disziplinen findet in den Naturwissenschaftsdidaktiken vermehrt eine Auseinandersetzung mit methodischen und methodologischen Fragen statt. Hierzu gehört unter anderem die Frage danach, was gute fachdidaktische Forschung überhaupt ausmacht. Obwohl die qualitative Inhaltsanalyse in der naturwissenschaftsdidaktischen Forschung als ein etabliertes Verfahren gilt, unterscheiden sich Vorschläge hinsichtlich anzulegender Gütekriterien und umzusetzender Maßnahmen der Qualitätssicherung. Im Sinne einer exemplarischen Sammlung erprobter Forschungspraxis wird in diesem Beitrag ein Überblick über die in der deutschsprachigen naturwissenschaftsdidaktischen Forschung etablierten Gütekriterien und Maßnahmen zur Qualitätssicherung bei der Umsetzung von Verfahren der qualitativen Inhaltsanalyse gegeben. Hierzu werden 50 in der Zeitschrift für Didaktik der Naturwissenschaften publizierte Artikel, die eine qualitative Inhaltsanalyse umsetzen, mit einem deduktiv-induktiv erstellten Kategoriensystem untersucht. Es werden 11 Gütekriterien identifiziert, wobei vorwiegend die klassischen Gütekriterien Validität, Reliabilität und Objektivität in den untersuchten Artikeln beschrieben werden. Von 16 identifizierten Maßnahmen der Qualitätssicherung werden am häufigsten die Modi der Kategorienbildung und die Interrater-Übereinstimmung beschrieben. Es ergibt sich ein signifikant positiver Zusammenhang zwischen dem Publikationsjahr und der Zahl der durchschnittlich beschriebenen Maßnahmen zur Qualitätssicherung. Allerdings werden die einzelnen Maßnahmen der Qualitätssicherung, über alle Artikel hinweg betrachtet, nicht konsistent jeweils einem Gütekriterium zugeordnet. Die Ergebnisse sollen die Diskussion in den Fachdidaktiken anregen und zur weiteren Etablierung und Systematisierung des Verfahrens der qualitativen Inhaltsanalyse in der naturwissenschaftsdidaktischen Forschung beitragen.In the course of developing and establishing the field of science education, methodical and methodological questions have become increasingly important. This includes questioning what poses as good qualitative research. Although, qualitative content analysis is considered an established method in science education research, there are different ideas regarding which quality criteria and measures of quality control have to be taken into account. To give an exemplary collection of reliable research practice, the present article reviews the quality criteria and measures of quality control used in science education publications utilizing a qualitative content analysis. 50 publications of the German journal Zeitschrift für Didaktik der Naturwissenschaften were investigated with a coding scheme, that was deductively-inductively developed. 11 quality criteria were identified, with validity, reliability and objectivity being the most common. 16 measures of quality control were identified, whereby descriptions of the development process of the categories and interrater-agreement were most commonly described. Also, there is a significant positive correlation between the year of publication and the number of measures of quality control described. However, these measures of quality control are inconsistently assigned to quality criteria, when all publications are taken into account. These results may spark further discussion in the science education community and support utilizing the method of qualitative content analysis more systematically

What Is Difficult in Modeling? the Identification and Description of Challenges Pre-service Science Teachers Encounter in Modeling Processes

Die Entwicklung von Modellkompetenz im naturwissenschaftlichen Unterricht kann potenziell zur Erreichung vielfältiger Ziele naturwissenschaftlicher Bildung beitragen. Studien deuten allerdings darauf hin, dass Modellierungsprozesse, in denen Modelle kritisch reflektiert oder als Werkzeuge zur Erkenntnisgewinnung eingesetzt werden, im Unterricht eher selten umgesetzt werden und dass Lehrkräfte in Bezug auf Fähigkeiten des naturwissenschaftlichen Modellierens weitere Förderung benötigen. Das Ziel der vorliegenden Studie ist die Identifikation und Beschreibung von Hindernissen, die in Modellierungsprozessen von Lehramtsstudierenden naturwissenschaftlicher Fächer auftreten. Hierzu wurden die Modellierungsprozesse von 36 Lehramtsstudierenden naturwissenschaftlicher Fächer bei der Untersuchung einer Blackbox qualitativ-inhaltsanalytisch ausgewertet. Es konnten 13 verschiedene Hinderniskategorien identifiziert und beschrieben werden. Die identifizierten Hinderniskategorien weisen teils Parallelen zum Experimentieren und naturwissenschaftlichen Arbeiten allgemein auf: Spezifische Hinderniskategorien für das Modellieren ergeben sich dagegen beim Umgang mit Analogien und Erfahrungen und treten vor allem beim Testen des entwickelten Modells auf. Basierend auf vorherigen Arbeiten wurden zudem die Modellierungsprozesse der Lehramtsstudierenden analysiert und sechs typischen Modellierungsstrategien zugeordnet. Es scheint kein eindeutiger Zusammenhang zwischen den identifizierten Hindernissen und den Modellierungsstrategien vorzuliegen, da ProbandInnen, die ähnlichen Hindernissen begegnen, sich hinsichtlich ihrer Modellierungsstrategien teils deutlich unterscheiden. Es wird diskutiert, inwiefern die identifizierten Hinderniskategorien für die weitere Entwicklung von Diagnoseinstrumenten und zur gezielten Planung von Förderangeboten genutzt werden können.The development of modeling competence can potentially support the achievement of diverse goals in science education. However, studies suggest that modeling processes in which models are critically reflected or used as epistemic tools are rarely implemented in educational settings. Other studies propose a need to promote science teachers’ abilities regarding scientific modeling. The present study aims to identify and describe challenges, which pre-service science teachers encounter while modeling. Therefore, the modeling practice of 36 pre-service science teachers engaging in a blackbox activity was analyzed using a qualitative content analytic approach. 13 different challenges were identified and described. Some of the identified challenges show parallels to previously identified challenges related to experimentation and scientific practices in general. In contrast, challenges, which seem to be specific for modeling are related to the use of analogies and occur mostly when the developed models are tested. Based on previous results, the pre-service science teachers’ modeling practices were assigned to one of six types of modeling strategies. The identified challenges and the type of modeling strategy used by the pre-service science teachers seem not to be related, as pre-service science teachers encountering similar challenges can utilize very different modeling strategies. It is discussed, how the identified challenges can be used to support the development of diagnostic instruments as well as targeted interventions of modeling competence

Context Matters: Accounting for Item Features in the Assessment of Citizen Scientists’ Scientific Reasoning Skills

Citizen science (CS) projects engage citizens for research purposes and promote individual learning outcomes such as scientific reasoning (SR) skills. SR refers to participants’ skills to solve problems scientifically. However, the evaluation of CS projects’ effects on learning outcomes has suffered from a lack of assessment instruments and resources. Assessments of SR have most often been validated in the context of formal education. They do not contextualize items to be authentic or to represent a wide variety of disciplines and contexts in CS research. Here, we describe the development of an assessment instrument that can be flexibly adapted to different CS research contexts. Furthermore, we show that this assessment instrument, the SR questionnaire, provides valid conclusions about participants’ SR skills. We found that the deep-structure and surface features of the items in the SR questionnaire represent the thinking processes associated with SR to a substantial extent. We suggest that practitioners and researchers consider these item features in future adaptations of the SR questionnaire. This will most likely enable them to draw valid conclusions about participants’ SR skills and to gain a deeper understanding of participants’ SR skills in CS project evaluation

Findings from the expert-novice paradigm on differential response behavior among multiple-choice items of a pedagogical content knowledge test – implications for test development

Pedagogical content knowledge (PCK) is one core dimension of teachers’ professional knowledge and comprises knowledge about conceptual ideas of learners and appropriate instructions. However, several challenges regarding the assessment of PCK are discussed in the literature: For example, PCK is a topic-specific construct and contains differentiable subdomains, which must be considered during test development. In addition, the choice of test type needs to be considered. While open-ended instruments can capture a broader range of cognitions, they often require a high level of interpretation; in contrast, multiple-choice instruments have advantages in terms of objectivity and test economy. Some challenges of assessing PCK are particularly related to multiple-choice instruments, such as an insufficient focus on specific components or the accidental assessment of teachers’ beliefs instead of PCK. To better understand and explain these challenges in developing multiple-choice PCK instruments, we exemparly used an instrument to assess PCK about scientific reasoning and considered the assumptions of the expert-novice paradigm to analyze differential response behavior between n = 10 researchers in the field of biology education (experts) and n = 10 undergraduate pre-service biology teachers (novices). As expected, experts scored significantly higher than novices. At the same time, experts answered the items more consistently than novices, i.e., showed less variance. However, the difference found was statistically insignificant. Regarding the explanations for choosing a response option, experts more often correctly identified the quintessence of the items, which means that they more often understand the items as intended and argued based on their PCK. On the other hand, novices focused more on surface characteristics, i.e., they argued rather with surface knowledge like intuition or personal experience, than choosing the response option based on their PCK. These crucial differences in how experts and novices understand the items of the used PCK instrument and how they respond based on their understanding affect different test characteristics. In conclusion, we recommend ensuring that instruments address only a few, specific PCK aspects, considering the target group of a test, and take into account that target groups with larger variability among their responses require a higher number of items to achieve satisfactory discrimination and reliability indices