Detecting Violations of Unidimensionality by Order-Restricted Inference Methods

The assumption of unidimensionality and quantitative measurement represents one of the key concepts underlying most of the commonly applied of item response models. The assumption of unidimensionality is frequently tested although most commonly applied methods have been shown having low power against violations of unidimensionality whereas the assumption of quantitative measurement remains in most of the cases only an (implicit) assumption. On the basis of a simulation study it is shown that order restricted inference methods within a Markov Chain Monte Carlo framework can successfully be used to test both assumptions

Using Differential Item Functioning to Analyze the Domain Generality of a Common Scientific Reasoning Test

A significant problem that assessments of scientific reasoning face at the level of higher education is the question of domain generality, that is, whether a test will produce biased results for students from different domains. This study applied three recently developed methods of analyzing differential item functioning (DIF) to evaluate the domain generality assumption of a common scientific reasoning test. Additionally, we evaluated the usefulness of these new, tree- and lasso-based, methods to analyze DIF and compared them with methods based on classical test theory. We gave the scientific reasoning test to 507 university students majoring in physics, biology, or medicine. All three DIF analysis methods indicated a domain bias present in about one-third of the items, mostly benefiting biology students. We did not find this bias by using methods based on classical test theory. Those methods indicated instead that all items were easier for physics students compared to biology students. Thus, the tree- and lasso-based methods provide a clear added value to test evaluation. Taken together, our analyses indicate that the scientific reasoning test is neither entirely domain-general, nor entirely domain-specific. We advise against using it in high-stakes situations involving domain comparisons

What can the Real World do for simulation studies? A comparison of exploratory methods

For simulation studies on the exploratory factor analysis (EFA), usually rather simple population models are used without model errors. In the present study, real data characteristics are used for Monte Carlo simulation studies. Real large data sets are examined and the results of EFA on them are taken as the population models. First we apply a resampling technique on these data sets with sub samples of different sizes. Then, a Monte Carlo study is conducted based on the parameters of the population model and with some variations of them. Two data sets are analyzed as an illustration. Results suggest that outcomes of simulation studies are always highly influenced by particular specification of the model and its violations. Once small residual correlations appeared in the data for example, the ranking of our methods changed completely. The analysis of real data set characteristics is therefore important to understand the performance of different methods

Evaluation of a new k-means approach for exploratory clustering of items

Evaluation of a new k-means approach for exploratory clustering of item

Das Führungskompetenzmodell des "Adaptive Leadership Competency Profile (ALCP)": Eine Gültigkeitsprüfung mithilfe der Probabilistischen Testtheorie

Ziel des Artikels ist die Analyse der psychometrischen Güte der deutschen Version des "Adaptive Leadership Competency Profile" (ALCP) (Sherron, 2000), eines Fragebogens zur Erfassung elf globaler Führungskompetenzen. Vorgestellt wird ein probabilistisch-testtheoretisches Vorgehen zur Überprüfung von mit (Führungs-)Kompetenzmodellen verbundenen Messmodellen auf der Grundlage des Rasch-Modells. Mit Hilfe des ordinalen Rasch-Modells wurden Items und Skalen analysiert und Rasch-Homogenität überprüft (N = 311). Mit Hilfe des Mixed-Rasch-Modells wurde die Annahme der gemeinsamen Skalierbarkeit von unterschiedlichen Mitarbeiterinnen und Mitarbeitern analysiert. Ein 11-dimensionales Rasch-Modell explorierte die latenten Korrelationen der Personenparameter. Sämtliche Skalen trennten Führungskräfte und Nicht-Führungskräfte nach Kompetenzgraden. Stärken des Ansatzes und praktische Implikationen werden diskutiert.We analyze the psychometric properties of the German version of the 'Adaptive Leadership Competency Profile' (ALCP) (Sherron, 2000), which contains eleven scales assessing broad leadership competencies. We illustrate an approach to the probabilistic test-theoretical analysis of competency models according to the Rasch model. Based on the ordinal Rasch model, item and scale analyses demonstrated Rasch homogeneity (N = 311). A mixed Rasch model was used to inspect whether the different types of staff can be scaled jointly. An 11-dimensional Rasch model was used to explore the latent correlations among the person parameters. All scale means discriminated between executive managers and staff members. We discuss strengths of the approach and practical implications

Das Führungskompetenzmodell des "Adaptive Leadership Competency Profile (ALCP)": Eine Gültigkeitsprüfung mithilfe der Probabilistischen Testtheorie

Ziel des Artikels ist die Analyse der psychometrischen Güte der deutschen Version des "Adaptive Leadership Competency Profile" (ALCP) (Sherron, 2000), eines Fragebogens zur Erfassung elf globaler Führungskompetenzen. Vorgestellt wird ein probabilistisch-testtheoretisches Vorgehen zur Überprüfung von mit (Führungs-)Kompetenzmodellen verbundenen Messmodellen auf der Grundlage des Rasch-Modells. Mit Hilfe des ordinalen Rasch-Modells wurden Items und Skalen analysiert und Rasch-Homogenität überprüft (N = 311). Mit Hilfe des Mixed-Rasch-Modells wurde die Annahme der gemeinsamen Skalierbarkeit von unterschiedlichen Mitarbeiterinnen und Mitarbeitern analysiert. Ein 11-dimensionales Rasch-Modell explorierte die latenten Korrelationen der Personenparameter. Sämtliche Skalen trennten Führungskräfte und Nicht-Führungskräfte nach Kompetenzgraden. Stärken des Ansatzes und praktische Implikationen werden diskutiert.We analyze the psychometric properties of the German version of the 'Adaptive Leadership Competency Profile' (ALCP) (Sherron, 2000), which contains eleven scales assessing broad leadership competencies. We illustrate an approach to the probabilistic test-theoretical analysis of competency models according to the Rasch model. Based on the ordinal Rasch model, item and scale analyses demonstrated Rasch homogeneity (N = 311). A mixed Rasch model was used to inspect whether the different types of staff can be scaled jointly. An 11-dimensional Rasch model was used to explore the latent correlations among the person parameters. All scale means discriminated between executive managers and staff members. We discuss strengths of the approach and practical implications

The Impact of Symmetry: Explaining Contradictory Results Concerning Working Memory, Reasoning, and Complex Problem Solving

Findings of studies on the unique effects of reasoning and working memory regarding complex problem solving are inconsistent. To find out if these inconsistencies are due to a lack of symmetry between the studies, we reconsidered the findings of three published studies on this issue, which resulted in conflicting conclusions regarding the inter-relations between reasoning, working memory, and complex problem solving. This was achieved by analysing so far unpublished problem solving data from the study of Bühner, Krumm, Ziegler, and Plücken (2006) (N= 124). One of the three published studies indicated unique effects of working memory and reasoning on complex problem solving using aggregated scores, a second study found no unique contribution of working memory using only figural scores, and a third study reported a unique influence only for reasoning using only numerical scores. Our data featured an evaluation of differences across content facets and levels of aggregation of the working memory scores. Path models showed that the results of the first study could not be replicated using content aggregated scores; the results of the second study could be replicated if only figural scores were used, and the results of the third study could be obtained by using only numerical scores. For verbal content, none of the published results could be replicated. This leads to the assumption that not only symmetry is an issue when correlating non-symmetrical data, but that content also has to be taken into account when comparing different studies on the same topi