Construction of school characteristics using multilevel confirmatory factor analysis

This project aims to compare the use of multilevel confirmatory factor analysis with more traditional methods of factor construction at a higher group level in the context of educational effectiveness research. School process characteristics were constructed (1) by means of two-level confirmatory factor analyses, and (2) using exploratory factor analyses at individual level. To this end data provided by 684 teachers from 54 schools on a school characteristics questionnaire were analyzed. The results indicate a less complicated factor structure at the school level as compared to the teacher level within the multilevel confirmatory factor models. Some differences in factor structure between the exploratory factor analysis and the multilevel confirmatory analyses at school level are observed, while no substantial differences in factor meaning were found.status: publishe

Multilevel confirmatory factor analysis: Evaluating its surplus value when constructing school process variables in secondary education

Until now, the multilevel confirmatory factor analysis has rarely been applied in educational effectiveness research to construct measures for school process variables. Therefore, this study investigates the surplus value of a multilevel confimatory factor analysis, compared to the use of a standard confirmatory factor analysis plus aggregation. Both methods are used to develop school process variables, based on data from 1550 teachers within 82 schools or 113 cycle groups. Differences between teachers from the first cycle and teachers from the second and the third cycle are taken into account by means of a multiple group analysis if necessary. We evaluate the interpretation of the measures as well as their internal consistency. Interestingly, both methods yield equivalent results at the teacher level, thus irrespective of taking the nesting within cycles into account. Moreover, with regard to the social and orderly school climate, identical measures are also constructed at the cycle group level. On the contrary, regarding organizational aspects like the cooperation between teachers and the decision making process at the school, more global measures are detected at the cycle group level in comparison with the teacher level. Noteworthy, concerning all constructed measures, we observe considerably higher internal consistencies at the cycle group level. Therefore, our findings support the use of a multilevel factor analysis when confronted with nested data.status: publishe

Multilevel confirmatory factor analysis: Evaluating its surplus value when constructing school process variables in secondary education

Until now, the multilevel confirmatory factor analysis has rarely been applied in educational effectiveness research to construct measures for school process variables. Therefore, this study investigates the surplus value of a multilevel confimatory factor analysis, compared to the use of a standard confirmatory factor analysis plus aggregation. Both methods are used to develop school process variables, based on data from 1550 teachers within 82 schools or 113 cycle groups. Differences between teachers from the first cycle and teachers from the second and the third cycle are taken into account by means of a multiple group analysis if necessary. We evaluate the interpretation of the measures as well as their internal consistency. Interestingly, both methods yield equivalent results at the teacher level, thus irrespective of taking the nesting within cycles into account. Moreover, with regard to the social and orderly school climate, identical measures are also constructed at the cycle group level. On the contrary, regarding organizational aspects like the cooperation between teachers and the decision making process at the school, more global measures are detected at the cycle group level in comparison with the teacher level. Noteworthy, concerning all constructed measures, we observe considerably higher internal consistencies at the cycle group level. Therefore, our findings support the use of a multilevel factor analysis when confronted with nested data.status: publishe

Assessing the "explanatory power" of school process variables constructed through multilevel exploratory factor analysis

This paper investigates the “explanatory power” of school process variables based on teacher data constructed through multilevel factor analysis, compared to school process variables developed using conventional factor analysis plus aggregation. To date, multilevel factor analysis has only scarcely been used in educational effectiveness research. The school process variables were associated with student achievement and non-cognitive outcomes. Our results indicated that some of the school process variables created through multilevel factor analysis were associated to a different extent to achievement than the variables obtained using conventional factor analysis plus aggregation. Therefore, these results underline the importance of using the correct constructional technique to develop school process variables.status: publishe

The construction of class and school process variables based on individual questionnaires

Summary In educational effectiveness research, class and school process variables are investigated to determine their association with diverse student outcomes. For example, the school environment and instructional characteristics are studied to establish the association with student achievement. Revealing the class and school process variables that matter for the learning of students could inspire policy makers and school principals to make schooling and teaching more effective. For both practical and financial reasons, student and teacher questionnaires are often the preferred means to capture these process variables. Next, the individual data from these questionnaires are used to construct measures for higher-order process variables at the class level and the school level. Traditionally, this is done through a factor analysis at the individual level, which can be exploratory or confirmatory. This way, scale scores or factor scores are developed, which are subsequently aggregated at the class level or the school level. However, this traditional approach does not take into account the dependency present in the hierarchical structure of educational data. Among others, Muthén argues to explicitly model this hierarchy using multilevel confirmatory factor analysis. This way, a confirmatory factor analysis is performed simultaneously at the individual level and at the group level. Recently, van de Vijver and Poortinga presented an exploratory variant of this multilevel factor analysis, based on the procedure proposed by Muthén. In the current research project, we propose to investigate the surplus value of the multilevel factor analysis methods, as compared to the use of a factor analysis at one level, to construct meaningful, reliable and valid measures for class and school process characteristics. Will the alternative approach of multilevel factor analysis lead to new understandings within the field of educational effectiveness? To this end, the traditional approach of factor analysis at one level will be compared to the alternative procedure of multilevel factor analysis with regard to a number of features: factor structure, meaning and interpretation of the constructed measures for process variables, reliability and criterion validity. Several data bases will be used to realize these research goals. Two of them are associated with longitudinal research projects in Flanders, one in primary education (i.e., SiBO: Schoolloopbanen in het BasisOnderwijs [School Careers in Primary Education]) and one in secondary education (i.e., LOSO: Longitudinaal Onderzoek in het Secundair Onderwijs [Longitudinal Research in Secondary Education]). In addition, data from a number of pilot studies within the framework of an international research project (i.e., ISTOF: International System for Teacher Observation and Feedback) will be included in the project. The current research project is relevant to the domain of educational effectiveness in particular as well as to environmental research in general.status: publishe

Multilevel exploratory factor analysis: Evaluating its surplus value in educational effectiveness research

This paper investigates the surplus value of a multilevel exploratory factor analysis, compared to the use of a standard exploratory factor analysis plus aggregation. Until now, this alternative multilevel technique has never been used in educational effectiveness research. Both methods were used to construct school process variables, based on data from 1550 teachers within 113 cycle groups. Our results show broader and, to some extent, also distinct factors at the group level in comparison with the teacher level, irrespective of taking the dependency into account. Furthermore, while the total amount of variance explained is not larger at the group level, we observe higher internal consistencies of the group-level factors. Thus, these findings clearly support the use of a multilevel analysis.status: publishe

Assessing the “explanatory power” of school process variables constructed through multilevel exploratory factor analysis

This paper investigates the “explanatory power” of school process variables based on teacher data constructed through multilevel exploratory factor analysis, compared to school process variables developed using conventional exploratory factor analysis plus aggregation. Measures for school process variables are associated with student achievement and non-cognitive outcomes. The results are mixed: While for some measures we observe different results depending on the constructional approach, for other measures we find exactly the same results irrespective of the constructional approach. Nevertheless, these results underline the importance of using the correct approach to develop measures for school process variables. Since multilevel exploratory factor analysis has only scarcely been used in educational effectiveness research, this study has major relevance for educational effectiveness researchers.status: publishe