Internal Consistency Reliability in Measurement: Aggregate and Multilevel Approaches

The purpose of the present paper was to evaluate the internal consistency reliability of the General Teacher Test assuming clustered and non-clustered data using commercial software (Mplus). Participants were 2,000 testees who were selected using random sampling from a larger pool of examinees (more than 65k). The measure involved four factors, namely: (a) planning for learning, (b) promoting learning, (c) supporting learning, and (d) professional responsibilities and was hypothesized to comprise a unidimensional instrument assessing generalized skills and competencies. Intra-class correlation coefficients and variance ratio statistics suggested the need to incorporate a clustering variable (i.e., university) when evaluating the factor structure of the measure. Results indicated that single level reliability estimation significantly overestimated the reliability observed across persons and underestimated the reliability at the clustering variable (university). One level reliability was also, at times, lower than the lowest acceptable levels leading to a conclusion of unreliability whereas multilevel reliability was low at the between person level but excellent at the between university level. It is concluded that ignoring nesting is associated with distorted and erroneous estimates of internal consistency reliability of an ability measure and the use of MCFA is imperative to account for dependencies between levels of analyses

Evaluation of Saudi International Educational Programs using Rossi and His Colleagues Evaluation Model

This study aimed to evaluate international educational programs (IEP) of the Saudi private schools, their need, design, inputs, processes, outputs and impact using the hierarchical evaluation model (HEM) of Rossi and his colleagues. The methodology adopts dual quantitative and qualitative methods. The evaluation includes Saudi private schools applying such programs, and samples of some foreign schools and foreign community schools employing deep comparisons, questionnaires, field observations and interviews of administrators, teachers, students and parents. This evaluation shows that HEM helped in grading evaluation and determining questions that suit program grade. Results show that IEP maintain students’ identity, and achieve outcomes reasonably. They reduced student enrollment in foreign programs, and helped raise their knowledge, language, personal and educational skills to a global level. However, there is lack of clear program theory that identifies inputs, processes and outcomes, lack of clarity in program mechanisms, design, needs assessment, performance monitoring and evaluation. They show limited external evaluation, with no standardized or uniform standards for teachers selection and intake as well as final achievement exams. They likewise show lack of information for parents and community members. This study recommends the expansion of experience provided that international standards of programs quality and monitoring be ensured

Examining Differences in Within- and Between-Person Simple Structures of an Engineering Qualification Test Using Multilevel MIMIC Structural Equation Modeling

The current study sought to meet three aims: (a) to understand the optimal factor structure of the Professional Engineering (ProfEng) test, a measure aiming to assess competency in engineering, within a multilevel (nested) perspective; (b) to examine the psychometric measurement invariance of the ProfEng test across levels due to nesting and across gender at the person level, and, (c) to examine the internal consistency of the engineering competency measure at both levels in the analysis. Data involved 1,696 individuals across 21 universities who took a national licensure test as part of the professional accreditation process to obtain a work permit and practice the engineering profession in the Kingdom of Saudi Arabia. Data were analyzed by use of Multilevel Structural Equation Modeling (MLSEM). Results indicated that a 2-factor model at both levels of analysis provided the best fit to the data. We also examined violation of measurement invariance across clusters (cluster bias). Results showed that all factor loadings were invariant across levels, suggesting the presence of strong measurement invariance. Last, invariance across gender was tested by use of the MIMIC multilevel model. Results pointed to the existence of significant differences between genders on levels of personal and professional skills with females having higher levels on personal skills and males on professional. Estimates of internal consistency reliability also varied markedly due to nesting. It is concluded that ignoring a multilevel structure is associated with errors and inaccuracies in the measurement of person abilities as both measurement wise and precision wise the multilevel model provides increased accuracy at each level in the analysis

The Utility of the Bifactor Method for Unidimensionality Assessment When Other Methods Disagree

This article provides an empirical illustration of the utility of the bifactor method for unidimensionality assessment when other methods disagree. Specifically, we used two popular methods for unidimensionality assessment: (a) evaluating the model fit of a one-factor model using Mplus, and (b) DIMTEST to show that different unidimensionality methods may lead to different results, and argued that in such cases the bifactor method can be particularly useful. Those procedures were applied to English Placement Test (EPT), a high-stakes English proficiency test in Saudi Arabia, to determine whether EPT is unidimensional so that a unidimensional item response theory (IRT) model can be used for calibration and scoring. We concluded that despite the inconsistency between the one-factor model approach and DIMTEST, the bifactor method indicates that, for practical purposes, unidimensionality assumption holds for EPT

The Impact of Non-Attempted and Dually-Attempted Items on Person Abilities and Scale Validity using Item Response Theory

The purpose of the present study was to relate response strategy with person ability estimates. Two behavioral strategies were examined: (a) the strategy to skip items in order to save time on timed tests, and, (b) the strategy to select two responses on an item, with the hope that one of them may be considered correct. Participants were 4,422 individuals who were administered a standardized achievement measure related to math, biology, chemistry, and physics. In the present evaluation, only the physics subscale was employed. Two analyses were conducted: (a) a person-based one to identify differences between groups and potential correlates of those differences, and, (b) a measure-based analysis in order to identify the parts of the measure that were responsible for potential group differentiation. For (a) person abilities the 2-PL model was employed and later the 3-PL and 4-PL models in order to estimate upper and lower asymptotes of person abilities. For (b) differential item functioning, differential test functioning, and differential distractor functioning were investigated. Results indicated that there were significant differences between groups with completers having the highest ability compared to both non-attempters and dual responders. There were no significant differences between no-attempters and dual responders. The present findings have implications for response strategy efficacy and measure evaluation, revision, and construction

Evaluación de las habilidades lingüísticas mediante modelos de clasificación diagnóstica: un ejemplo de uso de un instrumento lingüístico

The primary purpose of the present study was to inform and illustrate, using examples, the use of Diagnostic Classification Models (DCMs) for the assessment of skills and competencies in cognition and academic achievement. A secondary purpose was to compare and contrast traditional and contemporary psychometrics for the measurement of skills and competencies. DCMs are described along the lines of other psychometric models within the Confirmatory Factor Analysis tradition, such as the bifactor model and the known mixture models that are utilized to classify individuals into subgroups. The inclusion of interaction terms and constraints along with its confirmatory nature enables DCMs to accurately assess the possession of skills and competencies. The above is illustrated using an empirical dataset from Saudi Arabia (n = 2,642), in which language skills are evaluated on how they conform to known levels of competency, based on the CEFR (Council of Europe, 2001) using the English Proficiency Test (EPT).El propósito principal del presente estudio fue informar e ilustrar, mediante ejemplos, el uso de Modelos de Clasificación Diagnóstica (DCM) para la evaluación de habilidades y competencias en cognición y rendimiento académico. Un propósito secundario fue comparar y contrastar la psicometría tradicional y contemporánea para la medición de habilidades y competencias. Los DCM se describen siguiendo las líneas de otros modelos psicométricos dentro de la tradición del Análisis Factorial Confirmatorio, como el modelo bifactor y los conocidos modelos mixtos que se utilizan para clasificar a los individuos en subgrupos. La inclusión de términos y restricciones de interacción junto con su naturaleza confirmatoria permite a los DCM evaluar con precisión la posesión de habilidades y competencias. Lo anterior se ilustra utilizando un conjunto de datos empíricos de Arabia Saudita (n = 2,642), que evalúan cómo las habilidades lingüísticas se ajustan a los niveles de competencia conocidos, basados en el MCER (Council of Europe, 2001)

The Impact of Non-attempted and Dually-Attempted Items on Person Abilities Using Item Response Theory

The purpose of the present study was to relate response strategy with person ability estimates. Two behavioral strategies were examined: (a) the strategy to skip items in order to save time on timed tests, and, (b) the strategy to select two responses on an item, with the hope that one of them may be considered correct. Participants were 4,422 individuals who were administered a standardized achievement measure related to math, biology, chemistry, and physics. In the present evaluation, only the physics subscale was employed. Two analyses were conducted: (a) a person-based one to identify differences between groups and potential correlates of those differences, and, (b) a measure-based analysis in order to identify the parts of the measure that were responsible for potential group differentiation. For (a) person abilities the 2-PL model was employed and later the 3-PL and 4-PL models in order to estimate upper and lower asymptotes of person abilities. For (b) differential item functioning, differential test functioning, and differential distractor functioning were investigated. Results indicated that there were significant differences between groups with completers having the highest ability compared to both non-attempters and dual responders. There were no significant differences between no-attempters and dual responders. The present findings have implications for response strategy efficacy and measure evaluation, revision, and construction

Improving Utilization of the Chemotherapy Unit through Implementing the Medication Early Release Project

Introduction The outpatient oncology infusion unit is very busy, serving 60 to 70 patients per day. Due to a limited number of nurses, treatment chairs, only one pharmacy hood for bio-hazardous drug preparation, and other factors, patients wait a long time before starting their treatment, which affects the patient experience negatively. We conducted a quality improvement project to reduce the waiting time before starting the treatment, improve the patients' experience, and allow the unit to work more effectively through better resource utilization and accommodating more patients. Methods A committee was formed with representatives from oncology nursing and the quality specialist, chemotherapy pharmacy supervisor, data manager, and a medical consultant (team leader). We studied baseline data of patient waiting times from January to March 2019 and the factors that contributed to delays before starting the treatment. The charge nurse identified patients who could safely have their medication released early in the morning at 7 am, enabling the pharmacy to dispense at 8 am without their actual presence being required in the infusion suite (i.e., medication early release program or MERP). Multiple plan-do-study-act (PDSA) cycles were implemented to achieve a wait time from check-in to medication administration of less than 60 minutes. Data collected included check-in time, chair time, vital signs time, administration time, and discharge time. Additionally, reasons for drug wastage were assessed for patients who did not receive the prepared medication. A patient satisfaction survey was conducted with the patients before and after being enrolled in the program. Results At baseline, average waiting time for patients receiving similar medications in the MERP was 2 hours and 27 minutes. After the first intervention, average waiting time was reduced to 1 hour and 24 minutes, and small improvements were observed after each PDSA cycl. A major breakthrough occurred after an intensive patient education program and enforcement of strict compliance with the criteria in selecting the patients appropriate for theMERP. Average waiting time wasreduced to ≤ 60 minutes, and in November 2022, it was 30 minutes on average. Drug wastage was identified as a balancing measure. We were successful in reducing drug wastage by implementing several changes and patient education measures and achieved zero wastage. The patient satisfaction survey showed better satisfaction with the new changes. Conclusion A positive impact was achieved in this quality improvement project, with a significant reduction in the average waiting time for patients to start receiving chemotherapy. The outcome of this project has been maintained for 4 years and is still ongoing