Abbreviare test per la valutazione delle abilità tramite la metodologia competence-based test development

Verrà trattata la metodologia Competence-based Test Development (CbTD; Anselmi et al., 2022), un approccio innovativo per lo sviluppo di test più informativi possibile riguardo gli stati di competenza, ovvero l’insieme di abilità o tratti relativi ad un dominio di interesse posseduti da un individuo. La metodologia CbTD ha tre possibili ambiti di applicazione: la creazione di un test da zero, il miglioramento di un test esistente e l’accorciamento di un test esistente. Il lavoro si concentrerà sullo sviluppo della forma breve di un test. Viene illustrata la cornice teorica a cui si è ispirata la metodologia CbTD, ovvero la Knowledge Space Theory (KST). Successivamente, vengono introdotti gli elementi base e le proprietà caratteristiche della metodologia CbTD. Infine, la metodologia CbTD viene utilizzata per l’abbreviazione di un test già esistente, il Fraction Subtraction Test (Tatsuoka, 1990), per ottenere un nuovo test, con un numero ridotto di item, tanto informativo riguardo gli stati di competenza quanto quello di partenza. Questo risultato è teorico, poichè, per risalire agli stati di competenza, sono necessari gli stati di conoscenza (lo stato di conoscenza è il comportamento risolutivo in condizioni ideali), i quali, a livello pratico, non sono noti a priori

The Behavior-Driven Observation. Definition and development of an adaptive observational assessment

The observation in psychological assessment provides clinicians with a variety of useful insights about the symptoms of mental disorders. Nonetheless, the application of observational instruments has decreased during the last years, mainly due to their administration complexity and time consumption. A consequence of this general reduction in application is that some innovations fruitfully applied by other psychological assessment instruments, such as the self-reports, are still unexplored. For instance, little focus has been put on the possibility of implementing observational measures with adaptive algorithms. In observational assessment, these algorithms have been applied only by some software developed for observers training; their implementation in observational assessment instruments is still an open challenge. The aim of the present Ph.D. project is to develop an observational adaptive instrument able to help clinicians to generate accurate behavioral response patterns reducing, simultaneously, the time of the observational assessment. The definition of such an instrument has been a sequential process that started from a deep analysis of the items that should be observed, followed by the consideration of how to observe each of them. These first issues were accounted in Chapter 1, in which an overview of the literature was performed in order to examine all the features necessary to adequately conduct an observational assessment. A specific attention was dedicated on the possible biases that could affect raters, leading to higher probabilities of false positive and negatives on the observed behaviors. Finally, the state of the art relative to the application of adaptive algorithms in observational assessments was introduced and discussed. The second step toward the definition of the expected instrument consisted in defining a non adaptive checklist evaluating the behaviors of a mental disorder, possibly based on a formal methodology. In Chapter 2, the Formal Psychological Assessment (FPA) was introduced, describing its deterministic and probabilistic features. FPA is a methodology allowing to define assessment instruments starting from the relation between a set of items and a set of clinical issues of a disorder. In Chapter's end, it was shown how FPA could be extended also to observational assessment composed by multiple measures. In Chapter 3, the FPA was applied to develop the paper-and-pencil version of the final checklist. The negative symptomatology of schizophrenia was selected as the target mental disorder. A set of 138 items describing nonverbal behaviors was selected from instruments frequently used in the evaluation of schizophrenia. This list was then mapped to a list of 14 negative symptoms, selected in both scientific literature and DSM-5. The application of formal and logical steps provided by FPA led to a final checklist of 22 items, divided into two subscales, exhaustively investigating the 14 negative symptoms. In particular, it emerged how the mapping between items and investigated symptoms defined a deterministic model of assessment in which the clinician could be informed not only of which negative symptoms are evaluated by each item, but also of the relations among items. This model of assessment was later validated, in order to convert it into a probabilistic model that would have been correctly implemented into an adaptive instrument. In Chapter 4, the validation procedure is described. 172 videos of clinical interviews were observed by two independent raters, who filled the new checklist during one-zero sampling observations and generated modal response patterns for both subscales. Such patterns were used to apply the Basic local Independence Model (BLIM), a probabilistic model allowing to estimate the global fit indexes of the checklist and the false positive and negative rates for each item. Results showed adequate fit indexes for both subscales of the checklist with acceptable error rates for each item, which were extremely low especially in respect to false positive rates. The obtained probabilistic model of assessment and its parameters estimates were then used to calibrate an observational adaptive algorithm. In Chapter 5, the first version of the Behavior-Driven Observation (BDO) was introduced, namely the adaptive observational checklist proposed by the present project. After its formulation, the BDO was tested on real data by a simulation study in which both its accuracy and efficiency were examined. Results showed how the BDO algorithm was able to accurately reproduce almost all the non adaptive response patterns, with an average reduction by 38% of suggested items to complete the entire assessment. Finally, the accuracy and the efficiency of the BDO were tested during real observations, in order to understand if the BDO led to accurately replicate the non adaptive response patterns when used by human raters, with similar savings in terms of efficiency. Two independent trained raters observed twice the videos of twenty patients with a diagnosis of schizophrenia with negative symptoms, filling the two checklist's versions during observations. The observations on the same patient were far one week from each other. A very good intra-rater agreement emerged for each rater, suggesting both a good coherence over time of raters and a good ability of the BDO to replicate the response patterns of its non adaptive counterpart. Likewise, encouraging results were found in regard to BDO's efficiency: The savings in terms of suggested items were the same of the simulation study, for each rater; moreover, such savings corresponded to a reduction of the observational time. Taken together, the results of this Ph.D. project suggest that is possible to define an adaptive observational checklist able to help clinician to collect information not otherwise detectable with other assessment modalities. The BDO, in fact, could guide the observation by suggesting which behavior should be observed, taking into account the false positive/negative rates for each behavior. In this way, the accuracy of the final clinical output is increased as well as the efficiency of its generation. Such a clinical output could provide clinicians with a comprehensive set of information, such as the precise response pattern observed during the observation, the most plausible symptoms related to that response pattern and their probability values. All this information, in turn, can be finally integrated with other ones collected from different assessment instruments (e.g., interview, self-report), in order to have a broader frame of patient's condition and, maybe, set an individualized treatment