Assessing and inferring intra and inter-rater agreement

The research work wants to provide a scientific contribution in the field of subjective decision making since the assessment of the consensus, or equivalently the degree of agreement, among a group of raters as well as between more series of evaluations provided by the same rater, on categorical scales is a subject of both scientific and practical interest. Specifically, the research work focuses on the analysis of measures of agreement commonly adopted for assessing the performance (evaluative abilities) of one or more human raters (i.e. a group of raters) providing subjective evaluations about a given set of items/subjects. This topic is common to many contexts, ranging from medical (diagnosis) to engineering (usability test), industrial (visual inspections) or agribusiness (sensory analysis) contexts. In the thesis work, the performance of the agreement indexes under study, belonging to the family of the kappa-type agreement coefficients, have been assessed mainly regarding their inferential aspects, focusing the attention on those scenarios with small sample sizes which do not satisfy the asymptotic conditions required for the applicability of the standard inferential methods. Those scenarios have been poorly investigated in the specialized literature, although there is an evident interest in many experimental contexts. The critical analysis of the specialized literature highlighted two criticisms regarding the adoption of the agreement coefficients: 1) the degree of agreement is generally characterized by a straightforward benchmarking procedure that does not take into account the sampling uncertainty; 2) there is no evidence in the literature of a synthetic index able to assess the performance of a rater and/or of a group of raters in terms of more than one evaluative abilities (for example repeatability and reproducibility). Regarding the former criticism, an inferential benchmarking procedure based on non parametric confidence intervals, build via bootstrap resampling techniques, has been suggested. The statistical properties of the suggested benchmarking procedure have been investigated via a Monte Carlo simulation study by exploring many scenarios defined by varying: level of agreement, sample size and rating scale dimension. The simulation study has been carried out for different agreement coefficients and building different confidence intervals, in order to provide a comparative analysis of their performances. Regarding the latter criticism, instead, has been proposed a novel composite index able to assess the rater abilities of providing both repeatable (i.e. stable over time) and reproducible (i.e. consistent over different rating scales) evaluations. The inferential benchmarking procedure has been extended also to the proposed composite index and their performances have been investigated under different scenarios via a Monte Carlo simulation. The proposed tools have been successfully applied to two real case studies, about the assessment of university teaching quality and the sensory analysis of some food and beverage products, respectively

Student satisfaction and dissatisfaction - a study in the higher education context

This application for PhD by publication is the culmination of a series of investigations concerning the development of a conceptual model of student satisfaction with their higher education (HE) experience. The five core papers that sequentially contributed to knowledge are presented within this application, together with their supporting papers. In particular, the investigations focussed on what were the main determinants of student satisfaction utilising critical incident technique (CIT). New determinants of quality within higher education were identified as a result, namely motivation, praise/reward, social inclusion, usefulness, value for money and fellow student behaviour. The resultant research papers have made an original contribution to knowledge in the area of quality in HE. The rationale for using CIT to gather and analyse data was to investigate its effectiveness in triangulating with existing methods used in HE to measure student satisfaction. The CIT research was triggered by previous research that had developed and tested a student satisfaction questionnaire. CIT is used to gather data that is defined as ‘rich’. It is ideal when researching new areas. The questionnaire was based on the work of Sasser et al (1978) who proposed that service delivery consisted of a ‘bundle’ of goods and services, which incorporated distinct elements (the facilitating goods, the sensual service and the psychological service). The CIT survey instrument subsequently adopted, built upon the seminal work by Flanagan (1954) who developed it to assess the psychological impact (on pilots) when learning to fly. Both survey instruments were used to identify the determinants of quality in higher education from a student’s point of view and ultimately what was considered “critically critical” (Edvardsson & Nilsson-Wittel, 2004). An evaluation of teaching quality was carried out as part of this series during the mid-2000 era, and a comparison made of some of the existing measurement methods used at the time, such as student feedback questionnaires and peer review practices. A review of the potential use of mystery students in higher education was also carried out. This particular study highlighted the lack of confidence (by teaching personnel) in the existing methods for evaluating teaching quality as well as some mistrust of the concept of using mystery students in a classroom setting. A multi-method approach was chosen for this series of studies, because of the merits in using both quantitative and qualitative studies to generate data. Using such an approach provided a sequential method of analysing and presenting the data for this cross-sectional study. The initial student satisfaction questionnaire was a quantitative instrument and gathered data on importance and satisfaction ratings (using a five-point Likert scale) which were further analysed used quadrant analysis. The CIT survey was largely qualitative in nature and gathered written narratives from students about their positive and negative experiences; the resulting data used interpretive thematic analysis to identify key themes and any resulting patterns that could be coded quantitatively for input into the statistical software package SPSS. Each piece of research was underpinned by the existing literature at the time; this has inevitably progressed since then. CIT has been widely used in the service sector and additional determinants of quality within higher education have been identified within the current literature. A number of papers presented with this application have generated academic discussion in the field and these are evidenced by the number of citations for the applicant’s work. Moreover, the applicant’s additional supplementary papers also appended for background information have also been cited within the academic literature. The findings can be applied to teaching practice and within policy documents that support front-line teaching (and other) personnel in higher education

Portland Daily Press: July 18,1867

https://digitalmaine.com/pdp_1867/1217/thumbnail.jp

Rektoratsbericht 2015

Rektoratsbericht für das Jahr 201

Procceedings / 4th International Symposium of Industrial Engineering - SIE 2009, December 10-11, 2009., Belgrade

editors Dragan D. Milanović, Vesna Spasojević-Brkić, Mirjana Misit

Procceedings / 4th International Symposium of Industrial Engineering - SIE 2009, December 10-11, 2009., Belgrade

editors Dragan D. Milanović, Vesna Spasojević-Brkić, Mirjana Misit

Comparative Analysis of Student Learning: Technical, Methodological and Result Assessing of PISA-OECD and INVALSI-Italian Systems .

PISA is the most extensive international survey promoted by the OECD in the field of education, which measures the skills of fifteen-year-old students from more than 80 participating countries every three years. INVALSI are written tests carried out every year by all Italian students in some key moments of the school cycle, to evaluate the levels of some fundamental skills in Italian, Mathematics and English. Our comparison is made up to 2018, the last year of the PISA-OECD survey, even if INVALSI was carried out for the last edition in 2022. Our analysis focuses attention on the common part of the reference populations, which are the 15-year-old students of the 2nd class of secondary schools of II degree, where both sources give a similar picture of the students

Advanced Process Monitoring for Industry 4.0

This book reports recent advances on Process Monitoring (PM) to cope with the many challenges raised by the new production systems, sensors and “extreme data” conditions that emerged with Industry 4.0. Concepts such as digital-twins and deep learning are brought to the PM arena, pushing forward the capabilities of existing methodologies to handle more complex scenarios. The evolution of classical paradigms such as Latent Variable modeling, Six Sigma and FMEA are also covered. Applications span a wide range of domains such as microelectronics, semiconductors, chemicals, materials, agriculture, as well as the monitoring of rotating equipment, combustion systems and membrane separation processes

Proceedings / 6th International Symposium of Industrial Engineering - SIE 2015, 24th-25th September, 2015, Belgrade

editors Vesna Spasojević-Brkić, Mirjana Misita, Dragan D. Milanovi