Measurement of Rater Consistency and its Application in Voice Quality Assessments

Abstract

This paper concerns the assessment of voice quality as required for patients who attend a hospital voice clinic. Clinicians currently make assessments perceptually according to a well known 'GRBAS' scale. Our aim is to use machine learning (ML) to make computerised GRBAS assessments from measurements of features extracted and parameterised using digital signal processing (DSP). The ML is based on assessments by a group of clinicians (raters) and it is useful to measure and take into account the consistency of these assessments. This process has revealed some insight into commonly used techniques for measuring consistency, such as ICC and the Cohen and Fleiss Kappas. Results obtained from the application of these techniques to ML programs for GRBAS assessment are presented