Development and psychometric validation of pain scales in feline osteoarthritis

L’arthrose féline est détectable sur radiographie, surtout chez l’animal âgé. La rareté antérieure du diagnostic clinique s’explique par ses signes subtils et facilement attribués à d’autres maladies gériatriques ou au processus normal de vieillissement. Ces signes répondent néanmoins au traitement analgésique. Le but de ce projet de recherche était de développer et valider deux grilles de douleur arthrosique (Montreal Instruments for Cat Arthritis Testing), une pour les propriétaires de chats [MI-CAT(C)], et une pour les vétérinaires [MI-CAT(V)]. Le développement était fondé sur une revue de la littérature, notre expertise clinique en douleur et en comportement félin, et un sondage de propriétaires de chats arthrosiques. Des experts internes et externes ont confirmé la validité de contenu des grilles. Ensuite, une étude pilote sur chats de laboratoire a permis une évaluation préliminaire de leur fiabilité et validité. Dans le cadre d’un essai clinique chez des chats arthrosiques, la grille pour propriétaires MI-CAT(C) discriminait les groupes placebo et meloxicam, et ses changements de score corrélaient avec l’activité motrice et l’âge, soutenant sa validité. La grille était généralement facile à comprendre, appuyant de façon préliminaire sa validité de face (l’acceptabilité) et son interprétation. La mesure de fiabilité intra- et inter-observateur préconisait l’évaluation par le propriétaire principal vs. un(e) propriétaire secondaire. La grille MI-CAT(C) était homogène, sans redondance, selon l’évaluation préliminaire de la consistance interne. Une seconde évaluation de la grille vétérinaire MI-CAT(V) a été menée chez des chats de laboratoire (avec ou sans arthrose naturelle). L’évaluation de la fiabilité intra- et inter-observateur démontrait une courbe d’apprentissage pour le nouvel utilisateur de la grille. Seules les sous-catégories Gait (démarche) et Posture (allure) avaient une tendance (non-significative) à détecter le statut arthrosique; la palpation et la manipulation des articulations n’avait aucune sensibilité du même genre. Gait et Posture corrélaient avec une mesure objective, la force verticale d’appui au sol. Une analyse vidéo a ensuite été faite pour améliorer la sensibilité de la grille MI-CAT(V) à l’arthrose. La grille révisée a été soumise à des étapes successives de validation et de raffinement, via trois études thérapeutiques (utilisant la gabapentine, le tramadol, et le meloxicam sous forme orale transmuqueuse par vaporisateur). Sa fiabilité intra- et inter-observateur, et l’évaluation préliminaire de la consistance interne étaient bonnes à excellentes, et elle fut capable de détecter le statut arthrosique. Cependant, elle ne détecta pas les effets thérapeutiques démontrés par d’autres mesures objectives. Des recherches ultérieures devront confirmer que la grille pour propriétaires MI-CAT(C) distingue le statut arthrosique, et évaluer sa réponse, vs. placébo, à d’autres traitements que le meloxicam. La grille vétérinaire MI-CAT(V) requerra une confirmation de sa fiabilité et validité chez des chats de propriétaires ; elle nécessitera encore des raffinements pour détecter les effets de traitement. L’établissement de seuils (p. ex. : distinction arthrosique/non-arthrosique, différence minimale significative) pour les deux grilles est conseillé pour faciliter leur utilisation clinique, ainsi qu’une évaluation de leur faisabilité et utilité clinique, ainsi qu’une réévaluation de leur structure interne et de leur compréhension.Radiographic signs of osteoarthritis are prevalent in cats, becoming more common with age. Historically, the rate of diagnosis has tended to be low, suggesting that signs are subtle and/or tend to be attributed to normal age-related changes or to other geriatric diseases. However, cats with osteoarthritis display signs that are responsive to analgesic treatment. This project aimed to develop and validate rating scales for detection and measurement of feline osteoarthritis pain and related disability (the Montreal Instruments for Cat Arthritis Testing). Two such scales, one for use by caretakers/owners [MI-CAT(C)], and one for use by veterinarians [MI-CAT(V)], were developed based on a review of the literature, expert opinion, and a survey study of owners of cats with a diagnosis of osteoarthritis. The content validity (via expert review) was excellent for both scales. A pilot study in a colony of laboratory cats with naturally-occurring osteoarthritis evaluated reliability and ability to detect osteoarthritis status, for both scales; preliminary revisions were made based on the results. The MI-CAT(C) owner scale subsequently underwent validation in a clinical trial of meloxicam in client-owned osteoarthritic cats. Evidence for validity included the ability to distinguish placebo from active treatment, and correlations with objectively measured activity and age. Owners found most scale items clear/easy to understand, preliminarily supporting comprehensibility and face validity (acceptability). Evaluation of intra- and inter-rater reliability suggested that secondary owners varied substantially in their ability to complete the scale, compared to primary owners. A preliminary assessment of internal consistency reliability supported homogeneity, without redundancy, of the scale. The MI-CAT(V) veterinary scale was evaluated in a study of laboratory cats with and without naturally-occurring osteoarthritis. Intra- and inter-rater reliability assessments suggested that a naïve user’s ability to use the scale was influenced by experience with it. The scale was unable to distinguish osteoarthritic and non-osteoarthritic cats, but the subcategories Gait and Posture were somewhat promising based on a non-significant tendency to detect osteoarthritis status, and correlations with an objective measure of osteoarthritis pain, peak vertical force. Palpation of the limbs did not detect osteoarthritis status. A video analysis was performed to increase MI-CAT(V) scale sensitivity to osteoarthritis. Subsequent evaluation and refinements based on three therapeutic trials (involving gabapentin, tramadol, and oral transmucosal meloxicam treatments) in laboratory cats with and without naturally-occurring osteoarthritis resulted in good to excellent intra- and inter-rater reliability, and ability to detect osteoarthritis status. Preliminary evidence supported scale internal consistency. Therapeutic response detected by objective outcome measures was not demonstrable using the scale. It is recommended that the MI-CAT(C) owner scale be evaluated for ability to distinguish osteoarthritic from non-osteoarthritic cats. The MI-CAT(V) veterinary scale requires testing in client-owned cats, and potentially further refinements to permit detection of treatment effects, if it is to be used as more than a disease screening tool. Both scales require additional investigation of internal structure and comprehensibility, and determination of cut-points to guide clinical use (e.g., minimally important difference, and thresholds for classification of cats as osteoarthritic vs. non-osteoarthritic), and evaluation of their feasibility and clinical utility

Assessing Reliability of Expert Ratings Among Judges Responding to a Survey Instrument Developed to Study the Long Term Efficacy of the ABET Engineering Criteria, EC2000

In today’s assessment processes, especially those evaluations that rely on humans to make subjective judgements, it is necessary to analyze the quality of their ratings. The psychometric issues associated with assessment provide the lens through which researchers interpret results and important decisions are made. Therefore, inter-rater agreement (IRA) and inter-rater reliability (IRR) are pre-requisites for rater-dependent data analysis. A survey instrument cannot provide “good” information if it is not reliable; in other words, reliability is central to the validation of an instrument. When judges cannot be shown to reliably rate a performance, item, or target, the question becomes why the judges’ responses are different from one another. If the judges’ ratings covary unreliably because the construct is poorly defined or the rating framework is defective, then the resultant scores will have questionable meaning. On the other hand, if the judges’ ratings differ because they have a true difference in opinion, this is of importance to the researcher and may not necessarily diminish the validity of the scores. The intraclass correlation coefficient (ICC) is the most efficient method to assess these rater differences and identify the specific sources of inconsistency in measurement. This study examined how ICCs can be used to inform researchers of the extent in which legitimate differences of opinion may appear as a lack of reliability and/or agreement, demonstrating the need for analyzing survey data beyond standard descriptive statistics. Overall, both the IRA and IRR correlations, as calculated by ICC, ranged from .79 to .91 indicating high levels of agreement and consistency in the scoring among the judges\u27 ratings. When group membership was accounted for the IRA values increased suggesting the common judges agreed more than those judges who varied in their perspectives

Development And Psychometric Evaluation Of An Instrument To Assess The Treatment Fidelity Of A Brief Opportunistic Intervention To Reduce Substance Use Among Pregnant Women

Although abstinence from alcohol, tobacco, and other drugs (ATOD) among pregnant women is a leading national objective, prenatal use has not decreased. Evidence-based interventions that can be replicated in practice are critically needed and brief interventions have shown promise in reducing prenatal ATOD use. The I Am Concerned (IAC) brief opportunistic intervention is currently being implemented by frontline primary prenatal care staff members in several areas of the United States. Evaluation of treatment fidelity, to determine if behavioral interventions are delivered as intended, is essential to controlled research. This study constituted the first step in the development and psychometric evaluation of an instrument designed to measure the treatment fidelity with which the IAC brief opportunistic intervention is implemented. A conceptual framework derived from motivational interviewing and self-determination theory, both based on the fundamental assumption that individuals are inherently inclined toward positive change, guided operationalization of the IAC behavioral elements that ultimately took shape as the 18-item IAC treatment fidelity instrument. This methodologic study used a 6-phase protocol to develop and refine the IAC treatment fidelity instrument and evaluate its psychometric properties. Independent raters used the instrument to evaluate audio recordings (N = 49) of experienced frontline staff members implementing the IAC brief opportunistic intervention with standardized patients portraying ATOD-using pregnant women in a simulated clinic setting. Psychometric analysis provided evidence of content validity. Intra-class correlation coefficients (ICC) calculated for inter-rater reliability were satisfactory for subscales (0.64) and (0.62) and ranged from -0.07 to 0.81 for individual items. Internal consistency alpha coefficients were satisfactory for the total scale (0.72) and lower than acceptable for adherence (0.54) and competence (0.56) subscales. Overall high rater percentage agreement and negatively skewed ratings distribution indicated that reliability results were paradoxically low due to the base rate problem. The study results support revision and ongoing testing of the IAC treatment fidelity instrument

The parent programme implementation checklist (PPIC): the development and testing of an objective measure of skills and fidelity for the delivery of parent programmes

This is the author accepted manuscript. The final version is available from Taylor & Francis via the DOI in this recordBackground: Group-based parent programmes demonstrate positive benefits for adult and child mental health, and child behaviour outcomes. Greater fidelity to the programme delivery model equates to better outcomes for families attending, however, fidelity is typically self-monitored using programme specific checklists. Self-completed measures are open to bias, and it is difficult to know if positive outcomes found from research studies will be maintained when delivered in regular services. Currently, ongoing objective monitoring of quality is not conducted during usual service delivery. This is odd given that quality of other services is assessed objectively, for example by the Office for Standards in Education, Children's Services and Skills (OFSTED). Independent observations of programme delivery are needed to assess fidelity and quality of delivery to ensure positive outcomes, and therefore justify the expense of programme delivery. Methods: This paper outlines the initial development and reliability of a tool, the Parent Programme Implementation Checklist (PPIC), which was originally developed as a simple, brief and generic observational tool for independent assessment of implementation fidelity of group-based parent programmes. PPIC does not require intensive observer training before application/use. This paper presents initial data obtained during delivery of the Incredible Years BASIC programme across nine localities in England and Wales, United Kingdom (UK). Results: Reasonable levels of inter-rater reliability were achieved across each of the three subscales (Adherence, Quality and Participant Responsiveness) and the overall total score when applying percentage agreements (>70%) and intra-class correlations (ICC) (ICC range between 0.404 and 0.730). Intra-rater reliability (n = 6) was acceptable at the subscale level. Conclusions: We conclude that the PPIC has promise, and with further development could be utilised to assess fidelity of parent group delivery during research trials and standard service delivery. Further development would need to include data from other parent programmes, and testing by non-research staff. The objective assessment of quality of delivery would inform services where improvements could be made.This research was supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care – Yorkshire and Humber, and South West Peninsula

Informed consent in randomised controlled trials:further development and evaluation of the participatory and informed consent (PIC) measure

Background: Informed consent is an accepted ethical and legal prerequisite for trial participation, yet there is no standardised method of assessing patient understanding for informed consent. The participatory and informed consent (PIC) measure was developed for application to recruitment discussions to evaluate recruiter information provision and evidence of patient understanding. Preliminary evaluation of the PIC indicated the need to improve inter-rater and intra-rater reliability ratings and conduct further psychometric evaluation. This paper describes the assessment, revision and evaluation of the PIC within the context of OPTiMISE, a pragmatic primary care-based trial. Methods: This study used multiple methods across two phases. In phase one, one researcher applied the existing PIC measure to 18 audio-recorded recruitment discussions from the OPTiMISE study and made detailed observational notes about any uncertainties in application. Appointments were sampled to be maximally diverse for patient gender, study centre, recruiter and before and after an intervention to optimise information provision. Application uncertainties were reviewed by the study team, revisions made and a coding manual developed and agreed. In phase two, the coding manual was used to develop tailored guidelines for applying the PIC to appointments within the OPTiMISE trial. Two researchers then assessed 27 further appointments, purposively sampled as above, to evaluate inter-rater and intra-rater reliability, content validity and feasibility. Results: Application of the PIC to 18 audio-recorded OPTiMISE recruitment discussions resulted in harmonisation of the scales rating recruiter information provision and evidence of patient understanding, minor amendments to clarify wording and the development of detailed generic coding guidelines for applying the measure within any trial. Application of the revised measure using these guidelines to 27 further recruitment discussions showed good feasibility (time to complete), content validity (completion rate) and reliability (inter- and intra-rater) of the measure. Conclusion: The PIC provides a means to evaluate the content of information provided by recruiters, patient participation in recruitment discussions and, to some extent, evidence of patient understanding. Future work will use the measure to evaluate recruiter information provision and evidence of patient understanding both across and within trials

The assessment of non-technical skills in ENT surgery: a multidisciplinary simulation programme to improve patient safety

Surgical patients are at particular risk of harm, with 41% of all adverse events in hospital occurring in the operating theatre. Failures in Human factors are the leading cause. Despite recognition of the importance of human factors training to patient safety, there is a lack of theatre ENT crisis management simulation, and no formal assessment of the requisite skills. Aims: To Develop a psychometrically robust assessment tool for assessing Non-technical skills in the ENT theatre – to be termed ENT-NOTECHS. To Develop and validate an ENT themed multidisciplinary simulation programme for the assessment and feedback of non-technical skills. Methods: A multimodal method approach was used to create a novel behavioural marker tool to capture non-technical skills in the ENT theatre environment. Alongside this, a prospective, observational study involving a multidisciplinary team training day in ENT and airway themed crisis’ in a high fidelity simulated theatre environment was designed. Teams undertook 6 high fidelity simulation scenarios and non-technical skills were assessed using the ENT-NOTECHS tool. The ENT-NOTECHS tool was assessed for its psychometric robustness; reliability and construct validity. Candidate feedback was obtained to determine overall effectiveness of training. Results: We successfully designed and delivered a novel multidisciplinary team ENT themed training day. Over 15 months, 74 trainees (surgeons, anaesthetists and nurses) participated in 6 MDT simulation days, totalling 54 hours of simulation training and 210 assessments. Excellent Face and content validity was demonstrated. 100% of participants reported improved confidence in managing ENT crisis scenarios and demonstrated an improvement in non-technical skills (ENT-NOTECHS). The ENT-NOTECHS tool demonstrated excellent psychometric robustness. Good inter-rater reliability scores (cronbachs >0.7) were shown and the tool discriminated between novice and expert trainees (p<0.001). Conclusion: Multidisciplinary team training in ENT-themed crisis is feasible and well received training intervention. The simulated operating theatre serves as an excellent environment for the assessment and training of non-technical skills. ENT -NOTECHS is a novel assessment tool with evidence for reliability, content and construct validity in ENT teams.Open Acces

Leadership in medical ward rounds

Leadership in Medical Ward Rounds: Abstract Medical ward rounds are an under researched area. The medical post take ward round has been cited as being a source of anxiety for new consultants. The non-technical skills involved may well be those that new consultants feel underprepared for in contrast to clinical skills. Ward rounds historically have been a principal vehicle for teaching junior doctors. There have been many changes in how junior doctors work which has potentially impacted their training and preparation towards being a consultant. The overarching aims of this thesis are firstly to expand our current understanding of the incorporation of training into medical ward rounds, and secondly to translate this understanding into an instrument that evaluates senior trainees or consultants skills in leading a ward round. Ultimately, improved training and assessment of the ward round process should enhance patient safety and effectiveness of care on medical wards. This thesis incorporates a narrative review on training and ward rounds. There is also a literature review on non-technical skills tools used in hospital medicine, how they were developed and their psychometric evaluation. The second review of non-technical skills tools leads to a choice of tool on which to base the development of a ward round leadership tool. The review on training and ward rounds, provides background to the thesis but also some of the findings are used for the instrument development. A post take ward round simulation was developed alongside the ward round leadership tool, which serves 2 purposes. One is to develop a training program by which to train senior medical registrars to lead post take ward rounds, and secondly, it is used to psychometrically evaluate the developed medical ward round leadership tool. There is also a chapter reporting an interview study of medical consultants and patients about training and post take ward rounds. The findings from this chapter feed directly into the tool and simulation development. The development of the simulation and tool are described and evaluated in detail. The tool is evaluated in terms of reliability and validity.Open Acces

ADRIC: Adverse Drug Reactions In Children - a programme of research using mixed methods

Aims To comprehensively investigate the incidence, nature and risk factors of adverse drug reactions (ADRs) in a hospital-based population of children, with rigorous assessment of causality, severity and avoidability, and to assess the consequent impact on children and families. We aimed to improve the assessment of ADRs by development of new tools to assess causality and avoidability, and to minimise the impact on families by developing better strategies for communication. Review methods Two prospective observational studies, each over 1 year, were conducted to assess ADRs in children associated with admission to hospital, and those occurring in children who were in hospital for longer than 48 hours. We conducted a comprehensive systematic review of ADRs in children. We used the findings from these studies to develop and validate tools to assess causality and avoidability of ADRs, and conducted interviews with parents and children who had experienced ADRs, using these findings to develop a leaflet for parents to inform a communication strategy about ADRs. Results The estimated incidence of ADRs detected in children on admission to hospital was 2.9% [95% confidence interval (CI) 2.5% to 3.3%]. Of the reactions, 22.1% (95% CI 17% to 28%) were either definitely or possibly avoidable. Prescriptions originating in the community accounted for 44 out of 249 (17.7%) of ADRs, the remainder originating from hospital. A total of 120 out of 249 (48.2%) reactions resulted from treatment for malignancies. Off-label and/or unlicensed (OLUL) medicines were more likely to be implicated in an ADR than authorised medicines [relative risk (RR) 1.67, 95% CI 1.38 to 2.02; p  48 hours, the overall incidence of definite and probable ADRs based on all admissions was 15.9% (95% CI 15.0 to 16.8). Opiate analgesic drugs and drugs used in general anaesthesia (GA) accounted for > 50% of all drugs implicated in ADRs. The odds ratio of an OLUL drug being implicated in an ADR compared with an authorised drug was 2.25 (95% CI 1.95 to 2.59; p < 0.001). Risk factors identified were exposure to a GA, age, oncology treatment and number of medicines. The systematic review estimated that the incidence rates for ADRs causing hospital admission ranged from 0.4% to 10.3% of all children [pooled estimate of 2.9% (95% CI 2.6% to 3.1%)] and from 0.6% to 16.8% of all children exposed to a drug during hospital stay. New tools to assess causality and avoidability of ADRs have been developed and validated. Many parents described being dissatisfied with clinician communication about ADRs, whereas parents of children with cancer emphasised confidence in clinician management of ADRs and the way clinicians communicated about medicines. The accounts of children and young people largely reflected parents’ accounts. Clinicians described using all of the features of communication that parents wanted to see, but made active decisions about when and what to communicate to families about suspected ADRs, which meant that communication may not always match families’ needs and expectations. We developed a leaflet to assist clinicians in communicating ADRs to parents. Conclusion The Adverse Drug Reactions In Children (ADRIC) programme has provided the most comprehensive assessment, to date, of the size and nature of ADRs in children presenting to, and cared for in, hospital, and the outputs that have resulted will improve the management and understanding of ADRs in children and adults within the NHS. Recommendations for future research: assess the values that parents and children place on the use of different medicines and the risks that they will find acceptable within these contexts; focusing on high-risk drugs identified in ADRIC, determine the optimum drug dose for children through the development of a gold standard practice for the extrapolation of adult drug doses, alongside targeted pharmacokinetic/pharmacodynamic studies; assess the research and clinical applications of the Liverpool Causality Assessment Tool and the Liverpool Avoidability Assessment Tool; evaluate, in more detail, morbidities associated with anaesthesia and surgery in children, including follow-up in the community and in the home setting and an assessment of the most appropriate treatment regimens to prevent pain, vomiting and other postoperative complications; further evaluate strategies for communication with families, children and young people about ADRs; and quantify ADRs in other settings, for example critical care and neonatology