Calibration with confidence:A principled method for panel assessment

Frequently, a set of objects has to be evaluated by a panel of assessors, but not every object is assessed by every assessor. A problem facing such panels is how to take into account different standards amongst panel members and varying levels of confidence in their scores. Here, a mathematically-based algorithm is developed to calibrate the scores of such assessors, addressing both of these issues. The algorithm is based on the connectivity of the graph of assessors and objects evaluated, incorporating declared confidences as weights on its edges. If the graph is sufficiently well connected, relative standards can be inferred by comparing how assessors rate objects they assess in common, weighted by the levels of confidence of each assessment. By removing these biases, "true" values are inferred for all the objects. Reliability estimates for the resulting values are obtained. The algorithm is tested in two case studies, one by computer simulation and another based on realistic evaluation data. The process is compared to the simple averaging procedure in widespread use, and to Fisher's additive incomplete block analysis. It is anticipated that the algorithm will prove useful in a wide variety of situations such as evaluation of the quality of research submitted to national assessment exercises; appraisal of grant proposals submitted to funding panels; ranking of job applicants; and judgement of performances on degree courses wherein candidates can choose from lists of options.Comment: 32 pages including supplementary information; 5 figure

Logistic early warning scores to predict death, cardiac arrest or unplanned intensive care unit re-admission after cardiac surgery.

NHS England recently mandated that the National Early Warning Score of vital signs be used in all acute hospital trusts in the UK despite limited validation in the postoperative setting. We undertook a multicentre UK study of 13,631 patients discharged from intensive care after risk-stratified cardiac surgery in four centres, all of which used VitalPACTM to electronically collect postoperative National Early Warning Score vital signs. We analysed 540,127 sets of vital signs to generate a logistic score, the discrimination of which we compared with the national additive score for the composite outcome of: in-hospital death; cardiac arrest; or unplanned intensive care admission. There were 578 patients (4.2%) with an outcome that followed 4300 sets of observations (0.8%) in the preceding 24 h: 499 out of 578 (86%) patients had unplanned re-admissions to intensive care. Discrimination by the logistic score was significantly better than the additive score. Respective areas (95%CI) under the receiver-operating characteristic curve with 24-h and 6-h vital signs were: 0.779 (0.771-0.786) vs. 0.754 (0.746-0.761), p < 0.001; and 0.841 (0.829-0.853) vs. 0.813 (0.800-0.825), p < 0.001, respectively. Our proposed logistic Early Warning Score was better than the current National Early Warning Score at discriminating patients who had an event after cardiac surgery from those who did not

Ultrashort Echo Time Imaging of the Osteochondral Junction in Subjects with Knee Osteoarthritis and Age-matched Healthy Volunteers

SYNOPSIS We describe in vivo translation of ultrashort TE (UTE) imaging of the osteochondral junction (OCJ) at the knee in 9 subjects with osteoarthritis (OA) and 4 age-matched healthy volunteers. The OCJ plays an important role in onset and progression of OA. Our study demonstrates that UTE imaging of the OCJ is repeatable and demonstrates OCJ defects in OA subjects but not in healthy volunteers. Areas of OCJ damage commonly co-locate to other osteochondral pathology (bone marrow lesions and cartilage defects). UTE imaging of the OCJ may be a helpful tool for assessing OCJ damage in clinical studies of OA. INTRODUCTION Disruption of the osteochondral junction (OCJ) is thought to play an important role in the onset and progression of osteoarthritis (OA). Using conventional MR imaging, direct visualisation of the OCJ is not possible due to inherent short T1 and T2 relaxation times of the OCJ tissues. However, by achieving echo times (TEs) of < 1 ms, ultrashort echo time (UTE) MR imaging allows direct visualisation of the OCJ. The normal OCJ appears as an area of linear high signal intensity (SI) on UTE images at the bone-cartilage interface. In OA it has been shown that this area of linear high SI can become thinned or absent, compatible with histological findings of OCJ defects(1, 2). These findings have been described in a number of cadaveric MR studies, but there are limited in vivo data available(3-5). The aims of this study were to compare the in vivo appearance of the OCJ on UTE MR imaging between subjects with knee OA and age-matched healthy volunteers, to determine the relationship between OCJ defects and other osteochondral pathology, and to assess test-retest repeatability. METHODS We imaged 9 participants with mild-moderate knee osteoarthritis, characterised by radiographs with medial tibiofemoral compartment predominant disease and Kellgren-Lawrence grades 2-3, and 4 age-matched healthy volunteers. Participants were imaged at baseline and 1 month. MR studies were performed on a 3T system (GE 750, GE Healthcare). The MR protocol consisted of standard clinical sequences (coronal and sagittal intermediate-weighted fat-saturated fast spin echo (FSE) sequences plus a coronal T1-weighted FSE sequence) and a sagittal dual-echo UTE gradient echo sequence acquired using a 3D cones trajectory (research prototype; repetition time 15 ms, TE 0.03/4.5 ms, flip angle 13o, field-of-view 18 x 18 cm, matrix 430 x 430, slice thickness 2 mm, number of averages 1, acquisition time ~ 7.5 minutes). To increase conspicuity of short T2 tissues, we performed weighted digital image subtraction of the longer TE (4.5 ms) from the shorter TE images (0.03 ms)(6). The presence or absence of characteristic linear high SI at the OCJ was scored in 12 regions for each knee, corresponding to tibiofemoral subdivisions commonly used for semi-quantitative scoring. The presence of bone marrow lesions (BML) or cartilage defects in the same regions was also recorded. Assessment was performed by a single musculoskeletal radiologist with 5 years' experience in OA research, blinded to group assignment. We used descriptive statistics to compare the number of regions with OCJ defects in subjects with OA and healthy volunteers, and to assess the frequency with which OCJ defects co-located with BMLs or cartilage defects. Test-retest repeatability was evaluated using kappa statistics. RESULTS Participant characteristics are displayed in table 1. Six out of 9 OA participants (67%) had an OCJ defect in at least one region compared to 0 out of 4 controls (0%). The most commonly involved region was the central medial tibia (4 participants). OCJ defects commonly co-located to BMLs (7 out of 10 OCJ defects, 70%) and cartilage defects (6 out of 10 OCJ defects, 60%). Results are displayed in table 2. Sample images are displayed in figures 1 - 3. The kappa value for test-retest repeatability of OCJ assessment using UTE was 0.83 (95% confidence interval 0.64 to 1). DISCUSSION The appearances of OCJ defects in subjects with OA in vivo are in keeping with abnormalities predicted by cadaveric MR and histology studies(1, 3). The biological plausibility of the findings is enhanced by the frequency of co-location of OCJ damage to other osteochondral pathology (BMLs and cartilage defects). Our findings demonstrate in vivo translation of UTE imaging of the OCJ, and suggest that this is a useful tool for future studies of OA onset and progression. This may include predicting response to intervention, as equine studies have demonstrated that the presence or absence of OCJ damage is an important predictor of response to treatment of cartilage defects(7). Our results demonstrate that UTE imaging of the OCJ is repeatable with kappa values in keeping with 'near-perfect' test-retest repeatability for qualitative assessment(8). Previous in vivo studies have not used age-matched control subjects, therefore it has been unclear whether areas of OCJ damage are related to OA or normal ageing(4). The normal appearance of the OCJ in age-matched control subjects in this study suggests that the OCJ defects are not part of normal ageing, although at present the number of healthy volunteers imaged is small. CONCLUSION In vivo UTE MR imaging of the OCJ is repeatable and demonstrates OCJ defects in subjects with OA. OCJ defects commonly co-locate with other osteochondral pathology

Subchondral bone in osteoarthritis: association between MRI texture analysis and histomorphometry.

OBJECTIVE: Magnetic resonance imaging (MRI) texture analysis is a method of analyzing subchondral bone alterations in osteoarthritis (OA). The objective of this study was to evaluate the association between MR texture analysis and ground-truth subchondral bone histomorphometry at the tibial plateau. DESIGN: The local research ethics committee approved the study. All subjects provided written, informed consent. This was a cross-sectional study carried out at our institution between February and August 2014. Ten participants aged 57-84 with knee OA scheduled for total knee arthroplasty (TKA) underwent pre-operative MRI of the symptomatic knee at 3T using a high spatial-resolution coronal T1 weighted sequence. Tibial plateau explants obtained at the time of TKA underwent histological preparation to allow calculation of bone volume fraction (BV.TV), trabecular thickness (Tb.Th), trabecular separation (Tb.Sp) and trabecular number (Tb.N). Texture analysis was performed on the tibial subchondral bone of MRI images matched to the histological sections. Regression models were created to assess the association of texture analysis features with BV.TV, Tb.Th, Tb.Sp and Tb.N. RESULTS: MRI texture features were significantly associated with BV.TV (R2 = 0.76), Tb.Th (R2 = 0.47), Tb.Sp (R2 = 0.75) and Tb.N (R2 = 0.60, all P < 0.001). Simple gray-value histogram based texture features demonstrated the highest standardized regression coefficients for each model. CONCLUSION: MRI texture analysis features were significantly associated with ground-truth subchondral bone histomorphometry at the tibial plateau.Royal College of Radiologists Pump Priming Gran

Systematic review and meta-analysis of the reliability and discriminative validity of cartilage compositional MRI in knee osteoarthritis.

OBJECTIVE: To assess reliability and discriminative validity of cartilage compositional magnetic resonance imaging (MRI) in knee osteoarthritis (OA). DESIGN: The study was carried out per PRISMA recommendations. We searched MEDLINE and EMBASE (1974 - present) for eligible studies. We performed qualitative synthesis of reliability data. Where data from at least two discrimination studies were available, we estimated pooled standardized mean difference (SMD) between subjects with and without OA. Discrimination analyses compared controls and subjects with mild OA (Kellgren-Lawrence (KL) grade 1-2), severe OA (KL grade 3-4) and OA not otherwise specified (NOS) where not possible to stratify. We assessed quality of the evidence using Quality Appraisal of Diagnostic Reliability (QAREL) and Quality Assessment of Diagnostic Accuracy (QUADAS-2) tools. RESULTS: Fifty-eight studies were included in the reliability analysis and 26 studies were included in the discrimination analysis, with data from a total of 2,007 knees. Intra-observer, inter-observer and test-retest reliability of compositional techniques were excellent with most intraclass correlation coefficients >0.8 and coefficients of variation <10%. T1rho and T2 relaxometry were significant discriminators between subjects with mild OA and controls, and between subjects with OA (NOS) and controls (P < 0.001). T1rho showed best discrimination for mild OA (SMD [95% CI] = 0.73 [0.40 to 1.06], P < 0.001) and OA (NOS) (0.60 [0.41 to 0.80], P < 0.001). Quality of evidence was moderate for both parts of the review. CONCLUSIONS: Cartilage compositional MRI techniques are reliable and, in the case of T1rho and T2 relaxometry, can discriminate between subjects with OA and controls

Dynamic contrast-enhanced MRI of synovitis in knee osteoarthritis: repeatability, discrimination and sensitivity to change in a prospective experimental study

OBJECTIVES: Evaluate test-retest repeatability, ability to discriminate between osteoarthritic and healthy participants, and sensitivity to change over 6 months, of dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) biomarkers in knee OA. METHODS: Fourteen individuals aged 40-60 with mild-moderate knee OA and 6 age-matched healthy volunteers (HV) underwent DCE-MRI at 3 T at baseline, 1 month and 6 months. Voxelwise pharmacokinetic modelling of dynamic data was used to calculate DCE-MRI biomarkers including Ktrans and IAUC60. Median DCE-MRI biomarker values were extracted for each participant at each study visit. Synovial segmentation was performed using both manual and semiautomatic methods with calculation of an additional biomarker, the volume of enhancing pannus (VEP). Test-retest repeatability was assessed using intraclass correlation coefficients (ICC). Smallest detectable differences (SDDs) were calculated from test-retest data. Discrimination between OA and HV was assessed via calculation of between-group standardised mean differences (SMD). Responsiveness was assessed via the number of OA participants with changes greater than the SDD at 6 months. RESULTS: Ktrans demonstrated the best test-retest repeatability (Ktrans/IAUC60/VEP ICCs 0.90/0.84/0.40, SDDs as % of OA mean 33/71/76%), discrimination between OA and HV (SMDs 0.94/0.54/0.50) and responsiveness (5/1/1 out of 12 OA participants with 6-month change > SDD) when compared to IAUC60 and VEP. Biomarkers derived from semiautomatic segmentation outperformed those derived from manual segmentation across all domains. CONCLUSIONS: Ktrans demonstrated the best repeatability, discrimination and sensitivity to change suggesting that it is the optimal DCE-MRI biomarker for use in experimental medicine studies. KEY POINTS: • Dynamic contrast-enhanced MRI (DCE-MRI) provides quantitative measures of synovitis in knee osteoarthritis which may permit early assessment of efficacy in experimental medicine studies. • This prospective observational study compared DCE-MRI biomarkers across domains relevant to experimental medicine: test-retest repeatability, discriminative validity and sensitivity to change. • The DCE-MRI biomarker Ktrans demonstrated the best performance across all three domains, suggesting that it is the optimal biomarker for use in future interventional studies

Immune-mediated competition in rodent malaria is most likely caused by induced changes in innate immune clearance of merozoites

Malarial infections are often genetically diverse, leading to competitive interactions between parasites. A quantitative understanding of the competition between strains is essential to understand a wide range of issues, including the evolution of virulence and drug resistance. In this study, we use dynamical-model based Bayesian inference to investigate the cause of competitive suppression of an avirulent clone of Plasmodium chabaudi (AS) by a virulent clone (AJ) in immuno-deficient and competent mice. We test whether competitive suppression is caused by clone-specific differences in one or more of the following processes: adaptive immune clearance of merozoites and parasitised red blood cells (RBCs), background loss of merozoites and parasitised RBCs, RBC age preference, RBC infection rate, burst size, and within-RBC interference. These processes were parameterised in dynamical mathematical models and fitted to experimental data. We found that just one parameter μ, the ratio of background loss rate of merozoites to invasion rate of mature RBCs, needed to be clone-specific to predict the data. Interestingly, μ was found to be the same for both clones in single-clone infections, but different between the clones in mixed infections. The size of this difference was largest in immuno-competent mice and smallest in immuno-deficient mice. This explains why competitive suppression was alleviated in immuno-deficient mice. We found that competitive suppression acts early in infection, even before the day of peak parasitaemia. These results lead us to argue that the innate immune response clearing merozoites is the most likely, but not necessarily the only, mediator of competitive interactions between virulent and avirulent clones. Moreover, in mixed infections we predict there to be an interaction between the clones and the innate immune response which induces changes in the strength of its clearance of merozoites. What this interaction is unknown, but future refinement of the model, challenged with other datasets, may lead to its discovery

Automatic Network Fingerprinting through Single-Node Motifs

Complex networks have been characterised by their specific connectivity patterns (network motifs), but their building blocks can also be identified and described by node-motifs---a combination of local network features. One technique to identify single node-motifs has been presented by Costa et al. (L. D. F. Costa, F. A. Rodrigues, C. C. Hilgetag, and M. Kaiser, Europhys. Lett., 87, 1, 2009). Here, we first suggest improvements to the method including how its parameters can be determined automatically. Such automatic routines make high-throughput studies of many networks feasible. Second, the new routines are validated in different network-series. Third, we provide an example of how the method can be used to analyse network time-series. In conclusion, we provide a robust method for systematically discovering and classifying characteristic nodes of a network. In contrast to classical motif analysis, our approach can identify individual components (here: nodes) that are specific to a network. Such special nodes, as hubs before, might be found to play critical roles in real-world networks.Comment: 16 pages (4 figures) plus supporting information 8 pages (5 figures