Validation of spatial microsimulation models: a proposal to adopt the Bland-Altman method

Model validation is recognised as crucial to microsimulation modelling. However, modellers encounter difficulty in choosing the most meaningful methods to compare simulated and actual values. The aim of this paper is to introduce and demonstrate a method employed widely in healthcare calibration studies. The ‘Bland-Altman plot’ consists of a plot of the difference between two methods against the mean (x-y versus x+y/2). A case study is presented to illustrate the method in practice for spatial microsimulation validation. The study features a deterministic combinatorial model (SimObesity), which modelled a synthetic population for England at the ward level using survey (ELSA) and Census 2011 data. Bland-Altman plots were generated, plotting simulated and census ward-level totals for each category of all constraint (benchmark) variables. Other validation metrics, such as R2, SEI, TAE and RMSE, are also presented for comparison. The case study demonstrates how the Bland-Altman plots are interpreted. The simple visualisation of both individual- (ward-) level difference and total variation gives the method an advantage over existing tools used in model validation. There still remains the question of what constitutes a valid or well-fitting model. However, the Bland Altman method can usefully be added to the canon of calibration methods

Validation of spatial microsimulation models: a proposal to adopt the Bland-Altman method

Model validation is recognised as crucial to microsimulation modelling. However, modellers encounter difficulty in choosing the most meaningful methods to compare simulated and actual values. The aim of this paper is to introduce and demonstrate a method employed widely in healthcare calibration studies. The ‘Bland-Altman plot’ consists of a plot of the difference between two methods against the mean (x-y versus x+y/2). A case study is presented to illustrate the method in practice for spatial microsimulation validation. The study features a deterministic combinatorial model (SimObesity), which modelled a synthetic population for England at the ward level using survey (ELSA) and Census 2011 data. Bland-Altman plots were generated, plotting simulated and census ward-level totals for each category of all constraint (benchmark) variables. Other validation metrics, such as R2, SEI, TAE and RMSE, are also presented for comparison. The case study demonstrates how the Bland-Altman plots are interpreted. The simple visualisation of both individual- (ward-) level difference and total variation gives the method an advantage over existing tools used in model validation. There still remains the question of what constitutes a valid or well-fitting model. However, the Bland Altman method can usefully be added to the canon of calibration methods

Making the best use of new technologies in the National Diet and Nutrition Survey: a review

.Background Dietary assessment is of paramount importance for public health monitoring. Currently in the UK, the population’s diets are examined by the National Diet and Nutrition Survey Rolling Programme (NDNS RP). In the survey, diet is assessed by a four-day paper-based dietary diary, with accompanying interviews, anthropometric measurements and blood and urine sampling. However, there is growing interest worldwide in the potential for new technologies to assist in data collection for assessment of dietary intake. Published literature reviews have identified the potential of new technologies to improve accuracy, reduce costs, and reduce respondent and researcher burden by automating data capture and the nutritional coding process. However, this is a fast-moving field of research, with technologies developing at a rapid pace, and an updated review of the potential application of new technologies in dietary assessment is warranted. This review was commissioned to identify the new technologies employed in dietary assessment and critically appraise their strengths and limitations in order to recommend which technologies, if any, might be suitable to develop for use in the NDNS RP and other UK population surveys. Objectives The overall aim of the project was to inform the Department of Health of the range of new technologies currently available and in development internationally that have potential to improve, complement or replace the methods used in the NDNS RP. The specific aims were: to generate an itinerary of new and emerging technologies that may be suitable; to systematically review the literature and critically appraise new technologies; and to recommend which of these new technologies, if any, would be appropriate for future use in the NDNS RP. To meet these aims, the project comprised two main facets, a literature review and qualitative research. Literature review data sources The literature review incorporated an extensive search of peer-reviewed and grey literature. The following sources were searched: Cochrane Database of Systematic Reviews (CDSR), Database of Abstracts of Reviews of Effectiveness (DARE), Web of Science Core Collection, Ovid MEDLINE, Ovid MEDLINE In-Process, Embase, NHS EED (Economic Evaluation Database), National Cancer Institute (NCI) Dietary Assessment Calibration/Validation Register, OpenGrey, EPPI Centre (TRoPHI), conference proceedings (ICDAM 2012, ISBNPA 2013, IEEE Xplore, Nutrition Society Irish Section and Summer Meetings 2014), recent issues of journals (Journal of Medical Internet Research, International Journal of Medical Informatics), grants registries (ClinicalTrials.gov, BBSRC, report), national surveys, and mobile phone application stores. In addition, hand-searching of relevant citations was performed. The search also included solicitation of key authors in the field to enquire about Making the best use of new technologies in the NDNS: a review 4 as-yet unpublished articles or reports, and a Bristol Online Survey publicised via social media, society newsletters and meetings. Literature review eligibility criteria Records were screened for eligibility using a three-stage process. Firstly, keyword searches identified obviously irrelevant titles. Secondly, titles and abstracts were screened against the eligibility criteria, following which full-text copies of papers were obtained and, in the third stage of screening, examined against the criteria. Two independent reviewers screened each record at each stage, with discrepancies referred to a third reviewer. Eligibility criteria were pre-specified and agreed by the project Steering Group (Section 1.6). Eligible records included: studies involving technologies, new to the NDNS RP, which can be used to automate or assist the collection of food consumption data and the coding of foods and portion sizes, currently available or beta versions, public domain or commercial; studies that address the development, features, or evaluation of new technology; technologies appropriate for the requirements of the NDNS RP in terms of nutritional analysis, with capacity to collect quantifiable consumption data at the food level; primary sources of information on a particular technology; and journal articles published since the year 2000 or grey literature available from 2011 onwards. The literature search was not limited to Englishlanguage publications, which are included in the itinerary, although data were not extracted from non-English studies. Literature synthesis and appraisal New technologies were categorised into eleven types of technology, and an itinerary was generated of tools falling under each category type. Due to the volume of eligible studies identified by the literature searches, data extraction was limited to the literature focussing on selected exemplar tools of five technology categories (web-based diet diary, web-based 24- hour recall, handheld devices (personal digital assistants and mobile phones), nonautomated cameras to complement traditional methods, and non-automated cameras to replace traditional methods). For each category, at least two exemplars were chosen, and all studies involving the exemplar were included in data extraction and synthesis. Exemplars were selected on the basis of breadth of evidence available, using pre-specified criteria agreed by the Steering Group. Data were extracted by a single reviewer and an evidence summary collated for each exemplar. A quality appraisal checklist was developed to assess the quality of validation studies. The checklist was piloted and applied by two independent reviewers. Studies were not excluded on the basis of quality, but study quality was taken into account when judging the strength of evidence. Due to the heterogeneity of the literature, meta-analyses were not performed. References were managed and screened using the EPPI Reviewer 4 systematic review software. EPPI Reviewer was also used to extract data

The monetary value of diets consumed by British adults: an exploration into sociodemographic differences in individual-level diet costs

OBJECTIVE: To describe the diet costs of adults in the National Diet and Nutrition Study (NDNS) and explore patterns in costs according to sociodemographic indicators. DESIGN: Cross-sectional diet diary information was matched to a database of food prices to assign a cost to each food or non-alcoholic beverage consumed. Daily diet costs were calculated, as well as costs per 10 MJ to improve comparability across differing energy requirements. Costs were compared between categories of sociodemographic variables and health behaviours. Multivariable regression assessed the effects of each variable on diet costs after adjustment. SETTING: The NDNS is a rolling dietary survey, recruiting a representative UK sample each year. The study features data from 2008-2010. SUBJECTS: Adults aged 19 years or over were included. The sample consisted of 1014 participants. RESULTS: The geometric mean daily diet cost was £2·89 (95 % CI £2·81, £2·96). Energy intake and daily diet cost were strongly associated. The mean energy-adjusted cost was £4·09 (95 % CI £4·01, £4·18) per 10 MJ. Energy-adjusted costs differed significantly between many subgroups, including by sex and household income. Multivariable regression found significant effects of sex, qualifications and occupation (costs per 10 MJ only), as well as equivalized household income, BMI and fruit and vegetable consumption on diet costs. CONCLUSIONS: This is the first time that monetary costs have been applied to the diets of NDNS adults. The findings suggest that certain subgroups in the UK - for example those on lower incomes - consume diets of lower monetary value. Observed differences were mostly in the directions anticipated

Validation of spatial microsimulation models: a proposal to adopt the Bland-Altman method

Model validation is recognised as crucial to microsimulation modelling. However, modellers encounter difficulty in choosing the most meaningful methods to compare simulated and actual values. The aim of this paper is to introduce and demonstrate a method employed widely in healthcare calibration studies. The ‘Bland-Altman plot’ consists of a plot of the difference between two methods against the mean (x-y versus x+y/2). A case study is presented to illustrate the method in practice for spatial microsimulation validation. The study features a deterministic combinatorial model (SimObesity), which modelled a synthetic population for England at the ward level using survey (ELSA) and Census 2011 data. Bland-Altman plots were generated, plotting simulated and census ward-level totals for each category of all constraint (benchmark) variables. Other validation metrics, such as R2, SEI, TAE and RMSE, are also presented for comparison.The case study demonstrates how the Bland-Altman plots are interpreted. The simple visualisation of both individual- (ward-) level difference and total variation gives the method an advantage over existing tools used in model validation. There still remains the question of what constitutes a valid or well-fitting model. However, the Bland Altman method can usefully be added to the canon of calibration methods

Development of a new assessment tool for cervical myelopathy using a Virtual Reality hand tracking sensor

Introduction: Myelopathy hand is a characteristic feature of cervical myelopathy. Since there are only a few scales to quantify the severity of cervical compressive myelopathy, there is a need to introduce a universal objective platform in outpatient settings. Virtual-Reality offers promise as a means of producing quantitative data regarding the function of the neural system in the hand. The Leap Motion Controller (LMC) is a small, USB Virtual-Reality motion tracking device that could be used for this purpose. The aim of this study was to assess the reliability and validity of the LMC in the 15-second hand grip-and- release (G-R) test, as compared against human inspection of an external digital camera recording. Moreover, to set a baseline measurement of the number of hand flexion-extension cycles and analyse the degree of motion in young healthy individuals, besides examining gender and dominant hand differences. Materials and Methods: Fifty healthy participants were asked to fully grip-and-release their dominant hand as rapidly as possible for three tests, each separated by a 10-minute rest, while wearing a non-metal wrist splint. The first two tests lasted for 15 seconds, and a digital camera was used to film the anterolateral side of the hand on the first test. The third test lasted for a maximum of three minutes or until subjects fatigued. Three assessors counted the frequency of G-R cycles, of the recorded videos, independently and in a blinded fashion. One assessor counted the frequency of grip-and-release cycles as well as the number of motions (magnitude of motion) from the data output of the LMC. The average mean frequency of the three video observers was compared with that measured by LMC using the Bland-Altman method. Test-retest reliability was examined by comparing the two 15-second tests. Results: The mean number of G-R cycles recorded in each 15-second test was: 47.8 + 6.4 (test 1, video observer); 47.7 + 6.5 (test 1, LMC); and 50.2 + 6.5 (test 2, LMC). Bland Altman indicated a bias of 0.15 cycles (95%CI 1⁄4 0.10-0.20), with upper and lower limits of agreemen

Can big data solve a big problem? Reporting the obesity data landscape in line with the Foresight obesity system map.

BACKGROUND: Obesity research at a population level is multifaceted and complex. This has been characterised in the UK by the Foresight obesity systems map, identifying over 100 variables, across seven domain areas which are thought to influence energy balance, and subsequent obesity. Availability of data to consider the whole obesity system is traditionally lacking. However, in an era of big data, new possibilities are emerging. Understanding what data are available can be the first challenge, followed by an inconsistency in data reporting to enable adequate use in the obesity context. In this study we map data sources against the Foresight obesity system map domains and nodes and develop a framework to report big data for obesity research. Opportunities and challenges associated with this new data approach to whole systems obesity research are discussed. METHODS: Expert opinion from the ESRC Strategic Network for Obesity was harnessed in order to develop a data source reporting framework for obesity research. The framework was then tested on a range of data sources. In order to assess availability of data sources relevant to obesity research, a data mapping exercise against the Foresight obesity systems map domains and nodes was carried out. RESULTS: A reporting framework was developed to recommend the reporting of key information in line with these headings: Background; Elements; Exemplars; Content; Ownership; Aggregation; Sharing; Temporality (BEE-COAST). The new BEE-COAST framework was successfully applied to eight exemplar data sources from the UK. 80% coverage of the Foresight obesity systems map is possible using a wide range of big data sources. The remaining 20% were primarily biological measurements often captured by more traditional laboratory based research. CONCLUSIONS: Big data offer great potential across many domains of obesity research and need to be leveraged in conjunction with traditional data for societal benefit and health promotion

Development of a new assessment tool for cervical myelopathy using hand-tracking sensor: Part 2: normative values

Purpose To set a baseline measurement of the number of hand flexion–extension cycles and analyse the degree of motion in young healthy individuals, measured by leap motion controller (LMC), besides describing gender and dominant hand differences. Methods Fifty healthy participants were asked to fully grip-and-release their dominant hand as rapidly as possible for a maximum of 3 min or until subjects fatigued, while wearing a non-metal wrist splint. Participants also performed a 15-s grip-and-release test. An assessor blindly counted the frequency of grip-and-release cycles and magnitude of motion from the LMC data. Results The mean number of the 15-s G–R cycles recorded by LMC was: 47.7 ± 6.5 (test 1, LMC); and 50.2 ± 6.5 (test 2, LMC). In the 3-min test, the total number of hand flexion–extension cycles and the degree of motion decreased as the person fatigued. However, the decline in frequency preceded that of motion’s magnitude. The mean frequency of cycles per 10-s interval decreased from 35.4 to 26.6 over the 3 min. Participants reached fatigue from 59.38 s; 43 participants were able to complete the 3-min test. Conclusions Normative values of the frequency of cycles and extent of motion for young healthy individuals, aged 18–35 years, are provided. Future work is needed to establish values in a wider age range and in a clinical setting

Timing of high-dose methotrexate CNS prophylaxis in DLBCL: an analysis of toxicity and impact on R-CHOP delivery

High-dose methotrexate (HD-MTX) is increasingly used as prophylaxis for patients with diffuse large B-cell lymphoma (DLBCL) at high risk of central nervous system (CNS) relapse. However, there is limited evidence to guide whether to intercalate HD-MTX (i-HD-MTX) between R-CHOP-21 (rituximab, cyclophosphamide, doxorubicin, vincristine, and prednisolone given at 21-day intervals) or to give it at the end of treatment (EOT) with R-CHOP-21. We conducted a retrospective, multicenter analysis of 334 patients with DLBCL who received CNS prophylaxis with i-HD-MTX (n = 204) or EOT HD-MTX (n = 130). Primary end points were R-CHOP delay rates and HD-MTX toxicity. Secondary end points were CNS relapse rate, progression-free survival, and overall survival. The EOT group had more patients with a high CNS international prognostic index (58% vs 39%; P < .001) and more concurrent intrathecal prophylaxis (56% vs 34%; P < .001). Of the 409 cycles of i-HD-MTX given, 82 (20%) were associated with a delay of next R-CHOP (median, 7 days). Delays were significantly increased when i-HD-MTX was given after day 9 post–R-CHOP (26% vs 16%; P = .01). On multivariable analysis, i-HD-MTX was independently associated with increased R-CHOP delays. Increased mucositis, febrile neutropenia, and longer median inpatient stay were recorded with i-HD-MTX delivery. Three-year cumulative CNS relapse incidence was 5.9%, with no differences between groups. There was no difference in survival between groups. We report increased toxicity and R-CHOP delay with i-HD-MTX compared with EOT delivery but no difference in CNS relapse or survival. Decisions on HD-MTX timing should be individualized and, where i-HD-MTX is favored, we recommend scheduling before day 10 of R-CHOP cycles

Development of a new assessment tool for cervical myelopathy using hand-tracking sensor: Part 1: validity and reliability

Purpose To assess the reliability and validity of a hand motion sensor, Leap Motion Controller (LMC), in the 15-s hand grip-and-release test, as compared against human inspection of an external digital camera recording. Methods Fifty healthy participants were asked to fully grip-and-release their dominant hand as rapidly as possible for two trials with a 10-min rest in-between, while wearing a non-metal wrist splint. Each test lasted for 15 s, and a digital camera was used to film the anterolateral side of the hand on the first test. Three assessors counted the frequency of grip-and-release (G-R) cycles independently and in a blinded fashion. The average mean of the three was compared with that measured by LMC using the Bland–Altman method. Test–retest reliability was examined by comparing the two 15-s tests. Results The mean number of G-R cycles recorded was: 47.8 ± 6.4 (test 1, video observer); 47.7 ± 6.5 (test 1, LMC); and 50.2 ± 6.5 (test 2, LMC). Bland–Altman indicated good agreement, with a low bias (0.15 cycles) and narrow limits of agreement. The ICC showed high inter-rater agreement and the coefficient of repeatability for the number of cycles was ±5.393, with a mean bias of 3.63. Conclusions LMC appears to be valid and reliable in the 15-s grip-and-release test. This serves as a first step towards the development of an objective myelopathy assessment device and platform for the assessment of neuromotor hand function in general. Further assessment in a clinical setting and to gauge healthy benchmark values is warranted