Validation of spatial microsimulation models: a proposal to adopt the Bland-Altman method

Model validation is recognised as crucial to microsimulation modelling. However, modellers encounter difficulty in choosing the most meaningful methods to compare simulated and actual values. The aim of this paper is to introduce and demonstrate a method employed widely in healthcare calibration studies. The ‘Bland-Altman plot’ consists of a plot of the difference between two methods against the mean (x-y versus x+y/2). A case study is presented to illustrate the method in practice for spatial microsimulation validation. The study features a deterministic combinatorial model (SimObesity), which modelled a synthetic population for England at the ward level using survey (ELSA) and Census 2011 data. Bland-Altman plots were generated, plotting simulated and census ward-level totals for each category of all constraint (benchmark) variables. Other validation metrics, such as R2, SEI, TAE and RMSE, are also presented for comparison. The case study demonstrates how the Bland-Altman plots are interpreted. The simple visualisation of both individual- (ward-) level difference and total variation gives the method an advantage over existing tools used in model validation. There still remains the question of what constitutes a valid or well-fitting model. However, the Bland Altman method can usefully be added to the canon of calibration methods

Validation of spatial microsimulation models: a proposal to adopt the Bland-Altman method

Model validation is recognised as crucial to microsimulation modelling. However, modellers encounter difficulty in choosing the most meaningful methods to compare simulated and actual values. The aim of this paper is to introduce and demonstrate a method employed widely in healthcare calibration studies. The ‘Bland-Altman plot’ consists of a plot of the difference between two methods against the mean (x-y versus x+y/2). A case study is presented to illustrate the method in practice for spatial microsimulation validation. The study features a deterministic combinatorial model (SimObesity), which modelled a synthetic population for England at the ward level using survey (ELSA) and Census 2011 data. Bland-Altman plots were generated, plotting simulated and census ward-level totals for each category of all constraint (benchmark) variables. Other validation metrics, such as R2, SEI, TAE and RMSE, are also presented for comparison. The case study demonstrates how the Bland-Altman plots are interpreted. The simple visualisation of both individual- (ward-) level difference and total variation gives the method an advantage over existing tools used in model validation. There still remains the question of what constitutes a valid or well-fitting model. However, the Bland Altman method can usefully be added to the canon of calibration methods

Validation of spatial microsimulation models: a proposal to adopt the Bland-Altman method

Model validation is recognised as crucial to microsimulation modelling. However, modellers encounter difficulty in choosing the most meaningful methods to compare simulated and actual values. The aim of this paper is to introduce and demonstrate a method employed widely in healthcare calibration studies. The ‘Bland-Altman plot’ consists of a plot of the difference between two methods against the mean (x-y versus x+y/2). A case study is presented to illustrate the method in practice for spatial microsimulation validation. The study features a deterministic combinatorial model (SimObesity), which modelled a synthetic population for England at the ward level using survey (ELSA) and Census 2011 data. Bland-Altman plots were generated, plotting simulated and census ward-level totals for each category of all constraint (benchmark) variables. Other validation metrics, such as R2, SEI, TAE and RMSE, are also presented for comparison.The case study demonstrates how the Bland-Altman plots are interpreted. The simple visualisation of both individual- (ward-) level difference and total variation gives the method an advantage over existing tools used in model validation. There still remains the question of what constitutes a valid or well-fitting model. However, the Bland Altman method can usefully be added to the canon of calibration methods

Comparing methods of detecting and segmenting unruptured intracranial aneurysms on TOF-MRAS: The ADAM challenge

Accurate detection and quantification of unruptured intracranial aneurysms (UIAs) is important for rupture risk assessment and to allow an informed treatment decision to be made. Currently, 2D manual measures used to assess UIAs on Time-of-Flight magnetic resonance angiographies (TOF-MRAs) lack 3D information and there is substantial inter-observer variability for both aneurysm detection and assessment of aneurysm size and growth. 3D measures could be helpful to improve aneurysm detection and quantification but are time-consuming and would therefore benefit from a reliable automatic UIA detection and segmentation method. The Aneurysm Detection and segMentation (ADAM) challenge was organised in which methods for automatic UIA detection and segmentation were developed and submitted to be evaluated on a diverse clinical TOF-MRA dataset. A training set (113 cases with a total of 129 UIAs) was released, each case including a TOF-MRA, a structural MR image (T1, T2 or FLAIR), annotation of any present UIA(s) and the centre voxel of the UIA(s). A test set of 141 cases (with 153 UIAs) was used for evaluation. Two tasks were proposed: (1) detection and (2) segmentation of UIAs on TOF-MRAs. Teams developed and submitted containerised methods to be evaluated on the test set. Task 1 was evaluated using metrics of sensitivity and false positive count. Task 2 was evaluated using dice similarity coefficient, modified hausdorff distance (95th percentile) and volumetric similarity. For each task, a ranking was made based on the average of the metrics. In total, eleven teams participated in task 1 and nine of those teams participated in task 2. Task 1 was won by a method specifically designed for the detection task (i.e. not participating in task 2). Based on segmentation metrics, the top two methods for task 2 performed statistically significantly better than all other methods. The detection performance of the top-ranking methods was comparable to visual inspection for larger aneurysms. Segmentation performance of the top ranking method, after selection of true UIAs, was similar to interobserver performance. The ADAM challenge remains open for future submissions and improved submissions, with a live leaderboard to provide benchmarking for method developments at https://adam.isi.uu.nl/

Relationship between 3D Morphologic Change and 2D and 3D Growth of Unruptured Intracranial Aneurysms

BACKGROUND AND PURPOSE: Untreated unruptured intracranial aneurysms are usually followed radiologically to detect aneurysm growth, which is associated with increased rupture risk. The ideal aneurysm size cutoff for defining growth remains unclear and also whether change in morphology should be part of the definition. We investigated the relationship between change in aneurysm size and 3D quantified morphologic changes during follow-up. MATERIALS AND METHODS: We performed 3D morphology measurements of unruptured intracranial aneurysms on baseline and follow-up TOF-MRAs. Morphology measurements included surface area, compactness, elongation, flatness, sphericity, shape index, and curvedness. We investigated the relation between morphologic change between baseline and follow-up scans and unruptured intracranial aneurysm growth, with 2D and 3D growth defined as a continuous variable (correlation statistics) and a categoric variable (t test statistics). Categoric growth was defined as ≥1-mm increase in 2D length or width. We assessed unruptured intracranial aneurysms that changed in morphology and the proportion of growing and nongrowing unruptured intracranial aneurysms with statistically significant morphologic change. RESULTS: We included 113 patients with 127 unruptured intracranial aneurysms. Continuous growth of unruptured intracranial aneurysms was related to an increase in surface area and flatness and a decrease in the shape index and curvedness. In 15 growing unruptured intracranial aneurysms (12%), curvedness changed significantly compared with nongrowing unruptured intracranial aneurysms. Of the 112 nongrowing unruptured intracranial aneurysms, 10 (9%) changed significantly in morphology (flatness, shape index, and curvedness). CONCLUSIONS: Growing unruptured intracranial aneurysms show morphologic change. However, nearly 10% of nongrowing unruptured intracranial aneurysms change in morphology, suggesting that they could be unstable. Future studies should investigate the best growth definition including morphologic change and size to predict aneurysm rupture

Hemodynamic Parameters in the Parent Arteries of Unruptured Intracranial Aneurysms Depend on Aneurysm Size and Are Different Compared to Contralateral Arteries: A 7 Tesla 4D Flow MRI Study

Background: Different Circle of Willis (CoW) variants have variable prevalences of aneurysm development, but the hemodynamic variation along the CoW and its relation to presence and size of unruptured intracranial aneurysms (UIAs) are not well known. Purpose: Gain insight into hemodynamic imaging markers of the CoW for UIA development by comparing these outcomes to the corresponding contralateral artery without an UIA using 4D flow magnetic resonance imaging (MRI). Study Type: Retrospective, cross-sectional study. Subjects: Thirty-eight patients with an UIA, whereby 27 were women and a mean age of 62 years old. Field Strength/Sequence: Four-dimensional phase-contrast (PC) MRI with a 3D time-resolved velocity encoded gradient echo sequence at 7 T. Assessment: Hemodynamic parameters (blood flow, velocity pulsatility index [vPI], mean velocity, distensibility, and wall shear stress [peak systolic (WSSMAX), and time-averaged (WSSMEAN)]) in the parent artery of the UIA were compared to the corresponding contralateral artery without an UIA and were related to UIA size. Statistical Tests: Paired t-tests and Pearson Correlation tests. The threshold for statistical significance was P < 0.05 (two-tailed). Results: Blood flow, mean velocity, WSSMAX, and WSSMEAN were significantly higher, while vPI was lower, in the parent artery relative to contralateral artery. The WSSMAX of the parent artery significantly increased linearly while the WSSMEAN decreased linearly with increasing UIA size. Conclusions: Hemodynamic parameters and WSS differ between parent vessels of UIAs and corresponding contralateral vessels. WSS correlates with UIA size, supporting a potential hemodynamic role in aneurysm pathology. Level of Evidence: 2. Technical Efficacy: Stage 2

What is the cost of a healthy diet? Using diet data from the UK Women's Cohort Study

Background A healthy diet is important to promote health and well-being while preventing chronic disease. However, the monetary cost of consuming such a diet can be a perceived barrier. This study will investigate the cost of consuming a range of dietary patterns.Methods A cross-sectional analysis, where cost of diet was assigned to dietary intakes recorded using a Food Frequency Questionnaire. A mean daily diet cost was calculated for seven data-driven dietary patterns. These dietary patterns were given a healthiness score according to how well they comply with the UK Department of Health's Eatwell Plate guidelines. This study involved �+35 000 women recruited in the 1990s into the UK Women's Cohort Study.Results A significant positive association was observed between diet cost and healthiness of the diet (p for trend >0.001). The healthiest dietary pattern was double the price of the least healthy, -�6.63/day and -�3.29/day, respectively. Dietary diversity, described by the patterns, was also shown to be associated with increased cost. Those with higher education and a professional or managerial occupation were more likely to consume a healthier diet.Conclusions A healthy diet is more expensive to the consumer than a less healthy one. In order to promote health through diet and reduce potential inequalities in health, it seems sensible that healthier food choices should be made more accessible to al

Image Analysis and Deep Learning Techniques for the Detection and Characterisation of Unruptured Intracranial Aneurysms

This thesis presents image analysis techniques for the detection and characterisation of unruptured intracranial aneurysms (UIAs). Using such methods, robust and reliable growth and rupture assessment of UIAs can be made to aid in treatment decision making. The first half of this thesis consider automatic UIA detection and segmentation methods from angiographic scans (TOF-MRAs and CTAs). Chapter 2 describes the organisation of an international biomedical image analysis challenge. Teams submitted automatic UIA detection and segmentation methods, which were evaluated on a held-out test set. The winning detection method has since been developed into an open-source framework for medical image detection (nnDetection). The challenge remains as an important benchmark for UIA detection and segmentation methods. Chapter 3 describes an anomaly detection method using a variational autoencoder (VAE) trained on healthy TOF-MRAs. Reconstructed TOF-MRAs with diagnosed aneurysms had a lower Structural Similarity Index (SSIM), than TOF-MRAs of subjects with no aneurysms. Importantly, the results identified that structure and shape within the scans, and not just intensity, is important for UIA detection. The UIA detection method in Chapter 4 exploits the fact surface of a UIA is different from surrounding vessels. Vessels were segmented from TOF-MRAs and meshes were fitted to the surface. A mesh convolutional neural network was trained using the labelled vessel meshes, to detect UIAs. The modality-independent method has comparable performance for both TOF-MRAs and CTAs. Automatic UIA detection and segmentation described in Chapters 2-4 allow automatic 3D volume and morphology UIA measurements for potential growth and rupture risk assessment. Chapters 5 and 6 investigate UIA volume and morphology and their relationship to UIA growth assessment. In Chapter 5, 3D volume growth assessment was more reliable than 2D size, with smaller interobsever differences, and more consistency across location. The smallest detectable change for 2D growth was larger than the current growth definition (1mm), leading to ambiguity in the current definition. 3D UIA quantitative morphology measures, such as flatness and shape index, were introduced in Chapter 6 and their relationship with UIA growth investigated. Even in non-growing (stable) aneurysms, morphology changed, suggesting that non-growing aneurysms could still be unstable. Quantified morphologic change should be considered for UIA growth and rupture risk assessment. Finally, in Chapter 7, a UIA growth prediction model using a vessel surface mesh convolutional neural network was developed. The model had comparable performance to patient demographic growth prediction models (ELAPSS). Mesh/morphology and patient models could be combined to provide a complete UIA growth prediction model. In conclusion, this thesis provides complete UIA characterisation from TOF-MRAs using computer-aided techniques. The automatic UIA detection and segmentation allows reliable, automatic UIA volume and morphology measurements. Such measurements aid in UIA growth assessment and formal UIA growth definitions including these measures should be investigated. As the accuracy of automatic UIA segmentation methods and growth prediction models increase, these will become more commonplace in clinical workflows. This could result in a fully automatic UIA characterisation tool, determining UIA volume, morphology and growth prediction scores to aid in treatment decision and improve patient outcome

Image Analysis and Deep Learning Techniques for the Detection and Characterisation of Unruptured Intracranial Aneurysms

This thesis presents image analysis techniques for the detection and characterisation of unruptured intracranial aneurysms (UIAs). Using such methods, robust and reliable growth and rupture assessment of UIAs can be made to aid in treatment decision making. The first half of this thesis consider automatic UIA detection and segmentation methods from angiographic scans (TOF-MRAs and CTAs). Chapter 2 describes the organisation of an international biomedical image analysis challenge. Teams submitted automatic UIA detection and segmentation methods, which were evaluated on a held-out test set. The winning detection method has since been developed into an open-source framework for medical image detection (nnDetection). The challenge remains as an important benchmark for UIA detection and segmentation methods. Chapter 3 describes an anomaly detection method using a variational autoencoder (VAE) trained on healthy TOF-MRAs. Reconstructed TOF-MRAs with diagnosed aneurysms had a lower Structural Similarity Index (SSIM), than TOF-MRAs of subjects with no aneurysms. Importantly, the results identified that structure and shape within the scans, and not just intensity, is important for UIA detection. The UIA detection method in Chapter 4 exploits the fact surface of a UIA is different from surrounding vessels. Vessels were segmented from TOF-MRAs and meshes were fitted to the surface. A mesh convolutional neural network was trained using the labelled vessel meshes, to detect UIAs. The modality-independent method has comparable performance for both TOF-MRAs and CTAs. Automatic UIA detection and segmentation described in Chapters 2-4 allow automatic 3D volume and morphology UIA measurements for potential growth and rupture risk assessment. Chapters 5 and 6 investigate UIA volume and morphology and their relationship to UIA growth assessment. In Chapter 5, 3D volume growth assessment was more reliable than 2D size, with smaller interobsever differences, and more consistency across location. The smallest detectable change for 2D growth was larger than the current growth definition (1mm), leading to ambiguity in the current definition. 3D UIA quantitative morphology measures, such as flatness and shape index, were introduced in Chapter 6 and their relationship with UIA growth investigated. Even in non-growing (stable) aneurysms, morphology changed, suggesting that non-growing aneurysms could still be unstable. Quantified morphologic change should be considered for UIA growth and rupture risk assessment. Finally, in Chapter 7, a UIA growth prediction model using a vessel surface mesh convolutional neural network was developed. The model had comparable performance to patient demographic growth prediction models (ELAPSS). Mesh/morphology and patient models could be combined to provide a complete UIA growth prediction model. In conclusion, this thesis provides complete UIA characterisation from TOF-MRAs using computer-aided techniques. The automatic UIA detection and segmentation allows reliable, automatic UIA volume and morphology measurements. Such measurements aid in UIA growth assessment and formal UIA growth definitions including these measures should be investigated. As the accuracy of automatic UIA segmentation methods and growth prediction models increase, these will become more commonplace in clinical workflows. This could result in a fully automatic UIA characterisation tool, determining UIA volume, morphology and growth prediction scores to aid in treatment decision and improve patient outcome

Validation of spatial microsimulation models: a proposal to adopt the Bland-Altman method

Model validation is recognised as crucial to microsimulation modelling. However, modellers encounter difficulty in choosing the most meaningful methods to compare simulated and actual values. The aim of this paper is to introduce and demonstrate a method employed widely in healthcare calibration studies. The ‘Bland-Altman plot’ consists of a plot of the difference between two methods against the mean (x-y versus x+y/2). A case study is presented to illustrate the method in practice for spatial microsimulation validation. The study features a deterministic combinatorial model (SimObesity), which modelled a synthetic population for England at the ward level using survey (ELSA) and Census 2011 data. Bland-Altman plots were generated, plotting simulated and census ward-level totals for each category of all constraint (benchmark) variables. Other validation metrics, such as R2, SEI, TAE and RMSE, are also presented for comparison. The case study demonstrates how the Bland-Altman plots are interpreted. The simple visualisation of both individual- (ward-) level difference and total variation gives the method an advantage over existing tools used in model validation. There still remains the question of what constitutes a valid or well-fitting model. However, the Bland Altman method can usefully be added to the canon of calibration methods