A validation framework for neuroimaging software: The case of population receptive fields

Published: June 25, 2020Neuroimaging software methods are complex, making it a near certainty that some implementations will contain errors. Modern computational techniques (i.e., public code and data repositories, continuous integration, containerization) enable the reproducibility of the analyses and reduce coding errors, but they do not guarantee the scientific validity of the results. It is difficult, nay impossible, for researchers to check the accuracy of software by reading the source code; ground truth test datasets are needed. Computational reproducibility means providing software so that for the same input anyone obtains the same result, right or wrong. Computational validity means obtaining the right result for the ground-truth test data. We describe a framework for validating and sharing software implementations, and we illustrate its usage with an example application: population receptive field (pRF) methods for functional MRI data. The framework is composed of three main components implemented with containerization methods to guarantee computational reproducibility. In our example pRF application, those components are: (1) synthesis of fMRI time series from ground-truth pRF parameters, (2) implementation of four public pRF analysis tools and standardization of inputs and outputs, and (3) report creation to compare the results with the ground truth parameters. The framework was useful in identifying realistic conditions that lead to imperfect parameter recovery in all four pRF implementations, that would remain undetected using classic validation methods. We provide means to mitigate these problems in future experiments. A computational validation framework supports scientific rigor and creativity, as opposed to the oft-repeated suggestion that investigators rely upon a few agreed upon packages. We hope that the framework will be helpful to validate other critical neuroimaging algorithms, as having a validation framework helps (1) developers to build new software, (2) research scientists to verify the software’s accuracy, and (3) reviewers to evaluate the methods used in publications and grants.Supported by a Marie Sklodowska-Curie (https://ec.europa.eu/programmes/horizon2020/ en/h2020-section/marie-sklodowska-curie-actions) grant to G.L.-U. (H2020-MSCA-IF-2017-795807- ReCiModel) and National Institutes of Health (https://www.nih.gov/) grants supporting N.C.B. and J.W. (EY027401, EY027964, MH111417). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript

A comparison of behavioural procedures for measuring Vernier acuity and grating acuity

I evaluated a novel 4/4 vision testing procedure recently implemented by our laboratory in a larger study to estimate the maturation ages of three visual functions. This rigorous procedure requires participants to locate a stimulus on four consecutive presentations before progressing to more difficult stimuli. I determined the time-efficiency of the procedure and whether children are affected differently than adults by the procedure, which would suggest that maturation age estimates from the larger study are inaccurate. Fifty-five adults and 52 children were tested on grating acuity and vernier acuity using the Teller acuity cards (TAC) and vernier acuity cards, respectively. I hypothesized that the 4/4 procedure would agree with the widely-used staircase and TAC procedures for both visual functions and across age groups, and that the 4/4 procedure would be more time-efficient than the staircase procedure. The 4/4 procedure showed strong to acceptable agreement with the staircase and TAC procedures for grating acuity, while levels of agreement for vernier acuity were poorer due to the misalignment step sizes. For both visual functions, levels of agreement were stable across age indicating that the measurement of children’s acuity was not affected differently than the measurement of adults’ acuity when tested with the 4/4 procedure. Although for both visual functions, the 4/4 procedure had longer completion times than the TAC procedure, it was more time-efficient than the staircase procedure. Thus, this procedure is sufficiently accurate and practical to measure visual maturation

Brain Function in Early Childhood: Individual Differences in Age and Attentive Traits

Children, like adults, are unique individuals with complex interwoven relationships between brain function, behaviour, and phenotypic traits, which further interact with rapid developmental processes. A nuanced description of variability between children will add to our knowledge of how they think and behave, and potentially advance the development of personalized early interventions. With functional magnetic resonance imaging (fMRI), we have gained insight into brain responses – however, due to practical considerations, we have been unable to render a complete understanding of brain-behaviour relationships in young children. The use of naturalistic stimuli in fMRI studies has increased the ecological validity and the retention of developmental neuroimaging data. In this dissertation, I sought to explore the relationships between age, attentive traits, and inter-individual variability of brain function in young children in naturalistic paradigms. I conducted a scoping review to synthesize the current and historical task- and naturalistic-fMRI literature on the development of visual processing in the brain, through the lens of two influential theories: the interactive specialization and maturational frameworks. I found that while there is generally a consensus of progressive development of visual brain function throughout childhood, there is not enough evidence to fully support other aspects of these theories. I also conducted two experiments, using naturalistic fMRI and an analysis technique called inter-subject correlation (ISC), which quantifies the spatiotemporal similarity of brain activity between individuals, to explore how age and attentive traits affect inter-individual variability of brain function in children aged 4-8 years. I found that children’s brain responses to movies “homogenized” with increasing age in our sample, with greater variability seen in the younger children. Further, both inattention and hyperactivity were associated with ISC in the sample, though the relationships with these traits were different in widespread regions of the brain. Together, my research advances our understanding of functional brain responses in children and underscores the importance of an individual differences approach to developmental neuroimaging