The problem of evaluating automated large-scale evidence aggregators

In the biomedical context, policy makers face a large amount of potentially discordant evidence from different sources. This prompts the question of how this evidence should be aggregated in the interests of best-informed policy recommendations. The starting point of our discussion is Hunter and Williams’ recent work on an automated aggregation method for medical evidence. Our negative claim is that it is far from clear what the relevant criteria for evaluating an evidence aggregator of this sort are. What is the appropriate balance between explicitly coded algorithms and implicit reasoning involved, for instance, in the packaging of input evidence? In short: What is the optimal degree of ‘automation’? On the positive side: We propose the ability to perform an adequate robustness analysis as the focal criterion, primarily because it directs efforts to what is most important, namely, the structure of the algorithm and the appropriate extent of automation. Moreover, where there are resource constraints on the aggregation process, one must also consider what balance between volume of evidence and accuracy in the treatment of individual evidence best facilitates inference. There is no prerogative to aggregate the total evidence available if this would in fact reduce overall accuracy

The diversity of model tuning practices in climate science

Many examples of calibration in climate science raise no alarms regarding model reliability. We examine one example and show that, in employing classical hypothesis testing, it involves calibrating a base model against data that are also used to confirm the model. This is counter to the ‘intuitive position’ (in favor of use novelty and against double counting). We argue, however, that aspects of the intuitive position are upheld by some methods, in particular, the general cross-validation method. How cross-validation relates to other prominent classical methods such as the Akaike information criterion and Bayesian information criterion is also discussed

Can free evidence be bad? Value of informationfor the imprecise probabilist

This paper considers a puzzling conflict between two positions that are each compelling: it is irrational for an agent to pay to avoid `free' evidence before making a decision, and rational agents may have imprecise beliefs and/or desires. Indeed, we show that Good's theorem concerning the invariable choice-worthiness of free evidence does not generalise to the imprecise realm, given the plausible existing decision theories for handling imprecision. A key ingredient in the analysis, and a potential source of controversy, is the general approach taken for resolving sequential decision problems { we make explicit what the key alternatives are and defend our own approach. Furthermore, we endorse a resolution of the aforementioned puzzle { we privilege decision theories that merely permit avoiding free evidence over decision theories for which avoiding free evidence is uniquely admissible. Finally, we situate this particular result about free evidence within the broader `dynamic-coherence' literature

Model tuning in engineering: uncovering the logic

In engineering, as in other scientific fields, researchers seek to confirm their models with real-world data. It is common practice to assess models in terms of the distance between the model outputs and the corresponding experimental observations. An important question that arises is whether the model should then be ‘tuned’, in the sense of estimating the values of free parameters to get a better fit with the data, and furthermore whether the tuned model can be confirmed with the same data used to tune it. This dual use of data is often disparagingly referred to as ‘double-counting’. Here, we analyse these issues, with reference to selected research articles in engineering (one mechanical and the other civil). Our example studies illustrate more and less controversial practices of model tuning and double-counting, both of which, we argue, can be shown to be legitimate within a Bayesian framework. The question nonetheless remains as to whether the implied scientific assumptions in each case are apt from the engineering point of view

The problem of evaluating automated large-scale evidence aggregators

In the biomedical context, policy makers face a large amount of potentially discordant evidence from different sources. This prompts the question of how this evidence should be aggregated in the interests of best-informed policy recommendations. The starting point of our discussion is Hunter and Williams’ recent work on an automated aggregation method for medical evidence. Our negative claim is that it is far from clear what the relevant criteria for evaluating an evidence aggregator of this sort are. What is the appropriate balance between explicitly coded algorithms and implicit reasoning involved, for instance, in the packaging of input evidence? In short: What is the optimal degree of ‘automation’? On the positive side: We propose the ability to perform an adequate robustness analysis (which depends on the nature of the input variables and parameters of the aggregator) as the focal criterion, primarily because it directs efforts to what is most important, namely, the structure of the algorithm and the appropriate extent of automation. Moreover, where there are resource constraints on the aggregation process, one must also consider what balance between volume of evide

Belief Revision for Growing Awareness

The Bayesian maxim for rational learning could be described as conservative change from one probabilistic belief or credence function to another in response to newinformation. Roughly: ‘Hold fixed any credences that are not directly affected by the learning experience.’ This is precisely articulated for the case when we learn that some proposition that we had previously entertained is indeed true (the rule of conditionalisation). But can this conservative-change maxim be extended to revising one’s credences in response to entertaining propositions or concepts of which one was previously unaware? The economists Karni and Vierø (2013, 2015) make a proposal in this spirit. Philosophers have adopted effectively the same rule: revision in response to growing awareness should not affect the relative probabilities of propositions in one’s ‘old’ epistemic state. The rule is compelling, but only under the assumptions that its advocates introduce. It is not a general requirement of rationality, or so we argue. We provide informal counterexamples. And we show that, when awareness grows, the boundary between one’s ‘old’ and ‘new’ epistemic commitments is blurred. Accordingly, there is no general notion of conservative change in this setting

Racial Discipline Disproportionality in Montessori and Traditional Public Schools: A Comparative Study Using the Relative Rate Index

Research from the past 40 years indicates that African American students are subjected to exclusionary discipline, including suspension and expulsion, at rates two to three times higher than their White peers (Children’s Defense Fund, 1975; Skiba, Michael, Nardo, & Peterson, 2002). Although this phenomenon has been studied extensively in traditional public schools, rates of racially disproportionate discipline in public Montessori schools have not been examined. The purpose of this study is to examine racial discipline disproportionality in Montessori public elementary schools as compared to traditional elementary schools. The Relative Rate Index (RRI) is used as a measure of racially disproportionate use of out-of-school suspensions (Tobin & Vincent, 2011). Suspension data from the Office of Civil Rights Data Collection was used to generate RRIs for Montessori and traditional elementary schools in a large urban district in the Southeast. While statistically significant levels of racial discipline disproportionality are found in both the Montessori and traditional schools, the effect is substantially less pronounced in Montessori settings. These findings suggest that Montessori schools are not immune to racially disproportionate discipline and should work to incorporate more culturally responsive classroom management techniques. Conversely, the lower levels of racially disproportionate discipline in the Montessori schools suggests that further study of discipline in Montessori environments may provide lessons for traditional schools to promote equitable discipline

Racial Discipline Disproportionality in Montessori and Traditional Public Schools: A Comparative Study Using the Relative Rate Index

Research from the past 40 years indicates that African American students are subjected to exclusionary discipline, including suspension and expulsion, at rates two to three times higher than their White peers (Children’s Defense Fund, 1975; Skiba, Michael, Nardo, & Peterson, 2002). Although this phenomenon has been studied extensively in traditional public schools, rates of racially disproportionate discipline in public Montessori schools have not been examined. The purpose of this study is to examine racial discipline disproportionality in Montessori public elementary schools as compared to traditional elementary schools. The Relative Rate Index (RRI) is used as a measure of racially disproportionate use of out-of-school suspensions (Tobin & Vincent, 2011). Suspension data from the Office of Civil Rights Data Collection was used to generate RRIs for Montessori and traditional elementary schools in a large urban district in the Southeast. While statistically significant levels of racial discipline disproportionality are found in both the Montessori and traditional schools, the effect is substantially less pronounced in Montessori settings. These findings suggest that Montessori schools are not immune to racially disproportionate discipline and should work to incorporate more culturally responsive classroom management techniques. Conversely, the lower levels of racially disproportionate discipline in the Montessori schools suggests that further study of discipline in Montessori environments may provide lessons for traditional schools to promote equitable discipline