Non-Standard Errors

In statistics, samples are drawn from a population in a data-generating process (DGP). Standard errors measure the uncertainty in estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence-generating process (EGP). We claim that EGP variation across researchers adds uncertainty: Non-standard errors (NSEs). We study NSEs by letting 164 teams test the same hypotheses on the same data. NSEs turn out to be sizable, but smaller for better reproducible or higher rated research. Adding peer-review stages reduces NSEs. We further find that this type of uncertainty is underestimated by participants

Internet Appendix to: Black, Desai, Litvak, Yoo, and Yu, The SEC’s Short-Sale Experiment: Evidence on Causal Channels and on the Importance of Specification Choice in Randomized and Natural Experiments

This Appendix contains additional results and discussion for Black, Desai, Litvak, Yoo, and Yu, The SEC’s Short-sale Experiment: Evidence on Causal Channels and the Importance of Specification Choice in Randomized and Natural Experiments (2022). It contains sample details; details on differences between our specification and the best-match specifications; additional results for the best-match specifications; results for step-by-step moves from our specification to each best-match specification; technical replication for FHK and HHZ; robustness assessments for the FHK and HHZ exact samples and specifications; and analysis of the additional specifications and datasets publicly posted by FHK in response to an earlier draft of the paper. We confirm technical replication for the FHK accruals result and the HHZ result for audit fees, for their exact samples and specification, but show that both results are fragile. We also respond in detail to the FHK reply to an earlier draft of this paper.The underling paper is available at https://ssrn.com/abstract=3657196. Our pre-specified analysis plan is available at https://ssrn.com/abstract=3415529

The SEC\u27s Short-Sale Experiment: Evidence on Causal Channels and on the Importance of Specification Choice in Randomized and Natural Experiments

During 2005-2007, the Securities and Exchange Commission (SEC) conducted a randomized trial in which it removed short-sale restrictions from one-third of the Russell 3000 firms (pilot firms). Early studies found modest market microstructure effects of removing the restrictions but no effect on short interest, pilot firm returns, or price efficiency. More recently, many studies have attributed a wide range of indirect outcomes to this experiment, mostly without assessing the causal channels for those outcomes. We examine the three most often cited causal channels for these indirect effects: short interest, share returns and managerial fear. We find no evidence to support any of these channels. We then reexamine the principal findings in four recent studies using a pre-specified research design (similar across the four reexaminations) and a larger sample that closely matches the actual experiment, and find no support for the reported outcomes in any of these papers. We then switch to best-match specifications that closely match the samples and specifications reported in each paper, and still find only minimal support for the reported results. For two papers, we have the authors’ original data and code; the reported results technically replicate but are highly fragile. Our findings highlight the importance of confirming a causal channel in randomized trials or natural experiments as well as the importance of sample selection and other aspects of specification choice for the statistical significance of reported results.The Internet Appendix is available at https://ssrn.com/abstract=3657200.Our pre-specified analysis plan is available at https://ssrn.com/abstract=3415529

The SEC\u27s Short-Sale Experiment: Evidence on Causal Channels and on the Importance of Specification Choice in Randomized and Natural Experiments

During 2005-2007, the Securities and Exchange Commission (SEC) conducted a randomized trial in which it removed short-sale restrictions from one-third of the Russell 3000 firms (pilot firms). Early studies found modest market microstructure effects of removing the restrictions but no effect on short interest, pilot firm returns, or price efficiency. More recently, many studies have attributed a wide range of indirect outcomes to this experiment, mostly without assessing the causal channels for those outcomes. We examine the three most often cited causal channels for these indirect effects: short interest, share returns and managerial fear. We find no evidence to support any of these channels. We then reexamine the principal findings in four recent studies using a pre-specified research design (similar across the four reexaminations) and a larger sample that closely matches the actual experiment, and find no support for the reported outcomes in any of these papers. We then switch to best-match specifications that closely match the samples and specifications reported in each paper, and still find only minimal support for the reported results. For two papers, we have the authors’ original data and code; the reported results technically replicate but are highly fragile. Our findings highlight the importance of confirming a causal channel in randomized trials or natural experiments as well as the importance of sample selection and other aspects of specification choice for the statistical significance of reported results.The Internet Appendix is available at https://ssrn.com/abstract=3657200.Our pre-specified analysis plan is available at https://ssrn.com/abstract=3415529

Internet Appendix to: Black, Desai, Litvak, Yoo, and Yu, The SEC’s Short-Sale Experiment: Evidence on Causal Channels and on the Importance of Specification Choice in Randomized and Natural Experiments

This Appendix contains additional results and discussion for Black, Desai, Litvak, Yoo, and Yu, The SEC’s Short-sale Experiment: Evidence on Causal Channels and the Importance of Specification Choice in Randomized and Natural Experiments (2022). It contains sample details; details on differences between our specification and the best-match specifications; additional results for the best-match specifications; results for step-by-step moves from our specification to each best-match specification; technical replication for FHK and HHZ; robustness assessments for the FHK and HHZ exact samples and specifications; and analysis of the additional specifications and datasets publicly posted by FHK in response to an earlier draft of the paper. We confirm technical replication for the FHK accruals result and the HHZ result for audit fees, for their exact samples and specification, but show that both results are fragile. We also respond in detail to the FHK reply to an earlier draft of this paper.The underling paper is available at https://ssrn.com/abstract=3657196. Our pre-specified analysis plan is available at https://ssrn.com/abstract=3415529

Online Appendix for Black, Desai, Litvak, Yoo, and Yu, Specification Choice in Randomized and Natural Experiments: Lessons from the Regulation SHO Experiment

This Online Appendix contains additional results and discussion for Black, Desai, Litvak, Yoo, and Yu, Specification Search in Randomized and Natural Experiments: Lessons from the Regulation SHO Experiment (2020). In particular, it contains further analysis of the various specifications and datasets publicly posted by FHK in response to an earlier draft of the paper, details on sample and other differences between our specification and the best-match specification, and step-by-step results from moving from our specification to the best-match specification for each paper

Specification Choice in Randomized and Natural Experiments: Lessons from the Regulation SHO Experiment

During 2005-2007, the SEC conducted a randomized trial in which it removed short-sale restrictions from one-third of the Russell 3000 firms. Early studies found modest microstructure-related effects of removing the restrictions but no effect on short interest or share prices. More recently, however, many studies have attributed a wide range of substantive indirect outcomes to this experiment. We revisit the principal findings in four recent studies in major journals, using a sample that closely matches the actual experiment and a pre-specified research design, and find no support for any of the reported results. If we instead match their specifications as best we can based on the published descriptions, we still obtain quite different results; only two of 13 outcomes are statistically significant, barely so, and even those results are tenuous. For two papers, we have the authors’ original data and code. We can technically replicate two (different) results, but those results are highly sensitive to specification. Our findings have implications for the robustness of other studies finding indirect effects of the short-sale experiment. More broadly, they have implications for the credibility of “causal” research designs which rely on randomization or on natural experiments. Researchers retain extensive discretion over both the sample and model specification, even for a true randomized experiment. The choices in these studies produced significant results, when other reasonable choices would not

Online Appendix for Black, Desai, Litvak, Yoo, and Yu, Specification Choice in Randomized and Natural Experiments: Lessons from the Regulation SHO Experiment

This Online Appendix contains additional results and discussion for Black, Desai, Litvak, Yoo, and Yu, Specification Search in Randomized and Natural Experiments: Lessons from the Regulation SHO Experiment (2020). In particular, it contains further analysis of the various specifications and datasets publicly posted by FHK in response to an earlier draft of the paper, details on sample and other differences between our specification and the best-match specification, and step-by-step results from moving from our specification to the best-match specification for each paper

Specification Choice in Randomized and Natural Experiments: Lessons from the Regulation SHO Experiment

During 2005-2007, the SEC conducted a randomized trial in which it removed short-sale restrictions from one-third of the Russell 3000 firms. Early studies found modest microstructure-related effects of removing the restrictions but no effect on short interest or share prices. More recently, however, many studies have attributed a wide range of substantive indirect outcomes to this experiment. We revisit the principal findings in four recent studies in major journals, using a sample that closely matches the actual experiment and a pre-specified research design, and find no support for any of the reported results. If we instead match their specifications as best we can based on the published descriptions, we still obtain quite different results; only two of 13 outcomes are statistically significant, barely so, and even those results are tenuous. For two papers, we have the authors’ original data and code. We can technically replicate two (different) results, but those results are highly sensitive to specification. Our findings have implications for the robustness of other studies finding indirect effects of the short-sale experiment. More broadly, they have implications for the credibility of “causal” research designs which rely on randomization or on natural experiments. Researchers retain extensive discretion over both the sample and model specification, even for a true randomized experiment. The choices in these studies produced significant results, when other reasonable choices would not