The question of when to stop an unsuccessful experiment can be difficult to answer from an individual perspective. To help to guide these decisions, we turn to the social epistemology of science and investigate knowledge inquisition within a group. We focused on the expensive and lengthy experiments in high energy physics, which were suitable for citation-based analysis because of the relatively quick and reliable consensus about the importance of results in the field. In particular, we tested whether the time spent on a scientific project correlates with the project output. Our results are based on data from the high energy physics laboratory Fermilab. They point out that there is an epistemic saturation point in experimenting, after which the likelihood of obtaining major results drops. With time the number of less significant publications does increase, but highly cited ones do not get published. Since many projects continue to run after the epistemic saturation point, it becomes clearer that decisions made about continuing them are not always rational

Perovic, Slobodan

Radovanović, Sandro

Sikimić, Vlasta

PhilPapers

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/336364391When Should We Stop Investing in a Scientific Project? The Halting Problemin Experimental PhysicsConference Paper · January 2018CITATIONS0READS273 authors, including:Some of the authors of this publication are also working on these related projects:Predicting patient’s future health state: Development and deployment of fast, effective, and interpretable algorithms for healthcare View projectModular Proof Systems View projectVlasta SikimićUniversity of Tuebingen10 PUBLICATIONS   83 CITATIONS   SEE PROFILESandro RadovanovicUniversity of Belgrade34 PUBLICATIONS   241 CITATIONS   SEE PROFILEAll content following this page was uploaded by Vlasta Sikimić on 09 October 2019.The user has requested enhancement of the downloaded file.SOCIAL PSYCHOLOGY  105  When Should We Stop Investing in a Scientific Project? The Halting Problem in Experimental Physics  Vlasta Sikimić (vlastasikimic@gmail.com) Institute for Philosophy of the Faculty of Philosophy, University of Belgrade  Sandro Radovanović (sandroradovanovic@gmail.com) Faculty of Organizational Sciences, University of Belgrade  Slobodan Perović (perovicslobodan@gmail.com) Faculty of Philosophy, University of Belgrade   Abstract The question of when to stop an unsuccessful experiment can be difficult to answer from an individual perspective. To help to guide these decisions, we turn to the social epistemology of science and investigate knowledge inquisition within a group. We focused on the expensive and lengthy experiments in high energy physics, which were suitable for citation-based analysis because of the relatively quick and reliable consensus about the importance of results in the field. In particular, we tested whether the time spent on a scientific project correlates with the project output. Our results are based on data from the high energy physics laboratory Fermilab. They point out that there is an epistemic saturation point in experimenting, after which the likelihood of obtaining major results drops. With time the number of less significant publications does increase, but highly cited ones do not get published. Since many projects continue to run after the epistemic saturation point, it becomes clearer that decisions made about continuing them are not always rational. Keywords: project duration; epistemic saturation, efficiency, scientific experimentation  Introduction Social epistemology of science investigates the knowledge acquisition within a scientific group. The epistemic efficiency of scientific knowledge acquisition across scientific institutions has been a topic of considerable interest in the social epistemology of science and science policy studies (Kitcher 1990, Milojević 2014, Olson et al. 2007, Zollman 2007). We focus on the time point of epistemic saturation in contemporary experimental physics. In high energy physics, a large number of researchers are appointed to work on the same project with instruments which usage requires substantial funding. This makes experiments in contemporary physics particularly interesting from the perspective of organizing research teams and optimizing resources invested in them.  One of the key resources invested in a project is the time spent on it. As a halting problem in informatics demarks some algorithms as never-stopping, determining the stopping rules governing scientific research has been broadly investigated                                                           4 The full list of Fermilab experiments can be found on the following link: <https://ccd.fnal.gov/techpubs/fermilab-reports-proposal.html>. in the philosophy of science literature (e.g. Rogger 2012, Steel 2013). These rules should signal when a scientist should stop gathering data and start analyzing them. Fermilab is one of the biggest and most important high energy physics laboratories. Moreover, it has a thoroughly assembled online archive of conducted experiments.4 Using data envelopment analysis on data from Fermilab, Perović et al. (2016) showed that longer experiments are inefficient in comparison to shorter ones. A new extended and predictive study showed that projects lasting longer than 500 days will tend to be inefficient (Sikimić et al. submitted). This means that there is an epistemic saturation point after which further time investment in a project will not be fruitful. In the present paper, we go a step further and investigated correlations between time and productive output of the project – results. Method We used data from 49 experiments run in Fermilab in the period between 1975 and 2003. The time distance allows for the accurate assessment of the project impact. All experiments were analyzed qualitatively. Linked experiments, precision measurements and calibrations of instruments were excluded from the analysis since their impact is not comparable to the impact of a single exploratory experiment. The data we used were the project duration, the number of researchers, the number of teams, and the number of publications categorized into six ranks:  1: famous papers (250 + citations); 2: very well-known papers (100–249 citations); 3: well-known papers (50–99 citations); 4: known papers (10–49 citations); 5: less known papers (1–9 citations); 6: unknown papers (0 citations). This categorization is provided by the physicists themselves and is given on the HEP Inspire platform (http://inspirehep.net). Self-citations were excluded. When we talk about project duration, we refer to the time used for the result gathering, as publications might be delayed and not necessarily correlated with the project duration. Finally, the consensus on results in high energy physics is reached relatively quickly and remains stable for a long period of time Schulte (2000). This means SOCIAL PSYCHOLOGY  106  that citation analysis is a relevant measure of project success in the field. All this provides a good argument that the project results are mirrored by these six categories. By eliminating linked experiments, we focused our analysis on the experiments that started from scratch. We did not focus our analysis on the seniority ratio, because previous studies showed that the difference in the seniority ratio can only be considered as a secondary factor (Perović at al. 2016). The team size, on the other hand, played a more important role – smaller teams outperformed large ones.  Results The results show a weak negative correlation between time and very well-known papers, i.e. the papers classified as second in rank. The correlation coefficient is -0.083 (Figure 1).    Figure 1: Correlation between time in days and the number of very well-known papers.  Also, no significant correlation was found between time and number of famous papers i.e. papers of the highest rank. This was expectable since their total number was small (eight papers from all the 49 projects). Interestingly, all but one project resulting in several very well-known and famous publications lasted less than 3 years. On the other hand, with the time the number of publications with lower rank does increase. Yet the correlations are weak; the correlation coefficients of publications from rank 4 to 6 and time are between 0.223 and 0.280 (Figure 2). This means that over time scientists continue to put effort into their projects, even when important results are missing.  Another interesting point is that the number of researchers is positively correlated with the number of unknown and less known papers, but not with papers of the two highest ranks. This indicates that investing more human resources will bring some results, but not exceptional ones. As expected, the number of researchers is correlated with the number of teams (Figure 2).  Figure 2: Table of correlations.  There is a strong positive correlation between the number of articles with higher rank among each other, while their correlation with the number of less known articles is negative (Figure 2). This indicates that there is a relatively clear-cut between fruitful and futile projects.    Figure 3: Publications per project over time after the project started.  Finally, the longitudinal analysis (Figure 3) shows that the majority of well-known and famous papers are published 3-4 years after a project started. Note that some articles are published years after the experiments were conducted. Conclusions and Further Research The results indicate that simple extension of time spent on a project in high energy physics will not guarantee new and 00.10.20.30-2 3-4 5-6 7-8 9-10 11+Publications per projectYear after project started (if available, otherwise proposed date)Well-known papers Very well-known papersFamous papersSOCIAL PSYCHOLOGY  107  exciting results. These findings concord with previous research which argued that there is an epistemic saturation point in experimental physics (Perović et al. 2016, Sikimić et al. submitted). This realization might limit unnecessary and futile investments in already costly and lengthy experiments. The fact that research in experimental physics follows a strict inductive pattern governed by conservation laws (Schulte 2000), strengthens the argument in favour of the epistemic saturation point in the field. From the perspective of a scientist, the question of when to drop an unsuccessful experiment is particularly difficult because of the psychological principle of commitment and consistency. Once the commitment is made, i.e. once the research direction is established, one feels pressured to continue in the direction of her original believes in order to be consistent with them, even in situations when this is not reasonable (Cialdini 2001). This phenomenon leads to the irrational investment of resources (e.g. time, funding, human resources and equipment). Cleary, one can investigate internal reasons for unsuccessfulness of any experiment. However, using quantitative data we are able to predict the optimal duration of a successful experiment. Sometimes scientists keep investing in projects that are not giving results. During this struggle they do tend to publish less influential results, however, as shown, this type of investment will not reflect upon influential results. As further research, it would be interesting to see what lies behind this phenomenon. Apart from the practical financial pressure, the principle of commitment and consistency with previous believes (e.g. Cialdini 2001) is an important component in making the decision not to stop with an experiment, and as such, it should be investigated. Finally, these findings might relate to a sunk cost bias, which occurs when people continue to unsuccessfully invest, because of their previous investments which did not pay off (Arkes and Blumer, 1985). References Arkes, H. R. and C. Blumer (1985). The psychology of sunk cost Organizational Behavior and Human Decision Processes 35(1): 124-140.  Cialdini, R.B. (2001). Influence: Science and practice (4th ed.). Boston: Allyn & Bacon. Kitcher, P. (1990). The division of cognitive labor. Journal of Philosophy, 87(1), 5–22. Milojević, S. (2014). Principles of scientific research team formation and evolution. Proceedings of the National Academy of Sciences, 111(11), 3984–3989. Olson, B. J., Parayitam, S., & Bao, Y. (2007). Strategic decision making: The effects of cognitive diversity, conflict, and trust on decision outcomes. Journal of Management, 33(2), 196–222. Perović, S., Radovanović, S., Sikimić, V., & Berber, A. (2016). Optimal research team composition: data envelopment analysis of Fermilab experiments. Scientometrics, 108(1):83–111.   Roger S. (2012). Stopping rules and data monitoring in clinical trials. In H. W. de Regt, S. Hartmann & S. Okasha (eds.), EPSA Philosophy of Science: Amsterdam 2009, The European Philosophy of Science Association Proceedings Vol. 1, 375-386. Springer. pp. 375--386. Schulte, O. (2000). Inferring Conservation Laws in Particle Physics: A Case Study in the Problem of Induction. The British Journal for the Philosophy of Science, 51(4), 771-806. Sikimić, V., Radovanović, S., & Perović, S. Predictive Analysis of Experiment Efficiency in High Energy Physics: A Machine Learning Approach, submitted. Steele, K. (2013). Persistent Experimenters, Stopping Rules, and Statistical Inference. Erkenntnis 78 (4):937-961. Zollman, K. J. (2007). The communication structure of epistemic communities. Philosophy of Science, 74(5), 574–587View publication stats

When Should We Stop Investing in a Scientific Project? The Halting Problem in Experimental Physics

https://philpapers.org/archive/SIKWSW.pdf

When Should We Stop Investing in a Scientific Project? The Halting Problem in Experimental Physics

Abstract

Similar works

Full text

Available Versions

PhilPapers