Making Progress in Forecasting

Twenty-five years ago, the International Institute of Forecasters was established “to bridge the gap between theory and practice.” Its primary vehicle was the Journal of Forecasting and is now the International Journal of Forecasting. The Institute emphasizes empirical comparisons of reasonable forecasting approaches. Such studies can be used to identify the best forecasting procedures to use under given conditions, a process we call evidence-based forecasting. Unfortunately, evidence-based forecasting meets resistance from academics and practitioners when the findings differ from currently accepted beliefs. As a consequence, although much progress has been made in developing improved forecasting methods, the diffusion of useful forecasting methods has been disappointing. To bridge the gap between theory and practice, we recommend a stronger emphasis on the method of multiple hypotheses and on invited replications of important research. It is then necessary to translate the findings into principles that are easy to understand and apply. The Internet and software provide important opportunities for making the latest findings available to researchers and practitioners. Because researchers and practitioners believe that their areas are unique, we should organize findings so that they are relevant to each area and make them easily available when people search for information about forecasting in their area. Organisational barriers to change still remain to be overcome. Research into the specific issues faced when forecasting remains a priority

Research on Scientific Journals: Implications for Editors and Authors

A review of editorial policies of leading journals and of research relevant to scientific journals revealed conflicts between 'science' and 'scientists.” Owing to these conflicts, papers are often weak on objectivity and replicability. Furthermore, papers often fall short on importance, competence, intelligibility, or efficiency. Suggestions were made for editorial policies such as: (1) structured guidelines for referees, (2) open peer review, (3) blind reviews, and (4) full disclosure of data and method. Of major importance, an author's “Note to Referees” (describing the hypotheses and design, but not the results) was suggested to improve the objectivity of the ratings of importance and competence. Also, recommendations are made to authors for improving contributions to science (such as the use of multiple hypotheses) and for promoting their careers (such as using complex methods and obtuse writing).scientific journals, authors, research, editors

Discovery and Communication of Important Marketing Findings: Evidence and Proposals

My review of empirical research on scientific publication led to the following conclusions. Three criteria are useful for identifying whether findings are important: replication, validity, and usefulness. A fourth criterion, surprise, applies in some situations. Based on these criteria, important findings resulting from academic research in marketing seem to be rare. To a large extent, this rarity is due to a reward system that is built around subjective peer review. Rather than using peer review as a secret screening process, using an open process likely will improve papers and inform readers. Researchers, journals, business schools, funding agencies, and professional organizations can all contribute to improving the process. For example, researchers should do directed research on papers that contribute to principles. Journals should invite papers that contribute to principles. Business school administrators should reward researchers who make important findings. Funding agencies should base decisions on researchers' prior success in making important findings, and professional organizations should maintain web sites that describe what is known about principles and what research is needed on principles

“Excellence R Us”: university research and the fetishisation of excellence

The rhetoric of “excellence” is pervasive across the academy. It is used to refer to research outputs as well as researchers, theory and education, individuals and organisations, from art history to zoology. But does “excellence” actually mean anything? Does this pervasive narrative of “excellence” do any good? Drawing on a range of sources we interrogate “excellence” as a concept and find that it has no intrinsic meaning in academia. Rather it functions as a linguistic interchange mechanism. To investigate whether this linguistic function is useful we examine how the rhetoric of excellence combines with narratives of scarcity and competition to show that the hypercompetition that arises from the performance of “excellence” is completely at odds with the qualities of good research. We trace the roots of issues in reproducibility, fraud, and homophily to this rhetoric. But we also show that this rhetoric is an internal, and not primarily an external, imposition. We conclude by proposing an alternative rhetoric based on soundness and capacity-building. In the final analysis, it turns out that that “excellence” is not excellent. Used in its current unqualified form it is a pernicious and dangerous rhetoric that undermines the very foundations of good research and scholarship

Discovery and Communication of Important Marketing Findings: Evidence and Proposals

My review of empirical research on scientific publication led to the following conclusions. Three criteria are useful for identifying whether findings are important: replication, validity, and usefulness. A fourth criterion, surprise, applies in some situations. Based on these criteria, important findings resulting from academic research in marketing seem to be rare. To a large extent, this rarity is due to a reward system that is built around subjective peer review. Rather than using peer review as a secret screening process, using an open process likely will improve papers and inform readers. Researchers, journals, business schools, funding agencies, and professional organizations can all contribute to improving the process. For example, researchers should do directed research on papers that contribute to principles. Journals should invite papers that contribute to principles. Business school administrators should reward researchers who make important findings. Funding agencies should base decisions on researchers' prior success in making important findings, and professional organizations should maintain web sites that describe what is known about principles and what research is needed on principles.marketing, marketing findings

How to Create an Innovation Accelerator

Too many policy failures are fundamentally failures of knowledge. This has become particularly apparent during the recent financial and economic crisis, which is questioning the validity of mainstream scholarly paradigms. We propose to pursue a multi-disciplinary approach and to establish new institutional settings which remove or reduce obstacles impeding efficient knowledge creation. We provided suggestions on (i) how to modernize and improve the academic publication system, and (ii) how to support scientific coordination, communication, and co-creation in large-scale multi-disciplinary projects. Both constitute important elements of what we envision to be a novel ICT infrastructure called "Innovation Accelerator" or "Knowledge Accelerator".Comment: 32 pages, Visioneer White Paper, see http://www.visioneer.ethz.c

A Century of American Economic Review

Using information collected from American Economic Review publications of the last 100 years, we try to provide answers to various questions: Which are the top AER publishing institutions and countries? Which are the top AER papers based on citation success? How frequently is someone able to publish in AER? How equally is citation success distributed? Who are the top AER publishing authors? What is the level of cooperation among the authors? What drives the alphabetical name ordering? What are the individual characteristics of the AER authors, editors, editorial board members, and referees? How frequently do women publish in AER? What is the relationship between academic age, publication performance, and citation success? What are the paper characteristics? What influences the level of technique used in articles? Do connections have an influence on citation success? Who receives awards? Can awards increase the probability of publishing in AER at a later stage?American Economic Review, Publishing Economics, Rankings, Cooperation, Authors, Editors, Board Members, Referees, Connections, Awards, Paper Characteristics, Economic History, History of Economic Thought

A Century of American Economic Review

Using information collected from American Economic Review publications of the last 100 years, we try to provide answers to various questions: Which are the top AER publishing institutions and countries? Which are the top AER papers based on citation success? How frequently is someone able to publish in AER? How equally is citation success distributed? Who are the top AER publishing authors? What is the level of cooperation among the authors? What drives the alphabetical name ordering? What are the individual characteristics of the AER authors, editors, editorial board members, and referees? How frequently do women publish in AER? What is the relationship between academic age, publication performance, and citation success? What are the paper characteristics? What influences the level of technique used in articles? Do connections have an influence on citation success? Who receives awards? Can awards increase the probability of publishing in AER at a later stage?American Economic Review, publishing economics, rankings, cooperation,authors, editors, board members, referees, connections, awards, paper characteristics, economic history, history of economic thoug

Publishing an E-journal on a shoe string: Is it a sustainaible project?.

The aim of this article is to report on an experiment in publishing an open access journal and learn from it about the larger field of open access publishing. The experiment is the launch of the European Journal of Comparative Economics (EJCE), an on-line refereed and open access journal, founded in 2004 by the European Association for Comparative Economic Studies and LIUC University in Italy. They embarked upon this project in part to respond to the rising concentration in the market for scientific publishing and the resulting use of market power to raise subscription prices and restrict access to scientific output. We had hoped that open access journals could provide some countervailing power and increase competition in the field. Our experience running a poorly endowed journal has shown that entry to the field may be easy, yet that making it a sustainable enterprise is not straightforward.Open-access publishing, online journals, scientific publication

Impact Factor: outdated artefact or stepping-stone to journal certification?

A review of Garfield's journal impact factor and its specific implementation as the Thomson Reuters Impact Factor reveals several weaknesses in this commonly-used indicator of journal standing. Key limitations include the mismatch between citing and cited documents, the deceptive display of three decimals that belies the real precision, and the absence of confidence intervals. These are minor issues that are easily amended and should be corrected, but more substantive improvements are needed. There are indications that the scientific community seeks and needs better certification of journal procedures to improve the quality of published science. Comprehensive certification of editorial and review procedures could help ensure adequate procedures to detect duplicate and fraudulent submissions.Comment: 25 pages, 12 figures, 6 table