This paper discusses how stylized facts derived from bibliometric studies can be used to build social simulation models of science. Based on a list of six stylized facts of science it illustrates how they can be brought into play to consolidate and direct research. Moreover, it discusses challenges such a stylized facts based approach of modeling science has to solve.Bibliometrics, Stylized Facts, Methodology, Model Comparison, Validation

Matthias Meyer

English

Research Papers in Economics

ﾩCopyright JASSS
JASSS logo
Matthias Meyer (2011)
Bibliometrics, Stylized Facts and the Way Ahead: How to Build Good Social Simulation Models
of Science?
Journal of Artificial Societies and Social Simulation 14 (4) 4
<http://jasss.soc.surrey.ac.uk/14/4/4.html>
Received: 20-Jun-2011    Accepted: 26-Jul-2011    Published: 31-Oct-2011
Abstract
This paper discusses how stylized facts derived from bibliometric studies can be used to
build social simulation models of science. Based on a list of six stylized facts of science it
illustrates how they can be brought into play to consolidate and direct research. Moreover, it
discusses challenges such a stylized facts based approach of modeling science has to
solve.
Keywords:
Bibliometrics, Stylized Facts, Methodology, Model Comparison, Validation
*   Introduction
1.1
For some time now, sociologists - and more recently also economists - have realized that
science provides an excellent object of study (Hands 2001; Gilbert 1997). Not only do they
have an intuitive connection to the object as they are scientists themselves. Science also
leaves many traces such as publications, citations and biographies that make it an
excellent test-bed for both probing theories and modeling (Gilbert 1997). Simulation models
seem to be particularly attractive because of their capacity to examine highly relevant
aspects, for instance processes, networks, heterogeneous actors and cognition.
1.2
By now, a considerable amount of simulation models of the science system exists.
Therefore, consolidating and focusing these efforts seems to be worthwhile. Among the
many questions such a process has to address are: What has been achieved by the
different models published in the different streams of literature? What are the remaining
issues with promise for further investigation? How to assess whether and in which respect
one model is better than others?
1.3
This paper discusses how stylized facts derived from bibliometric studies could help
answering these questions. Stylized facts are broad but not necessarily universal
generalizations of empirical observations and describe essential characteristics of a
phenomenon that require an explanation (Kaldor 1968; Meyer 2008). I will argue that the
4.html 1 11/6/11phenomenon that require an explanation (Kaldor 1968; Meyer 2008). I will argue that the
benefit of using stylized facts to construct and assess models of science applies not only to
a single research project (see Gilbert 1997 for an example). In addition, stylized facts can
also be drawn on for combining and directing the modeling efforts of a community of
researchers building models of science.
1.4
The remainder of this paper is structured as follows. In the next section, I will present some
candidates for stylized facts in science that can be derived from bibliometric studies. Then I
will explain how stylized facts of science can be used for merging and guiding research
especially within a community of researchers. On this basis, it will be discussed how a
future path might look and which challenges might have to be overcome in the future.
*  Bibliometrics and stylized facts in science
2.1
With respect to empirical data about science, researchers are in quite a fortunate position.
Several data sources exist to gather information about scientific publications and citations,
e.g. Science Citation Index, Scopus and Google Scholar. While the Science Citation Index
has been considered as the gold standard for a long time, the latter have become
interesting alternatives in terms of quality and scope. Bibliometric methods use the
information from such data sources as a starting point and analyze them. Among the most
prominent approaches are publication analyses, citation analyses and co-citation analyses.
They yield information about the most prolific or influential authors in a field or about
structural properties such as the level of differentiation or the existence of invisible colleges
(for an application to the field of social simulation see Meyer, Lorscheid and Troitzsch 2009
and Meyer, Zaggl and Carley 2011).
2.2
Studying publication and citation data has revealed several recurring patterns. They are
sometimes called "bibliometric laws" or "stylized facts of science" due to the repetition.
Without claiming to be exhaustive, among the most important are:
1.  Lotka's law describing the number of authors contributing a particular number of
papers to a field or journal (Lotka 1926; Simon 1957).
2.  Matthew effect: Prominent scientists receive disproportionally more credit than less
eminent researchers (Merton 1968). This phenomenon can also be observed for
citations in papers or journals.
3.  Exponential growth of the number of scientists and journals (de Solla Price 1963;
Gilbert 1997).
4.  Invisible colleges of specialties for every 100 scientists (de Solla Price 1963).
5.  Short half-life of literature with recent literature being cited much more frequently. For
the natural sciences, the time frame is about 5 years (Burton and Kebler 1960;
Umst￤tter, Rehm and Dorogi 1982).
6.  Bradford's law of scattering describing the distribution of articles on a given subject
across journals, again showing a high concentration on a few core journals (Cole
1993).
These stylized facts are "natural" targets for social simulation models of science in general
and agent-based modeling in particular (Sun and Naveh 2009). They can be understood as
global patterns in science and the question is how to explain them bottom-up through
micro-behavior.
*  Using stylized facts of science to consolidate and direct research
3.1
4.html 2 11/6/11Several scientists have recognized the opportunity for deriving simulation model targets
from these patterns (e.g. Gilbert 1997, Saam and Reiter 1999 and Sun and Naveh 2009). It
should be noted, that no paper has claimed to predict or mimic actual data. They simulate
the processes at work that lead to some of the main aspects of the phenomena observed.
Hence, the output of their simulation models is validated by comparing it with the supposed
stylized facts of science.
3.2
While my approach is similar, it differs in one aspect: It uses stylized facts of science to
consolidate and direct the research efforts of an entire community of researchers (see
Heine, Meyer and Strangfeld 2005; Meyer 2008). They provide an excellent reference point
when building good social simulation models of science. This can be achieved at two main
levels.
3.3
First, one can use a list of stylized facts to assess the state of the debate by examining
which stylized facts the different models have reproduced. From this perspective, the
contribution of a respective model lies in its ability to explain the stylized facts of a
phenomenon. A model that can at least partially explain a stylized fact is therefore regarded
as more valuable than one that is geared toward side issues. If there are several stylized
facts, a model's merit increases with the number of stylized facts it is able to explain
alongside the absence of contradictions relating to the unexplained stylized facts. Figure 1
illustrates the basic approach for the bibliometric laws mentioned above. In addition, it
shows how this approach easily identifies issues left to be addressed in the future. In our
example several stylized facts still need to be addressed by future research. Ideally, there
are in the future models reproducing all stylized facts. Given that the list of stylized facts
does not claim to be exhaustive, identifying further stylized facts would provide additional
objectives for future research.
Figure 1. A stylized facts perspective on selected social simulation models of science[1]
3.4
Second, one can go beyond adopting a simple instrumentalist position by adding a step to
4.html 3 11/6/11building good social simulation models of science. As a stylized fact can be replicated
through different modes within a model, an analysis has to be introduced to examine how
exactly the respective stylized fact has been replicated. With one or more stylized facts a
model can relate to, they can serve as a starting point to identify the model's basic
assumptions and parameters that, in turn, determine the reproduction of stylized facts.
Hence, stylized facts can be used to isolate the corresponding model elements and,
thereby, to enable subsequent model validation with a sufficient focus on yielding relevant
results. This is of particular interest when stylized facts are not reproduced adequately. It is
also possible to identify direct links between the basic assumptions as well as parameters
and stylized fact properties if they exist. This introduces a check for interpretability and
plausibility of the basic model elements. Such a control mechanism is especially important
when the modeling approach allows for certain degrees of freedom such as setting
parameters and their values. Ultimately, this perspective can indicate future modeling
options.
3.5
Grimm et al. (2005) provide an illustrative example of this approach with respect to agent-
based modeling.[2] They argue that combining several stylized facts can lead to structurally
more realistic models. In particular, assessing whether a set of stylized facts can be
reproduced in a meaningful way allows for discriminating between competing assumptions
(e.g. theories about agent behavior) and even for deciding about parameter values. The
authors illustrate this for an ecological model of trout behavior. In order to validate the
model, one must decide between several competing theories about how individual fish
select their habitats. Only one of the theories actually reproduces all three observed
patterns concerning feeding hierarchy, response to predatory fish and competing species,
and response to reduced food availability (Grimm et al. 2005). Being able to select from
competing assumptions at the level of agents is remarkable to the authors because each of
the patterns by itself is quite weak in terms of its falsificatory power. It is not until combining
the patterns that the researchers were able to eliminate the other theories.
3.6
Applying this approach to the simulation models of science mentioned above, one could for
example use it to focus the discussion on the claim by Sun and Naveh (2009) that their
cognitive, more "realistic" models provide an added value. Hence, adopting a stylized facts
perspective allows for transcending an overly simplistic realism versus instrumentalism
discussion.
*  The way ahead
4.1
An obvious next step would be to address the stylized facts not reproduced so far. Ideally,
the same model would be used for addressing every gap. One should also look for
additional stylized facts that are potentially relevant and include them as well.
4.2
Some less straightforward issues have also to be addressed. A main problem inherent in
stylized facts is that most of the time they are simply stated rather than derived with
scientific rigor (Heine, Meyer and Strangfeld 2005). This drawback is less prominent for
stylized facts of science than for many other fields. The statements about these recurring
patterns in science are at least based on empirical data and not just on casual empiricism.
Still, I believe that finding a more systematic way of doing this and putting more efforts in
deriving stylized facts is desirable (see Meyer 2008 for a discussion). To provide a solid
reference point for the uses described above, a consensus must be reached on (at least
some of) the stylized facts. Otherwise, the debate merely shifts from the model level to the
level of stylized facts. Systematically deriving stylized facts from existing empirical
4.html 4 11/6/11evidence could also lead to refining our knowledge about these recurring patterns. This is
how early statements about Lotka's law have been advanced by subsequent empirical
investigations (Pao 1985). Similarly, one might find differences between fields. The half-
time of literature might be longer for the social sciences than for the natural sciences. Such
differences in themselves might be interesting aspects to be investigated in future
simulation studies.
4.3
When a list of accepted and empirically well grounded stylized facts exists, several issues
can be tackled. One element to be addressed is the question about the appropriate unit of
analysis in social simulation models of science: scientists or papers. In Gilbert's model,
papers represent the basic unit of analysis, whereas Saam and Reiter's model features
individual scientists. Sun and Naveh are positioned in the middle between the other two. I
believe that looking more closely "under the hood" (Hausman 1995) of the different models
by using the stylized facts will help answer this question, because it focuses model
analysis. Similarly, the related question about the elements to be included in such models
can be addressed from a stylized facts perspective when assessing the contributions of
more "realistic" models incorporating cognition and networks in terms of their added
explanatory value. Stylized facts might even become a useful reference point for reducing
the complexity of models by establishing which model elements can be simplified without
losing the ability to reproduce a given set of stylized facts (Sun and Naveh 2009).
4.4
Besides the many possible benefits using a set of stylized facts as a reference point for a
community of researchers, some challenges still exist. A first potential limitation for social
simulation studies based on bibliometric information is the data quality of the different
sources poses. It has been reported to be quite poor for SCI (Meyer, Lorscheid and
Troitzsch 2009). Similarly, the data provided by Google scholar often requires further
refinement. Another well-known problem of SCI data is that it only provides the name of the
first author of references. This can make it difficult examining citation behavior and
addressing questions such as whether scientists act based on direct and indirect reciprocity
when making citations.
4.5
A second issue stems from the fact that stylized facts typically refer to macro-data.
Therefore, currently stylized facts of science cannot be used for direct validation at the
micro-level (only in the indirect way described above). If stylized facts were derived for the
micro-level of the models as well, they would not be used for output validation but for input
validation. Introducing an appropriate terminology would reduce confusion and help to
distinguish between the two types of stylized facts with respect to their uses (e.g. stylized
facts used for input validation versus stylized facts used for output validation).
4.6
A final caveat refers to an overly dogmatic and static application of stylized facts. Used in
such a way for guiding the research efforts of a community stylized facts can be perceived
as a methodological straightjacket and they might even hamper progress. Not all interesting
aspects of a phenomenon are necessarily captured by the "accepted list of stylized facts".
Similarly, models might produce some results which pinpoint new aspects of a
phenomenon that have yet to be considered. This should remind researchers to subject
stylized facts to scientific scrutiny and revision if necessary in the same way that scientific
theories are.
*   Notes
4.html 5 11/6/111Please note that the mapping of stylized facts to the respective models is mainly for
illustration purposes and is solely based on the claims of the authors in the quoted papers.
Moreover, other valid criteria to assess the quality of models do exist beyond the ability to
reproduce stylized facts (e.g. Burton and Obel 1995).
2They call this model validation strategy "pattern-oriented modeling", but it follows a very
similar basic logic.
*   References
BURTON, R and OBEL, B (1995). The Validity of Computational Models in Organization
Science: From Model Realism to Purpose of the Model. Computational and Mathematical
Organization Theory, Vol. 1, No. 1, pp. 57-71.
BURTON, R E and KEBLER, R W (1960). The "half life" of some scientific and technical
literatures. American documentation, Vol. 11, No. 1, pp. 18-22.
COLE, P F (1993). A new look at reference scattering. Journal of Documentation, Vol. 18,
No. 2, pp. 58-64.
DE SOLLA PRICE, D J (1963). Little science, big science... and beyond. New York:
Columbia University Press.
GILBERT, N (1997). A Simulation of the Structure of Academic Science. Sociological
Research Online, Vol. 2, No. 2. http://www.socresonline.org.uk/2/2/3.html
GRIMM, V, REVILLA, E, BERGER, U, JELTSCH, F, MOOIJ, W M, RAILSBACK, S F,
THULKE, H-H, WEINER, J, WIEGAND, T and DEANGELIS, D L (2005). Pattern-Oriented
Modeling of Agent-Based Complex Systems: Lessons from Ecology. Science, Vol. 310,
No., pp. 987-991.
HANDS, D W (2001). Reflection without Rules: Economic Methodology and Contemporary
Science Theory. Cambridge: Cambridge Univ. Press.
HAUSMAN, D M (1995). Why Look Under the Hood? In Hausman, D M (Ed.): The
Philosophy of Economics: an Anthology. 2. ed., reprint. Cambridge: Cambridge Univ.
Press, pp. 217-222.
HEINE, B O, MEYER, M and STRANGFELD, O (2005). Stylised facts and the contribution
of simulation to the economic analysis of budgeting. Journal of Artificial Societies and
Social Simulation, 8 (4) 4 http://jasss.soc.surrey.ac.uk/8/4/4.html
KALDOR, N (1968). Capital Accumulation and Economic Growth. In Lutz, F A and Hague,
D C (Ed.): The Theory of Capital. Reprint. London: Macmillan, pp. 177-222.
LOTKA, A J (1926). The frequency distribution of scientific productivity. Journal of
Washington Academy Sciences, Vol. 16, No. 12, pp. 317-324.
MERTON, R K (1968). The Matthew effect in science. Science, Vol. 159, No. 3810, pp. 56-
63.
MEYER, M (2008). Stylized Facts as a Solid Basis for Validating Simulation Models? -
Some Epistemological Reflections. EPOS 2008. Lisbon.
MEYER, M, LORSCHEID, I and TROITZSCH, K G (2009). The Development of Social
4.html 6 11/6/11Simulation as Reflected in the First Ten Years of JASSS: A Citation and Co-Citation
Analysis. Journal of Artificial Societies and Social Simulation, 4 (12) 12.
http://jasss.soc.surrey.ac.uk/12/4/12.html
MEYER, M, ZAGGL, M A and CARLEY, K M (2011). Measuring CMOT's intellectual
structure and its development. Computational and Mathematical Organization Theory, Vol.
17, No. 1, pp. 1-34.
PAO, M L (1985). Lotka's law: A testing procedure. Vol. 21, No. 4, pp. 305-320.
SAAM, N S and REITER, S (1999). Lotka's law reconsidered: The evolution of publication
and citation distributions in scientific fields. Scientometrics, Vol. 44, No. 2, pp. 135-155.
SIMON, H A (1957). Models of man, social and rational. New York: Wiley.
SUN, R and NAVEH, I (2009). Cognitive simulation of academic science. International Joint
Conference on Neural Networks: 3011-3017.
UMSTￄTTER, W, REHM, M and DOROGI, Z (1982). Die Halbwertszeit in der
naturwissenschaftlichen Literatur. Nachrichten f￼r Dokumentation, Vol. 33, No. 2, pp. 50-52.
4.html 7 11/6/11

A new look at reference scattering.

A Simulation of the Structure of Academic Science.

Cognitive simulation of academic science.

Die Halbwertszeit in der naturwissenschaftlichen Literatur.

Little science, big science... and beyond.

Lotka's law reconsidered: The evolution of publication and citation distributions in scientific fields.

Lotka's law: A testing procedure.

Reflection without Rules: Economic Methodology and Contemporary Science Theory. Cambridge:

Stylised facts and the contribution of simulation to the economic analysis of budgeting.

Stylized Facts as a Solid Basis for Validating Simulation Models? -Some Epistemological Reflections. EPOS

The &quot;half life&quot; of some scientific and technical literatures.

The Development of Social 4.html 6 11/6/11Simulation as Reflected in the First Ten Years of JASSS: A Citation and Co-Citation Analysis.

The frequency distribution of scientific productivity.

The Matthew effect in science.

The Validity of Computational Models in Organization Science: From Model Realism to Purpose of the Model.

Why Look Under the Hood?

Bibliometrics, Stylized Facts and the Way Ahead: How to Build Good Social Simulation Models of Science?

http://jasss.soc.surrey.ac.uk/14/4/4/4.pdf

Bibliometrics, Stylized Facts and the Way Ahead: How to Build Good Social Simulation Models of Science?

Abstract

Similar works

Full text

Available Versions

Research Papers in Economics