Reviewing, indicating, and counting books for modern research evaluation systems

In this chapter, we focus on the specialists who have helped to improve the conditions for book assessments in research evaluation exercises, with empirically based data and insights supporting their greater integration. Our review highlights the research carried out by four types of expert communities, referred to as the monitors, the subject classifiers, the indexers and the indicator constructionists. Many challenges lie ahead for scholars affiliated with these communities, particularly the latter three. By acknowledging their unique, yet interrelated roles, we show where the greatest potential is for both quantitative and qualitative indicator advancements in book-inclusive evaluation systems.Comment: Forthcoming in Glanzel, W., Moed, H.F., Schmoch U., Thelwall, M. (2018). Springer Handbook of Science and Technology Indicators. Springer Some corrections made in subsection 'Publisher prestige or quality

The structural role of the core literature in history

The intellectual landscapes of the humanities are mostly uncharted territory. Little is known on the ways published research of humanist scholars defines areas of intellectual activity. An open question relates to the structural role of core literature: highly cited sources, naturally playing a disproportionate role in the definition of intellectual landscapes. We introduce four indicators in order to map the structural role played by core sources into connecting different areas of the intellectual landscape of citing publications (i.e. communities in the bibliographic coupling network). All indicators factor out the influence of degree distributions by internalizing a null configuration model. By considering several datasets focused on history, we show that two distinct structural actions are performed by the core literature: a global one, by connecting otherwise separated communities in the landscape, or a local one, by rising connectivity within communities. In our study, the global action is mainly performed by small sets of scholarly monographs, reference works and primary sources, while the rest of the core, and especially most journal articles, acts mostly locally

Approaches to Understanding and Measuring Interdisciplinary Scientific Research (IDR): A Review of the Literature

Interdisciplinary scientific research (IDR) challenges the study of science from a number of fronts, including one of creating output science and engineering (S&E) indicators. This literature review began with a narrow focus on quantitative measures of the output of IDR, but expanded the scope as it became clear that differing definitions, assessment tools, evaluation processes, and measures all shed light on aspects of IDR. Key among the broader aspects are (a) characterizing the concept of knowledge integration, and (b) recognizing that it can occur within a single mind or as the result of team dynamics. Output measures alone cannot adequately capture this process. Among the quantitative measures considered, bibliometrics (co-authorships, collaborations, references, citations and co-citations) are the most developed, but leave considerable gaps in understanding. Emerging measures in diversity, entropy, and network dynamics are promising, but require sophisticated interpretations and thus would not serve well as S&E indicators. Combinations of quantitative and qualitative assessments coming from evaluation studies appear to reveal S&E processes but carry burdens of expense, intrusion, and lack of reproducibility. This review is a first step toward providing a more holistic view of measuring IDR; several avenues for future research highlight the need for metrics to reflect the actual practice of IDR

Berita Autumn 2016

Table of Contents Chair’s Address ... 2 Editor’s Foreword ... 4 John A. Lent Prize ... 5 “After Decolonization” – A Panel Report ... 6 Association for Asian Studies 2017 (Toronto) – Panels with Malaysia-Singapore-Brunei Content ... 8https://ohioopen.library.ohio.edu/berita/1042/thumbnail.jp

Comparing nuclear power trajectories in Germany and the UK: from ‘regimes' to ‘democracies’ in sociotechnical transitions and Discontinuities

This paper focuses on arguably the single most striking contrast in contemporary major energy politics in Europe (and even the developed world as a whole): the starkly differing civil nuclear policies of Germany and the UK. Germany is seeking entirely to phase out nuclear power by 2022. Yet the UK advocates a ‘nuclear renaissance’, promoting the most ambitious new nuclear construction programme in Western Europe.Here,this paper poses a simple yet quite fundamental question: what are the particular divergent conditions most strongly implicated in the contrasting developments in these two countries. With nuclear playing such an iconic role in historical discussions over technological continuity and transformation, answering this may assist in wider understandings of sociotechnical incumbency and discontinuity in the burgeoning field of‘sustainability transitions’. To this end, an ‘abductive’ approach is taken: deploying nine potentially relevant criteria for understanding the different directions pursued in Germany and the UK. Together constituted by 30 parameters spanning literatures related to socio-technical regimes in general as well as nuclear technology in particular, the criteria are divided into those that are ‘internal’ and ‘external’ to the ‘focal regime configuration’ of nuclear power and associated ‘challenger technologies’ like renewables. It is ‘internal’ criteria that are emphasised in conventional sociotechnical regime theory, with ‘external’ criteria relatively less well explored. Asking under each criterion whether attempted discontinuation of nuclear power would be more likely in Germany or the UK, a clear picture emerges. ‘Internal’ criteria suggest attempted nuclear discontinuation should be more likely in the UK than in Germany– the reverse of what is occurring. ‘External’ criteria are more aligned with observed dynamics –especially those relating to military nuclear commitments and broader ‘qualities of democracy’. Despite many differences of framing concerning exactly what constitutes ‘democracy’, a rich political science literature on this point is unanimous in characterising Germany more positively than the UK. Although based only on a single case,a potentially important question is nonetheless raised as to whether sociotechnical regime theory might usefully give greater attention to the general importance of various aspects of democracy in constituting conditions for significant technological discontinuities and transformations. If so, the policy implications are significant. A number of important areas are identified for future research, including the roles of diverse understandings and specific aspects of democracy and the particular relevance of military nuclear commitments– whose under-discussion in civil nuclear policy literatures raises its own questions of democratic accountability

Comparing journal and paper level classifications of science

The classification of science into disciplines is at the heart of bibliometric analyses. While most classifications systems are implemented at the journal level, their accuracy has been questioned, and paper-level classifications have been considered by many to be more precise. However, few studies investigated the difference between journal and the paper classification systems. This study addresses this gap by comparing the journal- and paper-level classifications for the same set of papers and journals. This isolates the effects of classification precision (i.e., journal- or paper-level) to reveal the extent of paper misclassification. Results show almost half of papers could be misclassified in journal classification systems. Given their importance in the construction and analysis of bibliometric indicators, more attention should be given to the robustness and accuracy of these disciplinary classifications schemes

The distributed organization of science : with empirical illustrations from the field of diabetes medicine

The premise of this thesis holds that the organization of science is distributed in nature. That is, science takes place all over the world and within different spheres of society. Within the literature, the distributed organization of science is characterized in different ways. While some focus primarily on the substantive aspects of science such as the increasing importance of interdisciplinarity, others emphasize processes of de-institutionalization as seen in the cooperation between companies, universities and government agencies. Other contributions combine several aspects of science and as such for example speak of both the social and intellectual organization of science. More in general, then, the distributed organization of science can be characterized along different dimensions. The aim of this thesis is to provide insight into the nature and consequences of a distributed organization of science. We take the distinction between Mode 1 and Mode 2 knowledge production as a starting point to describe the distributed organization of science. Mode 1 knowledge production conforms in many respects to the traditional image of scientific knowledge production that takes place primarily within the university; under Mode 1 knowledge production science takes place within strongly defined disciplines and focuses on the fundamental comprehension of natural and social phenomena. In contrast, the idea of Mode 2 knowledge production is characterized by its heterogeneity along all dimensions. That is, besides universities, actors from industry and government are also involved in science; science takes place across disciplines rather than within disciplines only; and under Mode 2 knowledge production science involves not only a quest for fundamental laws and regularities, but has a clear public interest. Despite, or perhaps because of its rich description of science, the distinction between Mode 1 and Mode 2 knowledge production is much criticized. This criticism can be summarized in terms of three main points. First, the notion of Mode 2 knowledge production is conceptually vague. For example, it is unclear what exactly is meant by the notion of transdisciplinarity. Second, the empirical validity of claims on the emergence, prevalence, and persistence of Mode 2 knowledge production is debatable. Although at the project level there is evidence on an increase in the diversity of actors involved in science, it is unclear to what extent this also holds at the level of science systems at large. Finally, it is unclear to what extent Mode 2 knowledge production should be interpreted as a positive phenomenon in a normative sense. The normative implications of the idea of Mode 2 knowledge production have at least two sides. On the one hand the question arises to what extent and how diversity in the organization of distributed science indeed leads to more relevant knowledge. On the other hand, it could also be asked to what extent a development of science towards Mode 2 would prejudice the public interest of scientific knowledge. Within this thesis, the criticisms of the notion of Mode 2 knowledge production are picked up. As such, this thesis addresses (i) an analytical approach to the notion of Mode 2 knowledge production, (ii) the empirical validity of the notion of Mode 2 knowledge production, (iii) the establishment of relevance in a distributed organization of science, and (iv) the normative implications of Mode 2 knowledge production. To strengthen our arguments empirically, we use the case of diabetes medicine. Diabetes medicine is an interesting case for at least three reasons. First, diabetes is a socially relevant problem in the sense that a large group of people around the world are faced with this disease. Consequently, research on diabetes is also widespread. Second, diabetes constitutes a complex disease involving interacting factors such as genetics, lifestyle, and the environment. However, not only are the aspects involved in the constitution of this disease varied, as a consequence so are the people and organizations occupying themselves with finding solutions to this problem. What medical professionals call translational medicine seems to be especially accurate for diabetes, that is, as a description of medical science that concerns itself with diabetes duly takes into account the whole process from the laboratory bench to the patient bedside involving different actors. As such, the nature of diabetes as a scientific problem is immediately enmeshed with societal undertones whose provision of solutions is expected to be organized along various modes. Hence, we expect the organization of diabetes medicine to be characterized by Mode 2 rather than Mode 1 knowledge production. The patterns of a distributed organization of science are addressed empirically using bibliometric data. The choice for a quantitative approach in our empirical research is pragmatic; bibliometrics allows me to address the science system on a large (i.e. global) scale. More fundamentally, we take the scientific publication as a useful starting point to assess science. Here we make a distinction between research and science. Whereas research is about local knowledge production practices, science is first and foremost about the transformation of knowledge towards universal acceptance. As such, the scientific publication is taken as a first and necessary step towards the certification of knowledge as scientific. It follows that the organizational aspects of science that are displayed on the scientific publication such as authorships, affiliations and references provide an input to investigate the patterns underlying the distributed organization of science. Chapter 1 provides a first outline of an analytical approach to Mode 2 knowledge production. To substantiate the notion of Mode 2 knowledge production analytically we distinguish and define five forms of heterogeneity. First, institutional heterogeneity refers to the different value orientations and norms the different actors involved in science adhere to. Second, organizational heterogeneity refers to the different organizations involved in science. Third, geographical heterogeneity refers to science taking place in different countries, regions and cities. Fourth, cognitive heterogeneity refers to the various disciplinary backgrounds of actors. Finally, social heterogeneity refers to the different communities in which actors are active. While some descriptions of science focus on only one dimension of heterogeneity in the distributed organization of science, the notion of Mode 2 knowledge production provides a description of science along all of these five dimensions. The great advantage of studying science along these five dimensions is that we can now analytically address the distributed organization of science along multiple dimensions simultaneously. Then, chapter 2 provides an overview of the recent empirical literature on the distributed organization of science. Of central concern here is the relationship between proximity on the one hand and impact and collaboration on the other hand. At the relational level, the concept of proximity is taken as the counterpart of the concept of heterogeneity. That is to say, where cooperation takes place between operators who are in close proximity to each other, the relationships between these actors are described as homogenous rather than heterogeneous. The main conclusion of this literature review holds that most studies that examine the role of proximity in collaborative science look only at a limited number of proximity dimensions. It follows that to gain a complete understanding of the distributed organization of science we need to include multiple proximity dimensions simultaneously. Chapter 3 forms the prelude to the study of collaboration patterns between organizations in chapter 4. To come to such an analysis, first the idea of 'the organization' is discussed in chapter 3. While in quantitative science studies the nature of the organization is assumed to be unproblematic, within discussions of Mode 2 knowledge production precisely the opposite is true. That is, within the notion of Mode 2 knowledge production, the boundaries between, say, the university and the commercial enterprise have faded. As such, the organization is not an unambiguous unit of analysis in quantitative science studies. However, this does not mean that the organization cannot be used to study science at a higher level of aggregation. Rather, in taking the organization as the basic unit of analysis in quantitative science studies choices must be made in conceptualizing the organization in the first place. These choices are not completely value-free, but must be viewed in light of the research in which the organization as the unit of analysis is used. On the basis of various organization theories, chapter 3 shows how the organization can be conceptualized along several dimensions. Given our conceptualization of the organization that we proposed in chapter 3, chapter 4 analyzes collaboration patterns between organizations. Whereas in the literature on Mode 2 knowledge production the diversity of backgrounds of scientific actors is taken to form no obstacle for collaboration to take place, within the literature on proximity and innovation emphasis is put on the role of proximity in facilitating collaborative innovation. The latter does not mean that proximity needs to play a role along all five dimensions. On the contrary, there may be substitution between different forms of proximity. As such, differences may exist among science systems in how various proximity dimensions shape collaboration between organizations therein. Chapter 4 examines (i) to what extent proximity in all five dimensions plays a role in scientific collaboration in the field of type 2 diabetes and (ii) to what extent the European science system differs from the North American system of science in terms of the comparative importance of the five proximity dimensions. Regarding the role of proximity in scientific collaboration, the main conclusion of this chapter holds that in general all proximity dimensions play their role in shaping collaboration between organizations. In particular, geographical proximity plays an important role in scientific collaboration which suggests that a regional or national focus in the study of science and innovation systems is legitimate. On the other hand, the focus on a "Triple Helix" of university-industry-government relations is no less legitimate because of the relative importance of this type of collaboration, both in North America and in Europe. Regarding the comparison between the European and North American science system, a difference is observed in the role of geographical, social and organizational proximity in shaping scientific collaboration. Where geographical proximity plays a larger role within the European science system, social and organizational proximity play a larger role within the North American science system. The relative importance of geographical proximity within the European science system can be traced to the greater differences in terms of language and culture in Europe. On the other hand the relative importance of organizational and social proximity in North America suggests a more hierarchical system there. It is notable that with regard to the role of institutional proximity the two science systems do not differ. In other words, the attention paid in policy discussions to a relative absence of relationships between academic and non-academic actors in Europe as compared to North America is not justified. The last three chapters of this thesis discuss the implications of a distributed organization of science. First, chapter 5 addresses the citation as a measure of scientific (Mode 1) impact. Within science studies the citation is a contested measure of scientific impact. While some take little issue in using citation indicators, others completely dispense with the use of citation analysis as a tool for scientific evaluation. In order to get out of this impasse we turn to information science studies (in particular, "information retrieval" studies), in which the concept of relevance is important. Parallel to the debate on citation theories, where a distinction is made between a Mertonian perspective on citation as value and a rhetorical perspective on citation as personal, within the information science literature a distinction is made between relevance as system-oriented and relevance as user-oriented. Recently, however, a third perspective on relevance emerged within the information retrieval literature. This socio-cognitive perspective on relevance connects the system approach to the user approach on relevance by paying explicit attention to the context in which relevance is established. On the basis of this socio-cognitive perspective on relevance, we develop a supplement to existing citation theories on the basis of the notion of social embeddedness. The most important conclusion is that, on the basis of the concept of social embeddedness, the two opposite perspectives on citation can be connected. In all we argue that, in the context of Mode 1 knowledge production, the establishment of scientific relevance is contingent upon the structure of social networks and the position of scientists therein. Chapter 6 addresses the role of heterogeneity in relation to the societal relevance (Mode 2 impact) of science. Again we use the case of science in the field of type 2 diabetes. As in Chapter 4, we operationalize the distributed organization of science through five forms of heterogeneity. However, instead of talking about the role of distance (proximity) in the distributed organization of science we speak of the impact that diversity (singularity) in the organization of science has on in its societal relevance. To assess societal relevance, we use the references listed in a clinical practice guideline. Two types of references are distinguished: (i) references that are included in the clinical practice guideline but not as evidence for the treatment of type 2 diabetes and (ii) references that are included in the medical manual and also reflect evidence for the treatment of type 2 diabetes. In comparing the organizational aspects related to the publications associated with these two types of references, we assess the determinants of societal relevance in medical science in the field of type 2 diabetes. The main conclusion holds that, controlling for the scientific relevance of publications, only geographical diversity increases the likelihood of societal relevance. In all it seems that heterogeneity in the distributed organization of science does not naturally lead to a greater chance of societally relevant knowledge. Interesting fact is that publications in which industry is involved have a greater chance of becoming societally relevant. This suggests that the influence of industry in the creation of societally relevant knowledge is large. Finally, chapter 7 elaborates further on the position of industry in medicine. We assess the publication behavior of firms in a context of complete information disclosure where firms face the choice of publishing study outcomes either in scientific publications or in web publications. Due to recent institutional reforms it is now mandated to register clinical trial protocols before onset and publish basic results after study completion. For a sample of clinical trials on diabetes, we link clinical trial protocols to result publications and classify those publications based on the type of evidence they disclose. The results indicate that under conditions of complete information disclosure, firms do indeed not publish less than not-for-profit organizations. However, firms strategically publish in scientific journals where they highlight favorable outcomes to their therapies and clinically relevant studies, since regulators value evidence published in peerreviewed journals much more than evidence published on web sites without peer-review. Thus, despite institutional reforms, pharmaceutical firms still find a way to strategically highlight particular pieces of evidence in scientific journals. We conclude that concerns about publication based on the nature of evidence have shifted rather than disappeared. The presented results in this chapter thus signal a problem of persistent publication bias of a more fundamental nature which is not easily solved by regulatory reform alone. The general conclusion of this thesis is threefold. First, the framework of proximity (distance) and diversity (uniformity) along five dimensions provides a useful analytical tool to address the distributed organization of science. Using this framework, two important critiques on the idea of Mode 2 knowledge production, namely its lacking conceptual clarity and empirical validity, can be tackled. Characterizing scientific actors and their relations along lines of geographical, social, cognitive, institutional, and organizational heterogeneity, renders a more distinct picture of the distributed organization of science. Second, the idea of Mode 2 knowledge production as conceptualized along five dimensions of heterogeneity takes different shapes depending on the level of aggregation. On the level of individual organizations we argue that since the boundaries of the organizations are inherently blurred, many organizations can in principle be characterized as Mode 2. Yet, our empirical analysis shows that on an aggregate level the science system as a whole is not likely to be characterized as Mode 2. Rather, proximity plays an important role in shaping collaboration among organizations. Third, the implications of heterogeneity or Mode 2-ness in the distributed organization of science are ambiguous. On the one hand, heterogeneity in the distributed organization of (medical) science does not render societal relevant knowledge more likely per se. We only find evidence of an increase in the likelihood of societal relevant outcomes under geographical diversity and not for the other four dimensions of heterogeneity. On the other hand, the involvement of industrial actors does render societal relevant knowledge more likely. However, the extent to which such involvement is desirable from a normative perspective is unclear. What holds is that pharmaceutical companies publish their study outcomes strategically. This study has two important implications for further research on the distributed organization of science. First, the framework of proximity and diversity along several dimensions provides an input for further research on the distributed organization of science. Not only can local science systems in this way be compared; in principle one can also compare different disciplinary systems in the same way. In addition, the dynamics of scientific collaboration can be addressed using social network analysis techniques. Second, the relationship between scientific and societal relevance warrants further research. Some exceptions aside, much of quantitative science studies focuses primarily on the development of scientific relevance leaving societal relevance often unaddressed. Ultimately, such data sets enable us to get a better description and explanation of the whole sequence from research via publication towards science becoming societally relevant. More in general, quantitative studies of science might bring together our understanding of knowledge production by not only taking into account journal publication data but also considering alternative data sets reflecting upon research and science’s societal relevance. Finally, with regard to science policy, the question holds what kind of heterogeneity should be targeted given the particular scientific and societal problems at stake. In addition, science policy makers should take into account the institutional requirements of a heterogeneous science system. Non-traditional scientific actors such as companies have interests that are not necessarily in line with the public provision of scientific knowledge. This does not mean that these actors by definition should be excluded from science. Rather, this raises the question whether the way that science is traditionally organized still adequately serves the general interest of public knowledge provision

Interdisciplining Digital Humanities: Boundary Work in an Emerging Field

The first book to test the claim that the emerging field of Digital Humanities is interdisciplinary and also examines the boundary work of establishing and sustaining a new field of stud