F. De Battisti

S. Salini

AIR Universita degli studi di Milano

Bibliometric indicators for statisticians: criticalassessment in the italian contextFrancesca De Battisti, Silvia Salini1 IntroductionThe evaluation of the university and scientiﬁc research has become increasinglyimportant in recent years. In particular, there is a growing interest in the evaluationof scientiﬁc publications and related bibliometric indicators (Marchant, 2009). Thenew criteria acquired in the university context, setting up the funding on the basisof assessments of the scientiﬁc productivity of universities and departments, as wellas regulating the career advancement of individuals assessing their research prod-ucts, require careful examination of databases available in different ﬁelds and kindsof information obtained from their query. It is important to notice that bibliometricindicators can not be self-sufﬁcient instruments of assessment, but they must be in-tegrated into more complex system of assessment; their oversimpliﬁed use, orientedto reduce the complexity of the evaluation, would have a severely negative impacton the resulting decision-making process. Despite that, the output of the databasesis the image that the international reviewers (of journals, research projects, visitingdemands and partnerships) have about the Italian statistics researchers and scientiﬁccommunity. Knowing of operational limitations about use, coverage and updatingof databases (Falagas et al, 2008), the aim of this research is to gain awareness andknowledge of the image, true or false, obtained by them: the study analyses thescientiﬁc production of all italian statistics academic scholars (SECS/S01).2 Main resultsThe databases that will be considered are:1. Current Index to Statistics (CIS), created by the American Statistical Associationand the Institute of Mathematical Statistics (http://www.statindex.org/).Francesca De Battisti,University of Milan e-mail: francesca.debattisti@unimi.itSilvia Salini,University of Milan e-mail: silvia.salini@unimi.it12 Francesca De Battisti, Silvia Salini2. Web of Science (WoS), edited by the Institute for Scientiﬁc Information anddistributed by Thomson Reuters (http://isiwebofknowledge.com/).3. Scopus, sponsored by Elsevier (www.info.scopus.com).4. Google Scholar, with recommended interface Publish or Perish, developed byAnne-Wil Harzing (http://www.harzing.com/pop.htm).By the database query, made in the period from February to April 2010, a datasetwas built, in which there are the variables: number of publications for each database,corresponding time period and, excluding CIS, number of citations and h-index(Marchant, 2009). There are also descriptive variables such as title and afﬁliation,obtained by MIUR. Table 1 shows the joint distribution of the number of publica-tions of italian researchers according to the CIS and WoS databases.Table 1 Number of publications on CIS vs Number of publications WoSWoSTotal<= 5 6 - 10 11 - 15 16 - 20 21 - 25 26+CIS<= 5 203 21 2 0 0 0 2266 - 10 71 23 5 1 1 0 10111 - 15 24 18 10 1 0 0 5316 - 20 2 8 5 5 2 1 2321 - 25 5 7 1 1 1 1 1626+ 6 1 6 4 4 4 25Total 311 78 29 12 8 6 444First of all, the SECS/S01 scholars will be classiﬁed on the basis of 10 quan-titative variables obtained from the databases, adding an additional dichotomousvariable for each person that points out whether or not the subject has publishedon the top ﬁve journals resulting from the SIS Survey1. A preliminary classiﬁcationshows that there is a group of ”better” researchers, that have high values on all vari-ables, a group of scholars who publish much but have less citations, others have a lotof papers in other ﬁelds than statistics, etc. As a second step, using data reductiontechniques, latent variables that give reason for the detected clusters, are identiﬁed:productivity, multi-disciplinarity and author impact. As ﬁnal step, the possibility tobuild a composite index, based on all dimensinos and all databases, will be criticallyevaluated.ReferencesFalagas M.E., Pitsouni E. I., Malietzis G. A. and Pappas G. (2008). Comparisonof PubMed, Scopus, Web of Science, and Google Scholar: strenghts and weak-nesses. The FASEB Journal, 22, 338-342.Marchant T. (2009). An axiomatic characterization of the ranking based on the h-index and some other bibliometric rankings of authors Scientometrics, Vol. 80,No. 2 (2009) 3273441 http://www.stat.unibo.it/ScienzeStatistiche/Ricerca/Progetti+e+attivita/Materiali Giornata di Studio -La valutazione della ricerca nelle scienze statistiche.htmBIBLIOMETRIC INDICATORS FOR STATISTICIANS: CRITICAL ASSESSMENT IN THE ITALIAN CONTEXTFrancesca De Battisti and Silvia SaliniDepartment of Economics Business and StatisticsUniversity of MilanJoint MeetingGfKl - CLADAG 2010Florence, 8 - 10 September 2010OUTLINE Introduction Bibliometric Databases Data set: the case study Data preparation Data understanding Modelling: the clusters Modelling: data reduction Conclusion  Future tasks References2DE BATTISTI, SALINIGfKl -CLADAG 2010INTRODUCTION Evaluation and bibliometric indicators: a very topical theme  What happens to the statistics? Which databases and sources are used in the field? There are several sources with different characteristics. Are the information obtained from various sources consistent? Are the indicators obtained related to each other?  Is it possible to synthesize information from different sources? 3DE BATTISTI, SALINIGfKl -CLADAG 2010BIBLIOMETRIC DATABASES1. Current Index to Statistics, created by the American Statistical Association and the Institute of Mathematical Statistics (http://www.statindex.org/) (CIS).2. Web of Science, edited by the Institute for Scientific Information and distributed by Thomson Reuters (http://isiwebofknowledge.com/) (ISI).3. Scopus, the mayor competitor of Web of Science, sponsored by Elsevier (www.info.scopus.com) (SCO). 4. Google Scholar, scientific research version of the famous search engine on the web; recommended interface for querying, which allows proper data cleaning, is Publish or Perish, developed by Anne-Wil Harzing (http://www.harzing.com/pop.htm) (POP).4DE BATTISTI, SALINIGfKl -CLADAG 2010BIBLIOMETRIC DATABASES: CIS Only publications in statistics, probability and related topics Easy query Coverage time range: since 1974 and before Not free Updating Inclusion criteria: all journals in which reported statistical papers are  Operations: query only by surname Problems:- homonymy- some input errors in the databasePLUS MINUSGfKl -CLADAG 2010DE BATTISTI, SALINI5BIBLIOMETRIC DATABASES: ISI Selective coverage of most relevant journals (and other literature sources) Update Inclusion criteria: journals that meet particular technical criteria Operations: in the query it is possible to include only the surname and the initial, or to filter by category of work or affiliation. ISI also offers the possibility, by clicking on individual works, to identify sets of work automatically created by database; but not always  Not free Not easy query Coverage time range: University of Milan license since 1990 With regard of affiliation, several problems arise:1 also the affiliations of the coauthors are reported 2 it may be missing, in which case the paper is not detected3 it may have been some mobility, so you can lose all previous works4 it can be written in many different ways Problem: homonymy PLUS MINUSGfKl-CLADAG 2010DE BATTISTI, SALINI6BIBLIOMETRIC DATABASES: SCOPUS More extensive than ISI initiative Easy query Coverage time range: papers since 1970 Update  Inclusion criteria: only journals cited monitored by Science Direct (Elsevier) Operations: query by surname and firts name, without affiliation. Then the database produces affiliation history of the author, matching name and history  Not free, but it is possible a free partial query  Coverage time range: citations since 1996 Operation problems:- homonymy- some errors in the mathing between author and affiliationPLUS MINUSGfKl -CLADAG 2010DE BATTISTI, SALINI7BIBLIOMETRIC DATABASES: POP Free Inclusion criteria: anything on the web Coverage time range: unlimited It is more extended than the databases mentioned above Not easy query It is not a database Coverage time range: unlimited Worse data qualityPLUS MINUSGfKl -CLADAG 2010DE BATTISTI, SALINI8DATA SET: THE CASE STUDY Miur: SECS/S-01 (February 2010) 444 records  Field: affiliation (campus, faculty, department, title) Npub (CIS, ISI, SCO, POP) Ncit (ISI, SCO, POP) H-index1 (ISI, SCO, POP) TOP5 Journals2 (JASA, JRSSb, Annals, Biometrika, Biometrics) 9DE BATTISTI, SALINIGfKl -CLADAG 20101A scholar obtains a value h if he has h papers with at least h citations each and the remaining (N-h) papers have no more than h citations each.2 SIS Survey presented in Bologna on March 2010DATA PREPARATION 444 total Missing values: 29 are not applicable (NA) 13 have 0 occurrences for each database (3 associate professors, 9 researchers) Outliers no point in trying univariate outliers, scholars may simply be particularly productive or unproductive than other a multivariate outlier, which is based on all available output, is represented by an unusual combination of the outputs of the 4 databases. It could be a great scholar or a data that needs a check. 10DE BATTISTI, SALINIGfKl -CLADAG 2010DATA PREPARATION Outliers Multivariate outliers detection is a way to detect anomalies and discrepancies between the databases. R Package ‘mvoutlier’ Function dd.plot  Plots the classical Mahalanobis distance of the data against the robust Mahalanobis distance based on the mcd estimator. P. Filzmoser, R.G. Garrett, C. Reimann. Multivariate outlier detection in exploration geochemistry. Computers & Geosciences, 31:579-587, 2005. Function p.cout Fast algorithm for identifying multivariate outliers in high-dimensional and/or large datasets. P. Filzmoser, R. Maronna, M. Werner. Outlier identification in high dimensions, Computational Statistics and Data Analysis, 52, 1694-1711, 2008. 11DE BATTISTI, SALINIGfKl -CLADAG 2010DATA PREPARATION Function dd.plot 12DE BATTISTI, SALINIGfKl -CLADAG 201023 outliers identified$outliers[1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE  TRUE FALSE FALSE FALSE FALSE FALSE  TRUE[17] FALSE FALSE FALSE FALSE FALSE  TRUE FALSE FALSE FALSE FALSE FALSE  TRUE FALSE FALSE FALSE FALSE[33] FALSE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE FALSE FALSE FALSE FALSE  TRUE FALSE FALSE FALSE[49] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE[65] FALSE  TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE[81] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE[97] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE[113] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE[129] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE[145] FALSE FALSE FALSE  TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE[161] FALSE FALSE FALSE FALSE FALSE  TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE[177] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE[193] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE[209] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE FALSE FALSE[225] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE FALSE FALSE FALSE FALSE FALSE[241] FALSE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE[257] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE[273] FALSE FALSE  TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE FALSE[289] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE[305] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE FALSE FALSE FALSE FALSE FALSE[321] FALSE FALSE FALSE FALSE  TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE FALSE[337] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE FALSE FALSE FALSE FALSE FALSE[353] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE[369] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE[385]  TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE[401] FALSE FALSEDATA PREPARATION Function dd.plot 13DE BATTISTI, SALINIGfKl -CLADAG 2010DATA PREPARATION Function p.cout 14DE BATTISTI, SALINIGfKl -CLADAG 2010•More than 23, for the presence of a lot of zero and for the skewness•The 23 units identified before are the ones with the highest value of distance from the scatterDATA PREPARATION 23 outliers detailed inspection of the individual records, using, if needed,also the curriculum 9 correct records: the unusual combination of the outputs is due to particular publication patterns [books (POP+) , National Statistical Journals (CIS+), disciplines with high impact  (ISI+, SCOPUS+)] Errors: 5 POP3 SCOPUS2 ISI1 CIS1 POP & SCOPUS1 POP & ISI1 SCOPUS & POP & ISI15DE BATTISTI, SALINIGfKl -CLADAG 2010 special character in name homonymy change of affiliation wrong record in the databaseDATA UNDERSTANDING16DE BATTISTI, SALINIGfKl -CLADAG 2010DATA UNDERSTANDING17DE BATTISTI, SALINIGfKl -CLADAG 2010DATA UNDERSTANDING: TOP 518DE BATTISTI, SALINIGfKl -CLADAG 2010MODELLING: THE CLUSTERS Hierarchical Algorithm Ward’s method Square Euclidean Distance19DE BATTISTI, SALINIGfKl -CLADAG 2010MODELLING: THE CLUSTER PROFILES20DE BATTISTI, SALINIGfKl -CLADAG 2010MODELLING: THE CLUSTERS21DE BATTISTI, SALINIGfKl -CLADAG 2010MODELLING: THE CLUSTER PROFILES22DE BATTISTI, SALINIGfKl -CLADAG 20101) A very big group of scholars who have low values for all indices, half of them have at most one paper on ISI, but they have more than 2 statistical papers (CIS), they attend conferences and produce working papers (POP).2) A big group of scholars that have good value for each indexes, half of them have more than 6 paper on ISI and SCOPUS and more than 8 statistical papers. Moreover they produce working paper and they attend conferences. Values for the dissemination are not very high.3) A little group of scholars whose key feature is to have very high values for productivity and dissemination for POP. By analyzing in detail, they are people who have written important books, they often participate in conferences and events as organizers, they are editors of special issues and so on. The number of papers on ISI and Scopus is lower than in Cluster 2. 4) A group of scholars who have very high values for both production and dissemination on all databases, even if the amounts of POP are lower than in Cluster 3. Probably they invest more in the journals than in the other research activities. 5) Scholars with exceptional values on all databases for both productivity and dissemination.MODELLING: THE CLUSTER PROFILES23DE BATTISTI, SALINIGfKl -CLADAG 2010MODELLING: DATA REDUCTION Item-item correlation matrix Cronbach’s alpha24DE BATTISTI, SALINIGfKl -CLADAG 2010MODELLING: DATA REDUCTION Synthetic index? 25DE BATTISTI, SALINIGfKl -CLADAG 2010CONCLUSION Using a single source is not recommended  The combined use of multiple sources helps to control the results There is no single profile of a “good researcher” It is difficult to compare because everyone makes different research choices  POP seems to measure a different dimension SCOPUS and ISI are very similar for statisticians CIS does not use selective criteria for inclusion Everyone should check his record and notify to the manager of the database what must be corrected, every database has a link / path to report errors 26DE BATTISTI, SALINIGfKl -CLADAG 2010FUTURE TASKS Comparison between the journal coverage  More information on researchers, links with  outputs, co-authors MathSciNet instead of CIS Opportunity to use data from CINECA New scientific fields, comparisons27DE BATTISTI, SALINIGfKl -CLADAG 2010REFERENCES Abramo G. (2009), Ci vuole metodo per valutare la ricerca, www.lavoce.info. Bakkalbasi N., Bauer K., Glover J. And Wang L (2006), Three options for citation tracking: Google Scholar, Scopus and Web of Science, Biomedical Digital Libraries, 2006, 3:7. Bergstrom C.T., West J.D. and Wiseman M.A. (2008), The Eigenfactor metrics, Journal of Neuroscience 28 (45), pp. 11433–11434. Biolcati-Rinaldi F. (2010), Quali indicatori bibliometrici per le scienze sociali?, Working Paper 2, Dipartimento di Studi Sociali e Politici, UNIMI.  Checchi, D. e Jappelli, T. (2008), Ricerca per indice h, www.lavoce.info. Falagas M.E., Pitsouni E. I., Malietzis G. A. and Pappas G. (2008), Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strenghts and weaknesses, The FASEB Journal, 22, 338-342. Franceschet M. (2010a), Istruzioni per l'uso della bibliometrica, www.lavoce.info. Franceschet M. (2010b), A comparison of bibliometric indicators for computer science scholars and journals on Web of Science and Google Scholar, Scientometrics, 83(1), 243-258.  Franceschet M. (2010c), The difference between popularity and prestige in the sciences and in the social sciences: a bibliometric analysis, Journal of Informetrics, 4(1), 55-63. Franceschet M (2009), A cluster analysis of scholar and journal bibliometric indicators, Journal of the American Society for Information Science and Technology,60(10), 1950-1964. Marchant T. (2009), An axiomatic characterization of the ranking based on the h-index and some other bibliometric rankings of authors, Scientometrics, Vol. 80, No. 2 (2009) 327344. Norris M. and Oppenheim C. (2007), Comparing alternatives to the Web of Science for coverage of the social sciences’ literature, Journal of Infometrics, 1 (2007), 161-169.28DE BATTISTI, SALINIGfKl -CLADAG 2010

Bibliometric indicators for statisticians : critical assessment in the Italian context

https://air.unimi.it/retrieve/handle/2434/152106/142813/deba_sal_cladag2010_u.pdf

Bibliometric indicators for statisticians : critical assessment in the Italian context

Abstract

Similar works

Full text

Available Versions

AIR Universita degli studi di Milano