The impact profile of recent UK geoscience

research is set to change, say Nick Petford

& Jonathan Adam

Adams, J.

Petford, Nick

English

The impact profile of recent UK geoscience research is set to change, say Nick Petford* &amp; Jonathan Adams

Adams, Jonathan

University of Northampton's Research Explorer

Goalposts on the move

Bournemouth University Research Online

Contents4 Geonews News items relevant to the interests ofworking geologists6 People  Geoscientists in the news and on the move8 Society at large What your Society is doing at home andabroad, in London and the regions16 Opinion Reviews and letters28 Calendar A forward plan of Society activities30 CrosswordWin a special publication of your choiceVolume 18  •  No. 2 •  318 Opinion - Goalposts on the move - by Nick Petford and Jonathan Adams22 Feature - The wrong sort of rain by Michael PriceBeating the systemWill Earth System Scienceprove a saviour for someUK university geosciencedepartments in the newworld of research assessment?Nick Petford has been gazinginto the Government’s newestcrystal ball…Most readers will know by now that theGovernment is seeking to change theway research funding is allocated to UKuniversities following the next (2008)Research Assessment Exercise.  Instead of using the currentsystem of peer review, research outputs in the STEM subjects(science, technology engineering and mathematics) will beanalysed and ranked using a basket of metrics.The closing date for HEFCE consultation on this matter is 14 February 2008 (see http://www.hefce.ac.uk/Research/assessment/reform/).  A key indicator of research prowesswill be the number of times an individual article is cited byother researchers. The notion of citation counts as a measureof research impact has some degree of international currency,but has also been criticised as a blunt instrument at best. However, all this looks set to change as new, more sophisticatedmethodologies are used to unpack the statistical detailbehind bibliometric analysis. Key to this new world ofbibliometrics is the concept of citation profiling, under-pinned by a  renormalising technique called ReBased ImpactSoapbox – Geoscientist would like to offer the opportunity to write for this leaderpage to all Fellows. You can always write a letter to the Editor, of course; but perhapsyou feel you need more space? If you can write it entertainingly in 500 words, theEditor would like to hear from you. Email your piece, and a self-portrait, toted.nield@geolsoc.org.uk. Copy can only be accepted electronically. No diagrams, tablesor other illustrations please.(RBI), that scores each journal article by discipline andyear of publication against a world benchmark. The mosthighly cited (hence influential) journal papers have anRBI >8.  RBI can be calculated to provide a benchmarkedprofile of research activity at national, university,research group or individual level. It is a simple yetextremely powerful tool for analysing research output,and is coming to a Vice Chancellor near you soon. More detail behind this new citation analysis, which wasdevised by researchers at Evidence Ltd, a spin-outcompany from the University of Leeds advising theGovernment on metrics post-RAE2008 can be foundon page 18. The good news is that with a mean score of  1.3, the UKGeoscience RBI exceeds the world average over the period1995-2004. However, the analysis also shows that sometwo thirds of all UK Geoscience papers over the sameperiod were either never cited or fell below the world average.Moreover, given the current broad classification of“Geosciences” as defined by Thomson ISI, the analysishints that the largest number of highest scoring outputs(RBI > 8) may be in the area of meteorology/atmosphericsciences - subjects not generally thought of as residing inthe Geoscience stable. Although provisional, an earlyimplication is that those university departments, eitherthrough planning or sheer good fortune, who chose toembrace a highly interdisciplinary (Earth SystemsScience-plus) view of Geoscience, may stand to benefitmost from bibliometric-informed research funding thatwill follow the 2008 RAE.18 GeoscientistopinionThe impact profile of recent UK geoscienceresearch is set to change, say Nick Petford* & Jonathan Adams#It is generally accepted that the number of citations a paperreceives is useful measure of research impact1.  Indeed,citation counts, renormalised to a common benchmark,provide compelling evidence that UK science is truly worldclass - second only to the USA in terms of citationimpact2. In an article published recently in the journalScientometrics, researchers at Evidence Ltd, a consultancyspecialising in research performance analysis(www.evidence.co.uk), have added a new layer ofsophistication to the citation analysis game. The paper3looks in detail at UK bibliometric journal impact profilesfrom 1995 to 2004, and presents a new methodology withimportant implications for the way citations might be usedto assess research performance in the metrics-driven worldset to follow the 2008 Research Assessment Exercise4.Rather than reporting the average citation rate (for aresearch group or individual) as a measure of researchimpact, as is common practice, their analysis takes intoaccount research significance by profiling and comparingimpact factor distributions.  For example, while citationsper paper are clearly and demonstrably related to impact(and easy to understand) they absorb outlier values andhide the fact that research performance data of any kindare rarely normally distributed. Where data are skewed,the “average” can differ significantly from the median andmode.  This means that that a static value of average impactfactor (citations per paper, itself a ratio), while useful as ameasure of research quality, will not paint a full picture ofresearch activity and could even be misleading5. Citation profiles and Rebased impactIn order to move away from reporting a single value as aproxy for quality, the Evidence team have developed a methodbased on citation profiles. The primary data underpinningthe analysis were drawn from 750,376 individual UK journalarticles covering the major science technology engineeringand mathematics (STEM) subjects, held on the ThomsonScientific Inc. database. Using total UK publication datacovering a 10-year period, the team split citations perpaper into categories based on a reference value called theRebased Impact (RBI) index.  This normalises citationcounts to the world average for the year and disciplinecategory of publication.  In this model, an uncited articlehas an RBI of 0, while an RBI >1 means that it is above theworld average. An RBI of 0.5 is a paper with normalisedcitation count half world average while RBI = 2 is twiceworld average.  An exceptional RBI >8 corresponds to thetop cited 1-2% UK research papers and overlaps withThomson ‘Highly Cited’ papers that comprise the top 1%in a given field. Goalposts on the Results Using the this method, it is possible to analyse differencesin performance by subject area nationally, by university andeven at research centre level. Table 1 is a summary of researchimpact profiles by journal category that equated broadly tosubject discipline, drawn from Adams et al., (2007).The category marked Geosciences contains 22,939 UKoutputs over the period 1995-2004 along with the percentageof those outputs categorised by RBI along with % worldaverage. Also included for comparison are data from thejournal categories for Chemistry and Physics. These threesubject areas together comprise Main Panel D in RAE2008and thus make natural comparators for the analysis presentedhere (and Fig 1).Table 1. Summary of UK research impact (1995-2004) by Thomson Scientific subjectareas for Geosciences, Chemistry and Physics (from Adams et al., 2007). Without delving into too much detail, and ignoring for nowthe potentially thorny problem of a precise definition of"Geosciences", some broad conclusions can be drawn. Firstly,21.4% of all geoscience articles go uncited over the reviewperiod (similar to the UK average of 21.8%). Secondly, 42.2%of UK geoscience journal articles are cited but below the worldaverage (RBI <1). Similar performance in these categories ismirrored in Physics and Chemistry. When aggregated as awhole, it is apparent that while the UK RBI average is 1.33,some two-thirds of all outputs are at or below world average.This is strong evidence of the potential for misunderstandingthat would arise from superficial indices of skewed metrics,especially if those then affect funding.  For the geosciences,broadly defined, 63.6% of all publications are either nevercited or are cited less than the world rebased average.However, at the upper end of the citation scale, Geosciencescompares favourably with its Panel D comparators and thetotal UK average. For example, the percentage of outputs withRBI > 8 in Geosciences is 1.7 (UK average = 1.5), compared to1.8% for Chemistry and 3.0% for Physics. The percentage ofcitations greater than the world average in Geosciences (36.4)is only just behind Chemistry (36.5) and marginally aheadof Physics (34.2), despite the fact that these subject areasproduced nearly three times the research output over thecensus period.Journal CategoryGeosciencesChemistryPhysicsTotal UKOutput(1995-2004)229396002261205Uncited21.419.023.2RBI < 142.244.642.6RBI 1-834.734.631.2RBI > 81.71.83.0% >worldaverage36.436.534.2RBIAverage(2000-2004)1.331.231.42RBIMedian(1995-2004)0.650.630.54* University Executive Group, Bournemouth University, Fern Barrow Dorset, BH12 # Evidence Ltd, 103 Clarendon Road, Leeds, LS2 9DFVolume 18  •  No. 2 •  19The profiling methodology allows the UK GeosciencesRebased impact to be tracked over time. For this analysisEvidence has applied moving five-year windows (whichhelp to smooth annual volatility) for the period from theearly 1990s through to the most recent five years for whichdata are available. Fig. 2 shows how the RBI has risen froma value of around 1.17 around the 1996 RAE to its presentday value of 1.33. The steepest rate of change followed the2001 RAE where the RBI climbed rapidly over threeconsecutive years.  The hint of recent decline isprobably transitory.  The power of the profiling methodology is seen in Fig. 3,where the distribution of Geoscience outputs for the mostrecent 10 years are grouped into less coarse RBI bins. Datapresented in this way allow subject areas and researchperformance to be compared much more informativelyagainst national or international benchmarks than simpleaverages would do. Here, it is clear that as a subject group,Geosciences outperform the UK RBI baseline over theinterval 0.5-1.0 < RBI < 2.0-4.0 and have a lower percentageof uncited papers compared to the national average.Such high-level profiling is useful for assessing the relativeresearch strength of whole subject areas, but could also beapplied to universities and research centers. Benchmarkscould be based on internal metrics or comparator organizationsworking in the same field. It is now easy to benchmark auniversity against the UK subject area as a whole. Researchmanagers will of course be looking for enhanced performanceat the upper end of the RBI scale and a lower percentage ofuncited papers compared with the benchmark value. Theycan also analyse the distribution of authorship across theirhighly cited papers.Definition of “Geosciences”But before we all get too excited, we should pause toconsider what exactly is meant by Geosciences as definedcurrently by Thomson Scientific.  The journal categoryGeosciences contains 439 journals covering a very widerange of subject matter indeed (see http://www.in-cites.com/journal-list/index.html). As well as our very own Journal ofthe Geological  Society, it also includes the entire Journal ofGeophysical Research family, and publications on remotesensing, climatology, oceanography, glaciology – nearlyeverything, in fact, to do not just with the solid Earth, butatmospheres and oceans as well. This is a very broaddefinition, with its boundaries blurred into physicalgeography, meteorology, environmental sciences andecology; single honours geology it is not! And with thiswide spread of categories comes a rather awkward question:“Which bit (if any) of Geosciences sensu lato, dominatesthe “high impact” end?” Put another way, could thesustained greater-than-average performance since 1997(Fig. 2) be due to consistently high performance in aspecific subset of Geosciences as presently defined? moveFig 1.   RBI scores for Geosciences over the period 1995-2004, along with impact profiles for Chemistry andPhysics as comparators (see Table 1). All three subject areas display similar RBI citation profiles.Fig 2. UK Geoscience average rebased impact (RBI) for a sequence of five-year windows: 1993-1997 through2002-2006. Fig 3. Profiled 1997-2006 citation impact of 24574 Geoscience papers (blue curve) compared with the total UKcitation impact (740389 papers). The distribution shows the Geosciences ‘outperform’ the UK baseline with feweruncited papers, less activity in the low citation region (RBI < 1) and higher activity where RBI > 1. At the very topend (RBI > 4-8, RBI > 8), however, the UK baseline has a slight edge. opinioncontinued on page 21Institutional ranking based on RBI metricsPerhaps one way to answer this question is to look at the top ten UKinstitutions ranked by article count where RBI >8. This is shown in Table 2for the period 1997-2006 used in Figure 3. As a proxy for those institutionsthat retain a grounding in traditional geological subjects, the table alsoshows where F600 (Undergraduate BSc Geology) is offered.Table 2. Top UK Institutions Ranked by Geoscience Article Count where RBI > 8. Two out of 8 HEIs do notoffer single honours Geology. The table includes two research institutes (the Meteorological Office and theEuropean Centre for Medium Range Forecasts) specialising in atmospheric processes. While we should not read too much into the rankings compiled this way(the table summarises data for institutions, not specific geoscience departments),some interesting points emerge.   The most immediate is that the big jumpin RBI > 8 count (24) between the élite universities and the MeteorologicalOffice points clearly to the fact that within Thomson's Geosciences category,papers in atmospheric sciences/meteorology dominate the high impactend of the spectrum. This is further supported by the fact that theEuropean Centre for Medium Range Forecasts outperforms threeUK InstitutionMeteorological OfficeUniversity of East AngliaUniversity of CambridgeUniversity of OxfordUniversity of ReadingUniversity of BristolUniversity of LeedsEuropean Centre forMedium Range ForecastsUniversity of EdinburghUniversity of ManchesterUniversity of SouthamptonArticle count (1997-2006)where RBI > 8704645 4339242423201918F600Yes/NoNoNoYesYesNoYesYesNoYesYesYesRank123456 =6 =78910References1. Garfield, E, (1955), Citation Indexes for Science: A New Dimension in Documentationthrough Association of Ideas, Science, 122, 108-111.2. King, D.A. (2004), The Scientific impact of nations, Nature, 430, 311-316.3. Adams, J., Gurney, K & Marshall, S (2007), Profiling citation impact: a new methodolo-gy, Scientometrics, 72, 325-344.4. HEFCE, (2007), Consultation on Research Excellence Framework,http://www.hefce.ac.uk/Research/assessment/reform/5. Evidence, (2007), The Use of Bibliometrics to Measure Research Quality in UK HigherEducation Institutions, Report to Universities UK , ISBN 978 1 84036 165 4.6. Laird, J.D. & Bell, R.E. (2007), EOS 88, #38, 370-371.traditional universities (Edinburgh, Manchester and Southampton). Wehave no information yet as to the specific content of these high scoringpapers, but purely in volume terms they contribute significantly to theabove world average performance of UK Geosciences. An interestingopen question is this: were the criteria for Geosciences revised toexclude meteorology (but keep oceanography), would the field stillperform consistently above the world average?Implications for UK GeoscienceAcademics can expect to see a growth in the profiling of institutions, ascollective research output is ranked increasingly against national andinternational benchmarks. This could easily be extended to researchgroup level and even the individual. Indeed, US research looking intopublication behaviour based on a sample of AGU Fellows clearly identifiesthe majority of researchers as “prolific” (meaning that their work iscited more than 21 times6). Combining these gross behavioural traitswith new and more subtle metrics as a measure of research “quality”may become the norm (especially at the high-impact end).More contentiously, institutional profiling (Table 2) suggests that giventhe current wide definition of Geosciences, the highest scoring departmentsin a metrics-driven QR funding environment will be those that have astrong atmospheric sciences/meteorology research component. Thecommunity will therefore wish to evaluate HEFCE’s proposals2 withcare and engage closely with them in debating exactly how the metricsprocess is developed, structured and weighted by discipline.  UniversityUK’s recent report5 provides useful background to this debate. Volume 18  •  No. 2 •  21

Goalposts on the Move

Abstract

Similar works

Full text

Available Versions

University of Northampton's Research Explorer

Bournemouth University Research Online

University of Northampton's Research Explorer