Manual Evaluation of Robot Performance in Identifying Open Access Articles

Antelman et al. (2005) hand-tested the accuracy of the algorithm that Hajjem et al.'s (2005) software robot used to trawl the web and automatically identify Open Access (OA) and Non-Open-Access (NOA) articles (references derived from the ISI database). Antelman et al. found much lower accuracy than Hajjem et al. Had reported. Hajjem et al. have now re-done the hand-testing on a larger sample (1000) in Biology, and demonstrated that Hajjem et al.'s original estimate of the robot's accuracy was much closer to the correct one. The discrepancy was because both Antelman et al. And Hajjem et al had hand-checked a sample other than the one the robot was sampling. Our present sample, identical with what the robot saw, yielded: d' 2.62, bias 0.68, true OA 93%, false OA 12%. We also checked whether the OA citation advantage (the ratio of the average citation counts for OA articles to the average citation counts for NOA articles in the same journal/issue) was an artifact of false OA: The robot-based OA citation Advantage of OA over NOA for this sample [(OA-NOA)/NOA x 100] was 70%. We partitioned this into the ratio of the citation counts for true (93%) OA articles to the NOA articles versus the ratio of the citation counts for the false (12%) "OA" articles. The "false OA" advantage for this 12% of the articles was 33%, so there is definitely a false OA Advantage bias component in our results. However, the true OA advantage, for 93% of the articles, was 77%. So in fact, we are underestimating the true OA advantage

Self-selected or mandated, open access increases citation impact for higher quality research

Articles whose authors make them Open Access (OA) by self-archiving them online are cited significantly more than articles accessible only to subscribers. Some have suggested that this "OA Advantage" may not be causal but just a self-selection bias, because authors preferentially make higher-quality articles OA. To test this we compared self-selective self-archiving with mandatory self-archiving for a sample of 27,197 articles published 2002-2006 in 1,984 journals. The OA Advantage proved just as high for both. Logistic regression showed that the advantage is independent of other correlates of citations (article age; journal impact factor; number of co-authors, references or pages; field; article type; country or institution) and greatest for the most highly cited articles. The OA Advantage is real, independent and causal, but skewed. Its size is indeed correlated with quality, just as citations themselves are (the top 20% of articles receive about 80% of all citations). The advantage is greater for the more citeable articles, not because of a quality bias from authors self-selecting what to make OA, but because of a quality advantage, from users self-selecting what to use and cite, freed by OA from the constraints of selective accessibility to subscribers only. [See accompanying RTF file for responses to feedback. Four PDF files provide Supplementary Analysis.

Repositories for Institutional Open Access: Mandated Deposit Policies

Only 15% of articles are currently being made Open Access (OA) through spontaneous self-archiving efforts by their authors. They average 25%-250% more citations in all 12 disciplines tested so far. Ninety-four percent of journals endorse immediate OA self-archiving. There is no evidence that self-archiving induces subscription cancellations. The “OA advantage” consists of: Early Advantage (early self-archiving produces both earlier and more citations), Usage Advantage (more downloads for OA articles, correlated with later citations), Competitive Advantage (relative citation advantage of OA over non-OA articles: disappears at 100% OA), Quality Advantage (OA advantage is higher, the higher the quality of the article) and Quality Bias (authors selectively self-archiving their higher quality articles – a non-causal component: disappears at 100% OA). We are currently comparing the OA advantage for mandated and spontaneous (self-selected) self-archiving. Deposit rates in Institutional Repositories (IRs) remain at 15% if unmandated, but climb toward 100% OA if mandated, confirming surveys that predicted 95% compliance. In the UK, 4 of the 8 research funding councils and the Wellcome Trust mandate self-archiving and it is being considered by the European Commission and the US federal FRPAA. There is no reason for universities to wait for the passage of the legislation. Five universities and two research institutions (including CERN) have already mandated it, with documented success. An Immediate-Deposit/Optional-Access Mandate covers all cases and moots all legal issues: metadata are immediately visible webwide and, where needed, access to the postprint can be set as Closed Access instead of OA throughout any embargo period. Software to support this approach (that allows the author to email individual copies of non-Open Access papers to individual requesters) has been created for both EPrints and DSpace repository platforms

The Open Access Citation Advantage: Quality Advantage Or Quality Bias?

Many studies have now reported the positive correlation between Open Access (OA) self-archiving and citation counts ("OA Advantage," OAA). But does this OAA occur because (QB) authors are more likely to self-selectively self-archive articles that are more likely to be cited (self-selection "Quality Bias": QB)? or because (QA) articles that are self-archived are more likely to be cited ("Quality Advantage": QA)? The probable answer is both. Three studies [by (i) Kurtz and co-workers in astrophysics, (ii) Moed in condensed matter physics, and (iii) Davis & Fromerth in mathematics] had reported the OAA to be due to QB [plus Early Advantage, EA, from self-archiving the preprint before publication, in (i) and (ii)] rather than QA. These three fields, however, (1) have less of a postprint access problem than most other fields and (i) and (ii) also happen to be among the minority of fields that (2) make heavy use of prepublication preprints. Chawki Hajjem has now analyzed preliminary evidence based on over 100,000 articles from multiple fields, comparing self-selected self-archiving with mandated self-archiving to estimate the contributions of QB and QA to the OAA. Both factors contribute, and the contribution of QA is greater

Citation Advantage For OA Self-Archiving Is Independent of Journal Impact Factor, Article Age, and Number of Co-Authors

Eysenbach has suggested that the OA (Green) self-archiving advantage might just be an artifact of potential uncontrolled confounding factors such as article age (older articles may be both more cited and more likely to be self-archived), number of authors (articles with more authors might be more cited and more self-archived), subject matter (the subjects that are cited more, self-archive more), country (same thing), number of authors, citation counts of authors, etc. Chawki Hajjem (doctoral candidate, UQaM) had already shown that the OA advantage was present in all cases when articles were analysed separately by age, subject matter or country. He has now done a multiple regression analysis jointly testing (1) article age, (2) journal impact factor, (3) number of authors, and (4) OA self-archiving as separate factors for 442,750 articles in 576 (biomedical) journals across 11 years, and has shown that each of the four factors contributes an independent, statistically significant increment to the citation counts. The OA-self-archiving advantage remains a robust, independent factor. Having successfully responded to his challenge, we now challenge Eysenbach to demonstrate -- by testing a sufficiently broad and representative sample of journals at all levels of the journal quality, visibility and prestige hierarchy -- that his finding of a citation advantage for Gold OA (articles published OA on the high-profile website of the only journal he tested (PNAS) over Green OA articles in the same journal (self-archived on the author's website) was not just an artifact of having tested only one very high-profile journal

The Self-Archiving Impact Advantage: Quality Advantage or Quality Bias?

In astrophysics, Kurtz found that articles that were self-archived by their authors in Arxiv were downloaded and cited twice as much as those that were not. He traced this enhanced citation impact to two factors: (1) Early Access (EA): The self-archived preprint was accessible earlier than the publisher's version (which is accessible to all research-active astrophysicists as soon as it is published, thanks to Kurtz's ADS system). (Hajjem, however, found that in other fields, which self-archive only published postprints and do have accessibility/affordability problems with the publisher's version, self-archived articles still have enhanced citation impact.) Kurtz's second factor was: (2) Quality Bias (QB), a selective tendency for higher quality articles to be preferentially self-archived by their authors, as inferred from the fact that the proportion of self-archived articles turns out to be higher among the more highly cited articles. (The very same finding is of course equally interpretable as (3) Quality Advantage (QA), a tendency for higher quality articles to benefit more than lower quality articles from being self-archived.) In condensed-matter physics, Moed has confirmed that the impact advantage occurs early (within 1-3 years of publication). After article-age is adjusted to reflect the date of deposit rather than the date of publication, the enhanced impact of self-archived articles is again interpretable as QB, with articles by more highly cited authors (based only on their non-archived articles) tending to be self-archived more. (But since the citation counts for authors and for their articles are correlated, one would expect much the same outcome from QA too.) The only way to test QA vs. QB is to compare the impact of self-selected self-archiving with mandated self-archiving (and no self-archiving). (The outcome is likely to be that both QA and QB contribute, along with EA, to the impact advantage.

Evaluation of Algorithm Performance on Identifying OA

This is a signal-detection analysis of the accuracy of a robot in detecting open access (OA) articles (by checking by hand how many of the articles the robot tagged OA were really OA, and vice versa). We found that the robot significantly overcodes for OA. In our Biology sample, 40% of identified OA was in fact OA. In our Sociology sample, only 18% of identified OA was in fact OA. Missed OA was lower: 12% in Biology and 14% in Sociology. The sources of the error are impossible to determine from the present data, since the algorithm did not capture URL's for documents identified as OA. In conclusion, the robot is not yet performing at a desirable level, and future work may be needed to determine the causes, and improve the algorithm

The Access/Impact Problem and the Green and Gold Roads to Open Access: An Update

The research access/impact problem arises because journal articles are not accessible to all of their would-be users; hence, they are losing potential research impact. The solution is to make all articles open access (OA, i.e., accessible online, free for all). OA articles have significantly higher citation impact than non-OA articles. There are two roads to OA: the “golden” road (publish your article in an OA journal) and the “green” road (publish your article in a non-OA journal but also self-archive it in an OA archive). About 10% of journals are gold, but over 90% are already green (i.e., they have given their authors the green light to self-archive); yet only about 10–20% of articles have been self-archived. To reach 100% OA, self-archiving needs to be mandated by researchers’ employers and funders, as they are now increasingly beginning to d

Open Access to Research Increases Citation Impact

We analyzed the effect of providing 'Open Access' (OA; free online access to research articles) on their 'citation impact' (how often they are cited). Using a subset of the ISI CD-ROM database from 1992 - 2003, we compared, within each journal and year, articles to which their authors had (OA) or had not (NOA) provided open access by self-archiving them on the web. The number of OA and NOA articles and their respective citation counts were calculated within biology, business, psychology and sociology journals. The percentage of OA articles varied from 5-20% (mean and median, 12%). The citation counts (OA-NOA/NOA) showed a consistent OA advantage (mean 96%, median 73%) for all four fields and 28 subspecialties tested, varying from 25% to over 250%. An OA impact advantage has already been reported in the physical sciences and engineering (physics, computer science), but there was uncertainty about whether the same thing happens in other disciplines. Our data now show that both the biological and the social sciences show the OA advantage, and are hence likewise losing substantial amounts of potential impact for the 80-95% of their articles that are not yet self-archived. These results confirm that a mandatory self-archiving policy on the part of research institutions and funders would greatly enhance the impact of research results in all disciplines