Rheumatology Advance Access originally published online on February 20, 2006
Rheumatology 2006 45(5):646-647; doi:10.1093/rheumatology/kel035
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
LETTER TO THE EDITOR |
Can research quality be estimated from journal titles?
Bone Research Division, Department of Medicine, University of Cambridge, Cambridge, UK.
Correspondence to: jonathan.reeve{at}srl.cam.ac.uk
SIR, In the article by Wooding et al. [1] the method used to estimate the impact of each individual paper published as the result of receiving an Arthritis Research Campaign grant is not clearly stated. However, if it was calculated indirectly from the impact factor of the journal of publication, the approach used to rank authors, I have serious concerns about the value of the information derived.
To investigate the reliability of this approach I have used a database of papers published from the (now defunct) MRC Clinical Research Centre (CRC) between 1972 and 1985, as listed in the CRC's bi-annual (later annual) reports. The data collected for each individual paper included the journal of publication, number of pages and total citations for that paper. Single-page papers were excluded as possibly being abstracts.
There were 20 journals that published more than 20 CRC papers each scoring more than 10 citations (mean 34) among several hundred journals used overall by CRC scientists, who were working in clinical and translational research relevant to many clinical disciplines. The distribution of the total citations received in the first 20 yr after publication by the 680 papers was found to be logarithmic, after subtracting nine citations from each (to correct for self-citation), the highest score being well over 1000. After log-transformation, analysis of variance was used to relate total score to journal, and journal was found to account for only 10% of the variance in the data. The residuals were exponentially distributed, so that after log10 transformation the S.D. was 0.5. This means that the estimated mean and 95% confidence interval for predicting the citation count of an individual Lancet or Nature article (net of an estimated nine self-citations) was 42 (4.2, 420) and for the lowest scoring journal it was 10 (1, 100).
Estimating confidence intervals for the data published by Wooding et al. is impossible because they quote citations received annually. The CRC data showed clearly that clinical research had a much longer citation half-life than translational research for a similar number of total citations, or in other words had less early impact that was more sustained. This is appropriate for work destined to be applied by careful clinicians, working in a framework in which patients are maximally protected. To use annual rates appropriately they have to be averaged over a substantial number of years.
These results show just how variable are the citations relating to different articles in the same journal. True, if the intention is to consider the mean impact of several articles the confidence intervals may be reduced approximately in proportion to the square root of the number of articles, and Wooding et al. were considering categories of grant each producing between 56 and 101 papers. One also has to consider uncertainties produced by the change in impact factor for journals over the passage of time, which might in part result from the submissions in question. Interestingly, for the period up to 1980, the effect of journal of publication was not a significant statistical determinant of citation count in published CRC work. One has to wonder if this situation will return because in future, as better information retrieval eases journal access, many well-known authors will feel less pressure to publish in demanding and time-consuming high-impact journals. Such high-profile scientists will be confident their work will be read, because so many searches are for the author who in the searcher's mind is attached to an idea or a result.
Overall, there seems no alternative to the careful consideration of individual papers if it is desired to judge whether previous funding decisions gave good value. For clinical research it can take a decade for a paper's importance to become clear. It is comparatively easy to manipulate averaged citation data from the first three post-publication years to generate support for basic vs clinical research, while industry-based authors are known to orchestrate impact by publishing repetitive supporting research that draws attention to previously published clinical data. When it comes to commissioning studies of publication impact, caveat emptor!
The authors have declared no conflicts of interest.
References
- Wooding S, Hanney S, Buxton M, Grant J. Payback arising from research funding: evaluation of the Arthritis Research Campaign. Rheumatology 2005;44:1145–1156.
[Abstract/Free Full Text]
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||