JSS Editorial 1605

JSS Editorial

Measuring Your Scientific ImpactMeasuring the scientific output and quality of scientistsappears to be increasingly tempting for officials of uni-versities and related institutions all over the world. Theimpact factor of journals, which is one way of counting,was discussed in the editorial of JSS 11/03 by FrantisekSvec and myself. We mentioned that articles describingnew methods are often considered as giving high cita-tions over time, exemplified by the Lowry method for pro-tein determination. In response, Eugene Garfield, thepioneer of “scientometrics”, commented that this may bea commonly held view, but is not correct, and there is noeffect at all on journal impact factors, since only twoyears of citations are included in these measurements.

But what about other citations?

Among scientists there is general agreement on the im-portance of high citation numbers as a measure of bothproductivity and importance within a field. Whether thisis equivalent to quality is more difficult to prove, but thereare few other quantitative alternatives.The commonlyused database ISI Web of Science allows us to searchfor citation numbers, number of references, identificationof the references plus other related information.

Assuming that citations are important, the question ishow we measure and express the numbers. Fortunately,databases like ISI contain most of the tools we need,although refinements may be required, and there aremany loopholes. First, do not use any numbers withoutbeing absolutely certain that there is only one person be-hind the name involved in a search. Since there is noother means of identification, the reference list must bechecked manually. As a consequence this job shouldnot be done by persons without good knowledge of thefield.

Secondly, there are errors in the database. One com-mon example is that an article is referred to slightly dif-ferently by different authors, appearing as several differ-ent references in the database. This has no effect on thetotal number of citations, only on the citations per refer-ence. Other references of questionable value are perso-nal communications, submitted articles, and/or develop-ments which are not published as articles, such as com-puting programs and different versions thereof. In the ta-ble below you can see the effect of correcting the errorsand relocating the citations not related to a published ar-ticle or book.

For purposes of illustration the personal data of 3 scien-tists (A, B, and C) were collected from the ISI database,containing citations from 1987 to date. All three are veryproductive scientists, with 1500–5000 citations. All arearound 50 years old, from three different fields of chem-istry. Could a treatment of the data for only three personstell us anything about the validity of the available citationnumbers?

A strong effect on the citations per article is noted forscientist A, after corrections. In this case the number ofreferences (publications) was halved due to the refer-ences to (available) new versions of computing pro-grams. In my opinion this does not count as a new publi-cation, but should be grouped with the other referencesto the published program. However, usually the correc-tion effects would be expected to be smaller, as notedfor scientists B and C. E

dit

ori

al

J. Sep. Sci. 2003, 26, 1605–1606 www.jss-journal.de i 2003 WILEY-VCH Verlag GmbH & Co. KGaA, Weinheim

Citations 1987–2003uncorrected

Citations 1987–2003corrected

Citations 1994–2003Publications 1993–2002

ACitations (total) 5352 5336 3462Citations/publication 14.4 29.3 26.0

BCitations (total) 1644 1608 741Citations/publication 7.8 8.2 6.6

CCitations (total) 1585 1563 534Citations/publication 22.3 24.8 17.2

1606 JSS Editorial

Thirdly, in the database the references (publications) goback to the seventies. Whether using 20–25 year oldpublications is relevant for evaluating a scientist is anopen question, depending on the purpose of the evalua-tion. More than often the activity in recent years is ofgreater interest. In the last column of the table, publica-tions from the last decade have been selected and com-bined with (corrected) citations starting one year later.The absolute numbers are naturally reduced and the ef-fect of (missing) earlier highly cited papers is also evi-dent (scientist C).

Scientist A is in a class by himself. Assuming the “error”correction is accepted, the increase in citations per pub-lication is remarkable, compared to the numbers readilyavailable. Scientist C produces fewer articles, but withhigher citation numbers than scientist B. This could bedue to longer articles, articles of wider interest, and/orjournals of higher impact. A more detailed evaluationshould be performed.

Finally, what I am trying to say in this editorial is that ifyour numbers have been used to evaluate and rate yourcapability as a scientist, you are entitled to ask how thenumbers were produced. What about error corrections,reference and citation basis (period of time), comparingfields which are incomparable etc?

Therefore, don’t take the numbers at face value withoutknowing the background.

Tyge Greibrokk

J. Sep. Sci. 2003, 26, 1605–1606 www.jss-journal.de i 2003 WILEY-VCH Verlag GmbH & Co. KGaA, Weinheim