Reference accuracy in library andinformation science journals
Karen DaviesSchool of Information Studies, University of Wisconsin Milwaukee,
Milwaukee, Wisconsin, USA
Purpose This paper aims to determine the percentage of reference errors and type of errors in fourlibrary and information science (LIS) journals.
Design/methodology/approach Research articles from issues published in 2007 were selectedfor analysis. The references were compared to online freely available tables of contents. The errorsidentified were categorised into six elements: journal title; author(s); article title; publication year;volume; and page numbers.
Findings The highest percentage of reference errors was 49.1 percent (Information andManagement). The overall error rates were: author (56 percent), page number (22 percent), article title(15 percent), volume (3 percent), publication year (2 percent) and journal title (2 percent).
Research limitations/implications The tables of contents (ToCs) used to compare the articlereferences may not have correctly recorded the article details. Not all journal references could bereviewed as not all journal ToCs are available online. This one year, 2007, may not accurately reflectthe citation accuracy of the journal in other years. This study did not differentiate between errors thatwould prevent the location of the article and those that could still be located with perseverance.
Practical implications Error rates in these LIS journals are considerable. The current method ofauthors being responsible for the references is not resulting in accurate bibliographic information.
Originality/value Based on the findings, possible solutions are suggested that could improve theaccuracy of references.
Keywords Bibliography, Accuracy, Reference, Periodicals, Publishing, Internet
Paper type Research paper
1. IntroductionArticles published in academic journals usually contain a list of references, rangingfrom a few key articles to an extensive list of over a hundred publications. Theaccuracy of these references is important for a variety of reasons. The author of thepaper cites others to illustrate their knowledge and understanding of the topic. Thisadds credibility to their work. The reader uses these references to locate furtherresources and to develop their understanding of the area. In academic circles accuratereferences are important, as the frequency with which work is cited often impactspersonal standing and can have positive implications when applying for grants andscholarships.
However, previous research has found errors in the references in published journalarticles. An article published in Journal of the American Medical InformaticsAssociation found that 34.3 percent of the eligible references in five biomedicalinformatics journals contained at least one error (Aronsky et al., 2005). This study alsoreviews the references of a journal from that study ( Journal of the American MedicalInformatics Association) as well as three other information system journals (MISQuarterly, Information Systems Research, and Information and Management).
The current issue and full text archive of this journal is available at
Referenceaccuracy in LIS
Aslib Proceedings: New InformationPerspectives
Vol. 64 No. 4, 2012pp. 373-387
q Emerald Group Publishing Limited0001-253X
The aim of this study is to determine the reference accuracy of the top ranked LIS(Library and Information Science) ranked journals by impact factor (as taken from theThomson scientific database Journal Citation Reports for 2006), which actually allfocus on information systems/IT. Focusing on journals within the same subject areaenables direct analysis and comparison between these journals.
2. Background and context to the studyAccurate references enable readers to locate additional publications that are potentiallyrelevant to them. Minor errors with a reference should not be a major hindrance.However errors in the journal title, author(s), article title, year of publication, volume orfirst page number can hinder their successful location and subsequent retrieval. Errorsin the reference may mean successful retrieval of the reference is time-consuming andfrustrating. Therefore citing references accurately is critically important.
Masters (2005) published a case study on the misquoting of Michaels Gibbons fromthe original source to significant documents and journal articles, but whilst thesereferences refer to the original the work cited hardly resembles the original document.This article highlights the importance of securing the original documents if these are tobe cited and not relying on the interpretation of them by another author. This can leadto incorrect quotations as well as incorrect citations.
Inaccurate references may lead readers to wonder about the care and attention givento the writing and presentation of the article, including the actual research findingsdetailed in that publication. Inaccurate references reflect poorly on the author(s), and tosome extent the journal in which the article is published.
3. Literature reviewGatten (2010) defined the reference list as the bibliography at the end and thereference as an individual entry found on a reference list.
The review of the literature on references focused on articles published since 2000.This corresponds with a shift in LIS from locating information for the user touser-empowerment with users utilizing technology and online databases to locatejournal articles for themselves rather than through an intermediary (the librarian).
Many previous studies only evaluated references from one journal issue andidentified that this may not reflect the journals true error rate (Holt et al., 2000; Siebers,2000; Siebers and Holt, 2000; OConnor and Kristof, 2001; Siebers, 2001; Lukic et al.,2004). References were also randomly selected from journal issues published in oneparticular year (Faunce and Job, 2001; Al-Benna et al., 2009) or five years (Lok et al.,2001; OConnor, 2002; Raja and Cooper, 2006). This research does not utilizerandomization; instead, a complete years worth of issues for each journal wereanalyzed.
Many studies compared the references to the original article for accuracy(Sutherland et al., 2000; Faunce and Job, 2001; Lok et al., 2001; OConnor and Kristof,2001; Lukic et al., 2004). A few studies compared the references to other resources, suchas PubMed/Medline (Holt et al., 2000; Siebers and Holt, 2000; Siebers, 2001) orMedline/EMBASE/CINAHL (Raja and Cooper, 2006) or Medline/Web ofScience/CINAHL (Siebers, 2000). This research utilizes the electronic table ofcontents for the journals, which is not a methodology that has been utilized for multiplejournals in any previous research. The only previous research that utilized online
tables of contents did so for references for one journal only, Occupational andEnvironmental Medicine (Gehanno et al., 2005).
There has been little research into the accuracy of citation databases. Buchanan(2006) did find that there were data-entry errors in Science Citation Index Expandedand SciFinder Scholar.
Studies on citation analysis have often been conducted by two or more researchers(Aronsky et al., 2005; Al-Benna et al., 2009). Intercoder reliability is a measure of theagreement between multiple coders for how they apply codes to text data, and it isfrequently used in content analysis research where text in the form of sentences andparagraphs are being analyzed. The analysis of reference citations is morestraightforward, as this is comparing the cited reference to the information inanother place, such as the original journal article or, as in this case, the electronic tableof contents for the journal. Other studies have also been undertaken by a singleresearcher (Siebers, 2000; Siebers, 2001; OConnor, 2002; Wyles, 2004; Gatten, 2010).
There is a substantial range in the reference errors in published journal articlesreported in previous research, from 43 percent to 11.1 percent. These studies analyzed arange of subject areas:
. nursing, 43 percent (Lok et al., 2001);
. business and economics, 41.7 percent (OConnor and Kristof, 2001);
. biomedical informatics, 34.3 percent (Aronsky et al., 2005);
. experimental psychology, 32 percent (Faunce and Job, 2001);
. anatomy, 27.5 percent (Lukic et al., 2004);
. clinical chemistry, 25.3 percent (Siebers, 2001);
. emergency medicine, 19 percent (Raja and Cooper, 2006);
. burns, 13.7 percent (Al-Benna et al., 2009); and
. general surgery, 11.1 percent (Reddy et al., 2008).
However, direct comparison between these results is problematic due to the differentresearch methods adopted, particularly with regards to actual identification of areference error.
Previous studies have differentiated between major (or grade I) and minor (or gradeII) errors. Major errors were considered those that prevented or at least delayed theretrieval of the article and included the first page number, journal title and year(Sutherland et al., 2000; Gosling et al., 2004; Lukic et al., 2004; Boya et al., 2008).However, Wyles (2004) suggests that resources such as PubMeds citation checker thatallow users to input the author name and year, means that if the author is not correctthat the item may not successfully be retrieved, so this could now be considered amajor error.
Table I illustrates the results for seven studies, showing that 85 percent of thepapers (six) found the most errors in the author field. The remaining articles reportedthe most errors with the article title. Errors were least reported in all seven studies inthe year of publication.
OConnor and Kristof (2001) found that only two of the 92 articles examined in theirresearch were completely error-free, whilst the study of Aronsky et al. (2005) found thatsix (16 percent) of biomedical informatics articles contained no errors in the references.
Referenceaccuracy in LIS
Table I.Distribution of errors inpublished journal articlesby percentage (rankedorder in parentheses)
There has been previous research in LIS journals. Pope (1992) analyzed the accuracy often library science journals. In that research 26 percent of references had omitted issueinformation but this was not classified as an error, whereas incorrect or omitted issueinformation is considered an error in this research. Pandit (1993) published an articlecomparing five library science journals. The citation error rate ranged from 31.6percent (Library Resources & Technical Services) to 3.8 percent (Library Trends).
Meyer (2008) wrote an article that highlighted the steps publishers have been takingto address the issue of reference accuracy. Best practices identified include:
. checking the reference quality at peer review stage;
. separating the routine processes, such as citation checking from the other roles ofthe journal editor;
. incorporating automation into the reference process; and
. utilizing standards.
4. Research questionsThis research aimed to answer three research questions:
(1) What percentage of reference errors occur in the four top-impact factor rankedinformation science and library science journals?
(2) What types of errors occur most often?
(3) What types of errors occur least frequently?
5. Methods adopted5.1 Data collectionOnly original research articles were analyzed. Researchers hope to be cited by others,and to receive credit for this the references should be accurate. It would seem to belogical to assume that they would endeavor to ensure that the references they citewould therefore also be accurate. However, the error rates identified in the literaturereview suggest that this is not necessarily the case.
The journal impact factor is the average number of times that articles in a particularjournal are referenced by other articles. Thomson Scientific produces annual JournalCitation Reports ( JCR) that are based on a two-year impact factor. The 2006 JCR ImpactFactors were calculated by dividing the number of citations received in 2006 to articles(published in 2004 and 2005) in Journal A by the number of articles published inJournal A (in 2004 and 2005). The citation data does not refer to all journal publications,but only those indexed by Thomson Scientific (Cross, 2008).
The journals to be reviewed were selected by using the 2006 Journal Citation Report( JCR) Social Science from the ISI Web of Knowledge. Those journals with an impactfactor (IF) of 2 and above were selected for analysis. These were:
. MIS Quarterly (IF 4.731);
. Journal of the American Medical Informatics Association (IF 3.979);
. Information Systems Research (IF 2.537); and
. Information and Management (IF 2.119).
Referenceaccuracy in LIS
All of these journals stress their commitment to research and its dissemination. Journalof the American Medical Informatics Association includes articles that describe keyperspectives, novel applications, and leading-edge original investigations in the field(see http://jamia.bmj.com/site/about/index.xhtml). The Information Systems Researchwebsite states that it is a leading international journal of theory, research, andintellectual development, focused on information systems in organizations,institutions, the economy, and society (see www.informs.org/Journal/ISR).
References from research articles from the 2007 issues of these four journals wereselected. This study excluded references to books, book chapters, conferenceproceedings, abstracts, editorials, reviews, technical reports, letters, dissertations,theses, websites, manuals, government documents, monographs, newspaper articles,dictionaries, policy statements, personal communications or guidelines. Onlyreferences to published journal articles were reviewed in this study.
5.2 Data analysisOnline freely available tables of contents were utilized for the analysis. Not all journalarticle references could be verified. Some journals were only indexed recently, forexample from 1997 (Economic Policy, see www.economic-policy.org/backissues.asp;European Journal of Clinical Nutrition, see www.nature.com/ejcn/archive/index.html;Family Medicine, see www.stfm.org/fmhub/), from 2000 (American Journal of ClinicalPathology,...