Annals of Tourism Research 50 (2015) 19–21

Contents lists available at ScienceDirect

Annals of Tourism Research

journal homepage: www.elsevier .com/locate/atoures

Research Note

Accuracy of impact factors in tourism journals

http://dx.doi.org/10.1016/j.annals.2014.10.0040160-7383/� 2014 Elsevier Ltd. All rights reserved.

⇑ Corresponding author. Tel.: +852 3400 2181; fax: +852 2362 9362.E-mail addresses: hmroblaw@polyu.edu.hk (R. Law), gang.li@deakin.edu.au (G. Li).

1 Tel.: +61 (3) 925 17434; fax: +61 (3) 925 17604.

Rob Law a,⇑, Gang Li b,1

a The Hong Kong Polytechnic University, Hong Kongb Deakin University, Australia

The impact factor (IF) introduced by Eugene Garfield is a measure to evaluate the performance of ajournal based on the citations received in selected publication outlets. The citation frequency of ajournal indicates its perceived prestige and status in the academic community (Glanzel & Moed,2002). IF is used for a variety of purposes. For instance, it is used by librarians as a criterion forselecting library collections. IF is also used by researchers to decide where they will submit theirresearch manuscripts. IF also serves as an evaluation tool by institutions for recruitment, promotion,research grant allocation, and project funding (Law, 2012). IF has become a popular measure forbibliometrics, but reservations about the accuracy of this indicator exist (Moed & Van Leeuwen, 1996).

According to Garfield (1999), the numerator of IF in a year is the number of citations received in thecurrent year for published articles in the journal in the preceding two years. The denominator is thenumber of citable (or substantive) articles published in the same two years. A fundamental issue of IFis the decision of what to include in the numerator and denominator. According to Garfield’s definitionof IF, the numerator includes citations to any items published in the journal. However, thedenominator only includes citable/substantive articles, including research articles and review articles.

Garfield (2006, p. 91) stated that, ‘‘correspondence, letters, commentaries, perspectives, newsstories, obituaries, editorials, interviews, and tributes are not included in the JCR’s denominator.’’ Thiscalculation leads to two major problems. First, the concept of citable/substantive articles is not clearlydefined by Thomson Reuters. Prior studies have shown the IFs of some journals are inaccurate becauseof the unclear classification of citable items (Glanzel & Moed, 2002; Moed & Van Leeuwen, 1996).Different journals have different classifications of article types, but their integration to the calculationof Thomson Reuters, and whether an article is citable or not, is based on human judgment. An exampleof this is the Annals of Tourism Research (ATR). According to ScienceDirect (www.sciencedirect.com),the articles published in 2009 are classified into article, review, editorial, and correction. Inconsistencyexists for categorization in many journals. Rosser, Van Epps, and Hill (2007) indicated that publisherscan actually negotiate with Thomson Reuters on the number of citable articles, but the negotiationdetails are not available to the public. This situation raises a concern on the data integrity of IF.Additionally, the denominator excludes certain types of articles, such as letters, editorials, andcommentaries. However, such articles are sometimes cited as well. These citations will only contribute

20 Research Note / Annals of Tourism Research 50 (2015) 19–21

to the numerator but not to the denominator. Therefore, the impact of the journal is inflated. Thedistortion may also largely affect the accuracy of IF of tourism journals because the pool of citationsof tourism is smaller than that of other disciplines, such as medicine or science. Law (2012) stated thatthe top researcher in medicine received 155,000+ citations on Google Scholar’s ‘‘My Citations,’’whereas the corresponding number for the top researcher in tourism was only 6,000+.

To date, no published work has audited the IFs for tourism journals. Thus, the present study inves-tigates whether the number of citable articles used in the denominator of IF computation is accuratelyand consistently classified in tourism journals using IF data from 2011 to 2012 and publications from2009 to 2011 for verification.

The citation report generated from Web of Science (Journal Citation Reports, 2014), which is theresearch platform of Thomson Reuters, was used to collect the citation number for calculating IFs.Thomson Reuters announces IFs of its covered journals on an annual basis.

Four Social Science Citation Index (SSCI) listed tourism journals are available on ScienceDirect.com,a major full-text online database operated by Elsevier, including the following: ATR, TourismManagement (TM), International Journal of Hospitality Management (IJHM), and Journal ofHospitality, Leisure, Sport & Tourism Education (JHLSTE). JHLSTE was excluded for further analysisbecause ScienceDirect only carried publications starting from 2012.

Citation reports from 2008 to 2011 of the four journals were archived from April to May 2014 tocalculate the IFs of 2011 and 2012. The article-type designation on ScienceDirect, which comprisesdifferent types of articles, was used in determining citable articles because Thomson Reuters didnot make available the definition of citable articles. Table 1 displays the IF values of the three journals.The number of citation was collected from the citation report on Web of Science. Number of citablearticles was computed by dividing IF by total citation of the year.

The breakdowns of each journal are shown in Tables 2 to 4 according to the classification of articlesby ScienceDirect. These papers should be used to citable/substantive articles.

Theoretically speaking, the number of citable articles published in ATR in 2009 and 2010 should be88. Unfortunately, the total number of articles, research notes and reports, and review paperspublished in the period was 97 (see Table 2). Similarly, the expected number of citable articlespublished in ATR in 2010 and 2011 should be 142, but the total number of articles, researchnotes and reports, and review papers is 146. In other words, the expected numbers of citable articlesare smaller than the computed numbers based on data from ScienceDirect.

In TM, articles, research notes, case studies, progress in tourism management, and discussions areoriginal research articles or review articles and therefore should be classified as citable/substantivearticles. However, the number of these articles was 184 (or 185 if opinion is included) for the years

Table 2Breakdown of Annals of Tourism Research.

Articles Research Notes andReports

Publications inReview

Rejoinder andCommentary

Correction Editorial ReviewPapers

2009 29 8 27 2 1 0 02010 52 8 29 1 0 0 02011 67 15 38 0 1 1 4

Table 1Impact factor and expected article number (decimal points were truncated).

Number ofCitations in2011

IF in2011

Number ofCitable Articles

Number ofCitations in2012

IF in2012

Number ofCitable Articles

Annals of Tourism Research 288 3.259 88 523 3.683 142Tourism Management 520 2.597 200 654 2.571 254International Journal of

Hospitality Management298 1.771 168 332 1.692 196

Table 3Breakdown of Tourism Management.

Articles CaseStudy

ResearchNotes

Progress in TourismManagement

Discussion OpinionPiece

Editorial BookReview

Obituary

2009 81 1 0 7 0 1 1 34 02010 84 5 2 4 0 0 0 33 12011 126 11 11 3 4 0 0 44 1

Table 4Breakdown of International Journal of Hospitality Management.

Refereed Papers Editorial Research Notes Short Communications Reviews Erratum

2009 70 4 0 9 0 12010 78 4 0 7 2 02011 110 4 3 0 1 0

Research Note / Annals of Tourism Research 50 (2015) 19–21 21

2009 and 2010, and 250 for the years 2010 and 2011 (Table 3). The numbers are smaller than theexpected number of citable articles (200 and 254, respectively).

In IJHM, the number of refereed papers, reviews, short communications, and research notespublished in 2009 and 2010 was 166, and 201 in 2010 and 2011. These figures are different fromthe computed number of citable articles, as shown in Table 1.

The findings of this study show that none of the numbers of citable articles in Table 1 matches thecomputed numbers of citable articles based on the data from ScienceDirect. Apparently, the citationnumbers and/or number of citable articles used for calculating IFs were different from the data avail-able to the public. The discrepancies are likely due to the differences in data used. Another possibilityfor the discrepancy is that ScienceDirect used a categorization that is different from that used byThomson Reuters, and that Thomson Reuters used a subjective and inconsistent way of categorization.Drawing on the findings of this study, Thomson Reuters could, and probably should, publish theircategorization approach to make their IFs more credible.

Considering the questionable validity of the published IFs in tourism journals, the values may notbe effective in evaluating the overall prestige of journals. Thus, administrators and researchers intourism should not rely solely on IFs to judge the quality of journals. The findings of this study showthat IFs should be used with caution.

Acknowledgement

The work described in this paper was supported by a grant funded by the Research Grants Councilof the Hong Kong Special Administrative Region, China (GRF Project Number: 15503814B-Q45Y).

References

Garfield, E. (1999). Journal impact factor: A brief review. Canadian Medical Association Journal, 161(8), 979–980.Garfield, E. (2006). The history and meaning of the journal impact factor. Journal of the American Medical Association, 295(1),

90–93.Glanzel, W., & Moed, F. H. (2002). Journal impact measures in bibliometric research. Scientometrics, 53(2), 171–193.Journal Citation Reports (2014). Retrieved 14.10.14 from <http://thomsonreuters.com/journal-citation-reports/>.Law, R. (2012). The usefulness of impact factors to tourism journals. Annals of Tourism Research, 39(3), 1722–1724.Moed, H. F., & Van Leeuwen, T. N. (1996). Impact factors can mislead. Nature, 381, 186.Rosser, M., Van Epps, H., & Hill, E. (2007). Show me the data. Journal of Cell Biology, 17(6), 1091–1092

Received 10 May 2014; Revised 15 October 2014; Accepted 16 October 2014

Available online xxxx