Download pdf - Measuring characteristics of scientific research: A comparison of bibliographic and survey data

Scientometrics, VoL 24. No. 2 (1992) 359-370

M E A S U R I N G C H A R A C T E R I S T I C S O F S C I E N T I F I C

R E S E A R C H :

A C O M P A R I S O N O F B I B L I O G R A P H I C A N D S U R V E Y D A T A

H.H. GARRISON,* S.S. HERMAN,* J.A. LIPTON** �9

* Aspen Systems Corp., 962 Wayne Ave., Silver Spring~ MD 20910 (USA) ** National Institute of Dental Res., NIH, Bethesda, MD 20892 (USA)

(Received May 13, 1991)

Three characteristics of scientific research (subject matter, researchers' institutional sectors, and funding sources) were compared using bibliographic and survey data from a study of restorative dental materials research. Both types of data yielded similar findings on the distribution of research across subject areas and the distribution of researchers in government, university and industry sectors. Findings on the sources of research funding, however, were dissimilar and university research support appeared underreported in the bibliographic data. In general, data on publications (from bibliographic files or surveys) yielded lower estimates of industrial participation in research than data pertaining to projects.

Introduction

Survey research and bibliometric analysis are widely used methods for studying the structures and mechanisms of science. Surveys have been frequently used to collect data on the careers of scientists 1 and to assess the achievements of

participants in training programs. 2 In addition to studies of individuals, surveys have been used in examinations of organizations3, 4 and to explore other collective phenomena such as research relationships in newly emerging areas of science and

technology. 5 Bibliographic data have also been employed in studies of scientific careers 6-10 and to create measures of organizational success.11,12

This report compares bibliographic and survey data for three characteristics of research: subject matter, researchers' institutional sectors, and funding sources. Data used in this study come from an earlier investigation of collaborative research relationships 13 sponsored by the National Institute of Dental Research (NIDR). The original study examined the volume, organization, and outcomes of research on restorative dental materials in order to document the degree of industrial collaboration with government and academic researchers and to develop procedures that could be used to describe research activities in other areas of science and

Scientometrics 24 (1992) Elsevier, Amsterdam - Oxford - New York - Tokyo

Akad~miai Kiad6

H.H. GARRISON et al.: CHARACTERISTICS OF SCIENTIFIC RESEARCH

technology) 4 The field of dental materials was selected for this project because of its long history of commercial applications and its strong tradition of industrial research. Its small size, relative to other areas of biomedical research, also made dental materials an attractive field for exploratory study.

M e t h o d s

Survey data

The survey data come from a mail survey of U.S. dental materials researchers working in government, university, and not-for-profit settings. The survey frame was developed from the following sources:

- Members of two major dental research associations (the Academy of Dental Materials and the Dental Materials Group of the American Association for Dental Research).

- Recipients of dental materials research grants from NIDR (1978 through 1987). - Authors of dental materials research papers and posters presented at the 1988

meeting of the International Association for Dental Research. - Authors of dental materials articles appearing between 1983 and 1987 in five

major journals (Journal of Dental Research, Journal of the American Dental Association, Journal of Prosthetic Dentistry, Dental Materials, and Journal of Biomedical Materials Research.) 15

Researchers who resided outside of the U.S. were excluded from the survey frame along with researchers who were identified as employees of private industry. The latter group was dropped to avoid requesting information that might be considered proprietary or of commercial value. 16 After eliminating duplications from the various

source files, 965 U.S.-based dental materials researchers remained. The questionnaires were mailed in the spring of 1989. Three separate

questionnaire mailings and two reminder postcards yielded responses from 518 individuals, 59.3 percent of the 873 deliverable questionnaires. (In 92 cases, institutional affiliations were out of date and forwarding addresses could not be

obtained.) One hundred and thirty two of the returned questionnaires fell outside of the

scope of the study. Despite efforts to exclude them, the survey frame contained 53

360 Scientometrics 24 (1992)

H.H. G A R R I S O N et al.: CHARACTERISTICS OF SCIENTIFIC R E S E A R C H

persons who were employed by industry, resided outside the United States, or were not currently engaged in materials research. An additional 79 respondents were retired at the time of the survey and, therefore, also unable to provide the desired information on current research. The analyses presented below were based on the 386 in-scope responses. Comparison of findings from this survey to other data collected in the original study 13 indicated that the total volume of research activity was not grossly distorted by nouresponse. 17

The questionnaires developed for the dental materials researchers had two major sections. The first section, addressed to individuals who were principal investigators (PIs) of separately-budgeted research projects, 18 was designed to produce unduplicated data on research funding. The second section of the questionnaire collected information from all researchers regardless of PI status in order to examine the collaborative experiences of a broader research population.

Bibliographic data

A comprehensive literature search was conducted to identify restorative dental materials journal articles published from 1981 through 1985. A general search strategy was developed and executed using MEDLINE, an electronic index to serial publications developed and maintained by the National Library of Medicine. MEDLINE is the largest source of bibliographic data on dental research. There was, however, concern that some of the dental materials research which emphasized physical properties of materials over their dental application might be published in chemical and engineering journals rather than in the biomedical journals indexed by MEDLINE. Therefore, two additional data flies, CHEMICAL ABSTRACTS (the major index of publications in the field of chemistry) and METADEX (an index for metallurgy and materials science) were added to the bibliographic data collection process. Additional data sources were considered but rejected because of their overlap with these three files. 19

Bibliographic records from each source file were merged and sorted by author, year and title. After duplications were identified and deleted, the complete set of article titles was reviewed and classified into fourteen dental materials categories created for the purposes of this study. Articles outside of the scope of the study were dropped from the database at this point.

Scientometrics 24 (1992) 361


The information from the electronic searches was augmented with data from the title pages of the printed publications. Data on authors' institutional affiliations and sources of research support were collected, coded and added to the computerized records.

Results

Direct comparisons of the bibliographic and survey data were performed for three characteristics of dental materials research: area of research, researchers' institutional sectors, and sources of funding. The results of these comparisons are described in the following sections.

Areas of research

The distribution of categories of dental materials studied in research projects

reported by principal investigators in the survey was very similar to the distribution of materials addressed in the publications identified through the bibligraphic searches. (Principal investigators responding to the survey assigned their own projects into the same fourteen categories used to classify the data from the bibliographic fdes.)

In each data source, composite resins were the most frequently used category of materials, accounting for nearly 30 percent of the projects and published articles (see Table 1). The percentages for other categories were also very similar. Porcelains were the primary material in 5.7 percent of the projects and 4.5 percent of the articles. Impression materials were. featured in 4.7 percent of the projects and 5.4

percent of the journal publications. In only three cases (ionomers, casting alloys, and general properties research) did the distribution of materials studied in research projects differ by more than six percentage points from the distribution of materials

described in the publications. 2~ There is not a perfect correspondence between the distribution of projects and

publications. In some research areas, emphasis on product development or other factors may result in a lower rate of journal publication. Nevertheless, the close correspondence between the two distributions Suggests that either one may be used to describe the general topics under investigation in the field of dental materials. For

purposes of identifying areas of research emphasis, bibliographic data may be reliably

substituted for more expensive survey data.

362 Sciemometrics 24 (1992)


Table 1 Distibution of articles and projects by dental materials category

Dental materials Projects* Articles category (1988) (1981-1985)

1. Composite resins 63 626 (29.7%) (29.8%)

2. Ionomers 19 9.5 (9.0%) (0.5%)

3. Dental amalgam 17 281 (8.0%) (13.4%)

4. Cast alloys 10 282.5 (4.7%) (13.4%)

5. Porcelains 12 95.5 (5.7%) (4.5%)

6. Impression materials 10 114 (4.7%) (5.4%)

7. Investment/Die materials 6 53.5 (2.8%) (2.5%)

8. Denture materials 6 100 (2.8%) (4.8%)

9. Dental cements 12 191.5 (5.7%) (9.1%)

10. Waxes 1 6 (0.5%) (0.3%)

II. Maxillofacial prostheses 3 28 (1.4%) (1.3%)

12. Sintered alloys 6 96 (2.8%) (4.6%)

13. General properties 30 194.5 (14.2%) (9.2%)

14. Other 17 26 (8.0%) (1.2%)

Total 212 2104 (100%) (100%)

* From surveys of research performers.

Researchers' institutional sectors

Researchers' institutional affiliations were classified into three main groups (or institutional sectors): government, university, and industry.21 While similar data were derived from the bibliographic and survey methods, slightly different time periods were covered and industrial r~searchers were excluded from the survey. The impact of these differences was probably minimal. Analysis of the bibliographic data for the


H.H. G A R R I S O N et al.: CHARACTERISTICS OF SCIENTIFIC RESEARCH

1981-85 period covered by the literature search did not uncover major trends in authors' locations. 13 While the survey data were from a slightly later period, the

absence of temporal trends during the earlier period indicated that such trends were not a general characteristic of this aspect of dental materials research and unlikely to cause major shifts in the distribution of authors' locations. With regard to the exclusion of researchers based in industry, the number of papers with industrial authors in the bibliographic data file was small. (There were 46 articles with industrial authors only and 27 with industrial and other authors.) Their exclusion would not have had a major impact on the overall bibliographic findings.

It is highly unlikely that these two minor differences in coverage could have accounted for the substantial differences in the volume of data captured by the two data collection approaches (see Table 2). The survey respondents reported 450 journal dental materials journal articles in a single year (1988), twice the average for the five year period (1981-1985) covered in the bibliographic data. While some of this differential might have resulted from growth in the volume of publications, earlier analyses 13 found that rate of growth of dental materials journal articles was neither large nor linear between 1981 and 1985, with the average rate of growth for the period just over two percent per year. The excess volume of articles reported by the survey respondents cannot be attributed to growth alone and, in addition, contradicts the expectation that file sources would lead to greater (more complete) enumeration.

Several factors could explain the higher number of publications reported in the survey. Respondents may have been overly inclusive in reporting papers for the reference period. Perhaps~ to inaccurate recall of the actual publication dates,

earlier publications may have been included. Publications outside of the study's definition of dental materials would also have increased the total number of papers reported, as would those papers published in non-peer reviewed journals not indexed

by MEDLINE or the other bibliographic files. With the exception of the volume of papers reported, the distribution of authors'

locations were very similar for the two data sources. University-based authors contributed 82.6 percent of the papers in the bibliographic file and 91.1 percent of the papers in the survey file. (Adjusting for the exclusion of industry authors in the survey file by eliminating the 46 industry-based authors from the bibliographic file brought these percentages even closer. Dividing the 909 academic publications by the adjusted total, 1055, yielded 86.2 percent.) While papers by government authors were slightly more prominent in the bibliographic file, the percentage of collaborative

364 Scientomettics 24 (1992)


papers with authors from more than one institutional sector was nearly identical in each file.

Table 2 Researchers' institutional sectors from bibliographic and survey data

Publications* Publications** Projects (Principial Researchers' (Bibliographic (Researchers Investigators institution Files, 1981-1985) Survey, 1988) Survey, 1988)

Only government 77 (7.0%)

Only university 909 (82.6%)

Only industry 46 (4.2%)

Government and university 42 (3.8%)

Government and industry 3 (0.3%)

Industry and university 21 (1.9%)

Government-university-industry 3 (0.3%)

Total 1101 (100.0%)

14 4 (3.1%) (3.4%)

410 92 (91.1%) (77.3%) * * * * * *

25 2 (5.6%) (1.7%) 1 1

(0.2%) (0.8%) 0 18

(0.0%) (15.1%) 0 2

(0.0%) (1.7%) 450 119

(100.0%) (100.0%)

* U.S.-based authors only. ** Journal articles only. *** Null cell due to the exclusion of industry researchers from survey.

The distribution of researchers' institutional sectors differed slightly when separately-budgeted projects (rather than publications) became the unit of analysis. The largest differences involved collaboration with industry. For projects, the rate of university-industry collaboration was 15.1 percent, far higher than the rate for research articles computed from either bibliographic or survey data sources. Some of the difference may be a function of disincentives or alternatives to publication in research involving industry. Case studies of dental materials research projects 13 found that journal publication was frequently not an objective of industry sponsored research. Confidentiality may have been a major consideration in these cases.

Overall, the close correspondence between survey and bibliographic data was an important finding. For studies seeking to examine collaboration and the locus of research activity, bibliographic data may be useful. Their lower costs and elimination of response bias issues are important advantages. Their major drawback is the



underestimation of industry's role in research. If an investigator's primary objective is to focus on activity involving industry, however, the use of publication data (from any source) may lead be deficient.

Sources of funding

Information on sources of research support was also collected and classified into the same three institutional sectors used to describe the institutional affiliations of researchers. Comparison of data on research sponsorship from the bibliographic flies

and the survey, however, revealed substantial differences across the data sources (see Table 3).

Perhaps the most important finding was the difference in the rate of funding indexed by each data sources. Research support was acknowledged in 262 of the articles from the bibliographic file, just under one-fourth of the 1101 articles in the study's bibliographic database. Research support, however, was reported in 343 of the publications listed by the survey respondents (76.2 percent).

Acknowledgments in published papers may seriously underrepresent the extent of research support. Authors may not routinely report all sources of support in footnote acknowledgments in published papers, while more complete information may be provided in response to a direct inquiry. It was also possible that the typical survey respondent had greater access to research support than the typical author of a published paper. The selection of the survey population from the mainstream of the dental materials research community could have excluded the more peripheral participants in the research enterprise who might have been less likely to have funding for their research than the mainstream researchers. A response bias might have also contributed to some of the difference between the bibliographic and survey data. Funded researchers, and NIDR-funded researchers in particular, might have

been more likely to respond to a request for information sent out under NIDR

letterhead. The differences between the survey data and the journal acknowledgments were

particularly pronounced with respect to university support. The bibliographic data contained only 19 papers reporting university funds, while survey respondents reported 179 such publications. This difference, in both absolute and percentage terms, was substantial. University-based researchers may have assumed that they had implicitly/acknowledged university support by listing their institutional affiliation on the tit/k( page, and that only external funding needed to be explicitly noted. A



tendency on the part of researchers to understate the role of university contributions in their publication acknowledgments can have a substantial impact since the majority of the authors were university-based.

Table 3 Sponsors institutional sectors from bibliographic and survey data

Publications* Publications* Projects Sponsors' (Bibliographic (Researchers (Researchers Institution Files, 1981-19edi) Survey, 1988) Survey, 1988)

Only government 175 89 28 (66.8%) (25.9%) (23.5%)

Only university 19 179 31 (7.3%) (52.2%) (26.1%)

Only industry 42 26 34 (16.0%) (7.6%) (28.6%)

Goverment and university 15 37 4 (5.7%) (10.8%) (3.4%)

Government and industry 10 6 11 (3.8%) (1.7%) (9.2%)

Industry and university 1 5 10 (0.4%) 0_5%) (8.4%)

Government-university university 0 1 2 (0.0%) (0.3%) (0.8%)

Total 262* ** 343 + 119 (100.0%) (99.9%) (100.0%)

* U.S.-based authors only. ** Journal articles only. *** Excludes 839 publications with no reported research funding. + Excludes 107 publications with no reported research funding

In contrast to the rates for university research sponsorship, which differed in the bibliographic and the survey data, the distributions of support from nonuniversity sources (external support in most cases) were extremely similar. A perspective on extramural funding was created by subtracting the papers with "only university funding" from totals presented in Table 3. This procedure left 243 papers in the bibliographic file and 164 publications in the survey file. Government funding (in combination with support from other sources) accounted for 200 of the remaining papers in the bibliographic file (82.3 percent) and 133 of the papers in the survey f'de (81.1 percent). Industry funding (again including combinations with other sources) was reported for 53 of the papers in the bibliographic file (21.8 percent) and 38



papers in the survey file (23.2 percent). Because some papers had both government and industry funding, these percentages sum to more that 100 percent.

While reporting conventions may not be uniform for internal support, there appeared to be more consistency in reporting external funding. Acknowledgment is expected (and often required) on industry and government grants. Prestige may be another contributing factor. For authors submitting a paper to a selective, peer- reviewed journal, government support in general (and highly competitive NIH grant support in particular) may serve as a sign that their work has already been approved by a very selective peer review panel. Intramural research funding does not carry the same level of prestige.

For certain types of comparisons (patterns of external support, where relative distribution across institutional sector is more important than absolute level of funding) the survey and bibliographic data are substitutable. In general, however, the conventions governing the reporting of sponsorship do not appear to be sufficiently inclusive or uniform to make bibliographic data a complete source of general information on research funding.

Discussion and conclusions

Data from survey and bibliographic sources are quite comparable for several characteristics of the research, enterprise. The distribution of projects and publications across dental materials categories are remarkably similar and a close correspondence is also found between the distributions of researchers' locations

reported in survey and bibliographic data. Major differences are also found in the data derived from the two sources. For

data on research sponsorship, intramural academic support appears to be undercounted in the bibliographic data. The extent of industrial involvement is underestimated in analyses of journal publication, regardless of whether surveys or bibliographic files are used as a date source. If a research field is largely university-

based, then bibliographic data may be sufficient for many purposes. If an industry perspective is necessary, so are other data collection methods.

Both survey and bibliographic methods provide policy researchers with the opportunity to examine the structures and mechanisms of science. Surveys, of the type designed and executed for this study, can be used in investigations of other disciplines. Mailing lists from membership organizations provide a good starting point for constructing survey frames of researcher populations. Reliance on journal



authorship for a sampling frame, however, may underrepresent industry-based researchers. Publication policies of industrial employers and rates of industrial employment may effect the utility of bibliographic flies for the development of sampling frames in certain fields.

Surveys are more flexible and allow the investigator to focus on the organiTation

of the research enterprise itself. Their chief drawbacks are their cost in terms of both time and labor. Poor sampling frames and low response rates can also undermine the quality of survey data. Bibliographic data, when augmented with citation and funding

data, provide a very powerful tool for assessing outcomes of scientific research. Typically, however, bibliographic data sources are able to provide only limited information about the conditions under which the research was produced.

Contributions of individuals and organizations may be incompletely documented and range of data on research organization and support contained in the bibliographic source materials is limited.

An earlier version of this paper was presented at the annual meeting of the American Sociological Association, August 14, 1990 in Washington, DC. Data used in this study were collected for the National Institute of Dental Research, National Institutes of Health under the provisions of contract no. 263-87-C- 0269. Interpretations expressed in this report are those of the authors and do not represent those of the organizations that sponsored or performed the data collection.

References

1. W. ZuMET^,Extending the Educational Ladder: The Changing Quality and Value of Postdoctoral Study, Lexington, MA, D.C. Heath, 1985.

2. H.H. GARRISON, P.W. BROWN, The Career Achievements of NIH Postdoctoral Trainees and Fellows, Washington, DC, National Academy Press, 1986.

3. A.M. CAm~'ER, An Assessment of Quality in Graduate Education, Washington, DC, American Council on Education, 1966.

4. K.D. RoosE, C.J. ANDERSON, A Rating of Graduate Programs, Washington, DC, American Council on Education, 1970.

5. D. BLUMENTHAL, M. GLUCK, ]CS. Louis, D. WISE, Industrial support of university research in biotechnology, Science, 231(1986) 242-246.

6. B.F. RESgJN, Sex differences in status attainment in science: The case of the postdoctoral fellowship, American Sociological Review, 41(1976) 597-612.

7. B.F. REStaN, Scientific productivity and the reward structure in science, American Sociological Review, 42(1977) 491-504.

8. J.S. LoNG, P.D. ALUsos, IL McGtNNIS, Entry into the academic career, American Sociological Review, 44(1979) 816-830.

9. J.S. LoNG, R. MCGINNIS, Organizational context and scientific productivity, American Sociological Rev/ew, 46(1981) 422-442.



10. IL McGl~ l s , P.D. ALUSON, J.S. LONG, Postdoctoral training in bioscience: Allocation and outcomes, SocialForces, 60(1982) 701-722.

11. L.V. Jolq~, G. LINDZ~, P. C-.OC, GESHALL (Eds), An Assessment of Research-Doctorate Programs in the United States: Biological Sciences, Washington, DC., National Academy Press, 1982.

12. B.IL MAgTXra, J. IaVINE, CERN: Past performance and future prospects, Part I. CERN's Position in world high-energy physics, Research Policy, 13(1984) 183-210.

13. Applied Management Sciences, Assessment of NIH, Industry and Academic Relationships in Restorative Dental Materials Research. Final Report to the National Institute of Dental Research, National Institutes of Health as part of Contract No. 263-87-C-0269 (1989). Copies of this report may be requested from H. Garrison.

14. Case studies and patent analyses were also used in the original study. However, these methods covered different stages of the R&D process and were less directly comparable to the two methods discussed here.

15. A decision was made to exclude individuals with brief or minor participation in the field. These individuals would be difficult to locate and were expected to be less motivated to respond to a dental materials survey than active researchers in the field. Individuals who appeared only once as co- authors were dropped from the survey frame, while researchers who were sole author of an article or co-authors of more than one article were retained.

16. This definition of the study population did not prohibit the examination of collaboration involving industry researchers. It did require, however, that collaboration be studied from the perspective of the more completely enumerated government and academic researchers.

17. A separate survey of research sponsors achieved a response rate of 80 percent. The number of projects reported by sponsors and the combined budgets of these activities were very similar to the totals derived from the principal investigators portion of the research performers survey (see Applied Management Sciences 1989 for details).

18. These were defined as projects with external funding or internal funds earmarked for research under the direct control of the investigator.

19. Analyses showed that over 98 percent of the articles cited in dental materials articles published in the Journal of Dental Research over a four year period were indexed by these three data files (see Applied Management Sciences 1989: Appendix A ).

20. Deleting rows with expected frequencies less than five and combining the two main outlying categories (casting alloys and general properties research), the null hypothesis could not be rejected in the one-sample goodness of fit test. With eight degrees of freedom, the chi-square value of 8.60 has a probability value greater than 0.30.

21. Free standing research institutes and other not-for-profit foundations were classified with universities and other academic institutions.
