Measuring characteristics of scientific research: A comparison of bibliographic and survey data

  • Published on

  • View

  • Download

Embed Size (px)


<ul><li><p>Scientometrics, VoL 24. No. 2 (1992) 359-370 </p><p>MEASURING CHARACTERIST ICS OF SC IENT IF IC </p><p>RESEARCH: </p><p>A COMPARISON OF B IBL IOGRAPHIC AND SURVEY DATA </p><p>H.H. GARRISON,* S.S. HERMAN,* J.A. LIPTON** 9 </p><p>* Aspen Systems Corp., 962 Wayne Ave., Silver Spring~ MD 20910 (USA) ** National Institute of Dental Res., NIH, Bethesda, MD 20892 (USA) </p><p>(Received May 13, 1991) </p><p>Three characteristics of scientific research (subject matter, researchers' institutional sectors, and funding sources) were compared using bibliographic and survey data from a study of restorative dental materials research. Both types of data yielded similar findings on the distribution of research across subject areas and the distribution of researchers in government, university and industry sectors. Findings on the sources of research funding, however, were dissimilar and university research support appeared underreported in the bibliographic data. In general, data on publications (from bibliographic files or surveys) yielded lower estimates of industrial participation in research than data pertaining to projects. </p><p>Introduction </p><p>Survey research and bibliometric analysis are widely used methods for studying the structures and mechanisms of science. Surveys have been frequently used to collect data on the careers of scientists 1 and to assess the achievements of participants in training programs. 2 In addition to studies of individuals, surveys have been used in examinations of organizations3, 4 and to explore other collective phenomena such as research relationships in newly emerging areas of science and technology. 5 Bibliographic data have also been employed in studies of scientific careers 6-10 and to create measures of organizational success.11,12 </p><p>This report compares bibliographic and survey data for three characteristics of research: subject matter, researchers' institutional sectors, and funding sources. Data used in this study come from an earlier investigation of collaborative research relationships 13 sponsored by the National Institute of Dental Research (NIDR). The original study examined the volume, organization, and outcomes of research on restorative dental materials in order to document the degree of industrial collaboration with government and academic researchers and to develop procedures that could be used to describe research activities in other areas of science and </p><p>Scientometrics 24 (1992) Elsevier, Amsterdam - Oxford - New York - Tokyo </p><p>Akad~miai Kiad6 </p></li><li><p>H.H. GARRISON et al.: CHARACTERISTICS OF SCIENTIFIC RESEARCH </p><p>technology) 4 The field of dental materials was selected for this project because of its long history of commercial applications and its strong tradition of industrial research. Its small size, relative to other areas of biomedical research, also made dental materials an attractive field for exploratory study. </p><p>M e t h o d s </p><p>Survey data </p><p>The survey data come from a mail survey of U.S. dental materials researchers working in government, university, and not-for-profit settings. The survey frame was developed from the following sources: </p><p>- Members of two major dental research associations (the Academy of Dental Materials and the Dental Materials Group of the American Association for Dental Research). </p><p>- Recipients of dental materials research grants from NIDR (1978 through 1987). - Authors of dental materials research papers and posters presented at the 1988 </p><p>meeting of the International Association for Dental Research. - Authors of dental materials articles appearing between 1983 and 1987 in five </p><p>major journals (Journal of Dental Research, Journal of the American Dental Association, Journal of Prosthetic Dentistry, Dental Materials, and Journal of Biomedical Materials Research.) 15 </p><p>Researchers who resided outside of the U.S. were excluded from the survey frame along with researchers who were identified as employees of private industry. The latter group was dropped to avoid requesting information that might be considered proprietary or of commercial value. 16 After eliminating duplications from the various source files, 965 U.S.-based dental materials researchers remained. </p><p>The questionnaires were mailed in the spring of 1989. Three separate questionnaire mailings and two reminder postcards yielded responses from 518 individuals, 59.3 percent of the 873 deliverable questionnaires. (In 92 cases, institutional affiliations were out of date and forwarding addresses could not be obtained.) </p><p>One hundred and thirty two of the returned questionnaires fell outside of the scope of the study. Despite efforts to exclude them, the survey frame contained 53 </p><p>360 Scientometrics 24 (1992) </p></li><li><p>H.H. GARRISON et al.: CHARACTERISTICS OF SCIENTIFIC RESEARCH </p><p>persons who were employed by industry, resided outside the United States, or were not currently engaged in materials research. An additional 79 respondents were retired at the time of the survey and, therefore, also unable to provide the desired information on current research. The analyses presented below were based on the 386 in-scope responses. Comparison of findings from this survey to other data collected in the original study 13 indicated that the total volume of research activity was not grossly distorted by nouresponse. 17 </p><p>The questionnaires developed for the dental materials researchers had two major sections. The first section, addressed to individuals who were principal investigators (PIs) of separately-budgeted research projects, 18 was designed to produce unduplicated data on research funding. The second section of the questionnaire collected information from all researchers regardless of PI status in order to examine the collaborative experiences of a broader research population. </p><p>Bibliographic data </p><p>A comprehensive literature search was conducted to identify restorative dental materials journal articles published from 1981 through 1985. A general search strategy was developed and executed using MEDLINE, an electronic index to serial publications developed and maintained by the National Library of Medicine. MEDLINE is the largest source of bibliographic data on dental research. There was, however, concern that some of the dental materials research which emphasized physical properties of materials over their dental application might be published in chemical and engineering journals rather than in the biomedical journals indexed by MEDLINE. Therefore, two additional data flies, CHEMICAL ABSTRACTS (the major index of publications in the field of chemistry) and METADEX (an index for metallurgy and materials science) were added to the bibliographic data collection process. Additional data sources were considered but rejected because of their overlap with these three files. 19 </p><p>Bibliographic records from each source file were merged and sorted by author, year and title. After duplications were identified and deleted, the complete set of article titles was reviewed and classified into fourteen dental materials categories created for the purposes of this study. Articles outside of the scope of the study were dropped from the database at this point. </p><p>Scientometrics 24 (1992) 361 </p></li><li><p>H.H. GARRISON et al.: CHARACTERISTICS OF SCIENTIFIC RESEARCH </p><p>The information from the electronic searches was augmented with data from the title pages of the printed publications. Data on authors' institutional affiliations and sources of research support were collected, coded and added to the computerized records. </p><p>Results </p><p>Direct comparisons of the bibliographic and survey data were performed for three characteristics of dental materials research: area of research, researchers' institutional sectors, and sources of funding. The results of these comparisons are described in the following sections. </p><p>Areas of research </p><p>The distribution of categories of dental materials studied in research projects reported by principal investigators in the survey was very similar to the distribution of materials addressed in the publications identified through the bibligraphic searches. (Principal investigators responding to the survey assigned their own projects into the same fourteen categories used to classify the data from the bibliographic fdes.) </p><p>In each data source, composite resins were the most frequently used category of materials, accounting for nearly 30 percent of the projects and published articles (see Table 1). The percentages for other categories were also very similar. Porcelains were the primary material in 5.7 percent of the projects and 4.5 percent of the articles. Impression materials were. featured in 4.7 percent of the projects and 5.4 percent of the journal publications. In only three cases (ionomers, casting alloys, and general properties research) did the distribution of materials studied in research projects differ by more than six percentage points from the distribution of materials described in the publications. 2~ </p><p>There is not a perfect correspondence between the distribution of projects and publications. In some research areas, emphasis on product development or other factors may result in a lower rate of journal publication. Nevertheless, the close correspondence between the two distributions Suggests that either one may be used to describe the general topics under investigation in the field of dental materials. For purposes of identifying areas of research emphasis, bibliographic data may be reliably substituted for more expensive survey data. </p><p>362 Sciemometrics 24 (1992) </p></li><li><p>H.H. GARRISON et al.: CHARACTERISTICS OF SCIENTIFIC RESEARCH </p><p>Table 1 Distibution of articles and projects by dental materials category </p><p>Dental materials Projects* Articles category (1988) (1981-1985) </p><p>1. Composite resins 63 626 (29.7%) (29.8%) </p><p>2. Ionomers 19 9.5 (9.0%) (0.5%) </p><p>3. Dental amalgam 17 281 (8.0%) (13.4%) </p><p>4. Cast alloys 10 282.5 (4.7%) (13.4%) </p><p>5. Porcelains 12 95.5 (5.7%) (4.5%) </p><p>6. Impression materials 10 114 (4.7%) (5.4%) </p><p>7. Investment/Die materials 6 53.5 (2.8%) (2.5%) </p><p>8. Denture materials 6 100 (2.8%) (4.8%) </p><p>9. Dental cements 12 191.5 (5.7%) (9.1%) </p><p>10. Waxes 1 6 (0.5%) (0.3%) </p><p>II. Maxillofacial prostheses 3 28 (1.4%) (1.3%) </p><p>12. Sintered alloys 6 96 (2.8%) (4.6%) </p><p>13. General properties 30 194.5 (14.2%) (9.2%) </p><p>14. Other 17 26 (8.0%) (1.2%) </p><p>Total 212 2104 (100%) (100%) </p><p>* From surveys of research performers. </p><p>Researchers' institutional sectors </p><p>Researchers' institutional affiliations were classified into three main groups (or institutional sectors): government, university, and industry.21 While similar data were derived from the bibliographic and survey methods, slightly different time periods were covered and industrial r~searchers were excluded from the survey. The impact of these differences was probably minimal. Analysis of the bibliographic data for the </p><p>Scientometrics 24 (1992) 363 </p></li><li><p>H.H. GARRISON et al.: CHARACTERISTICS OF SCIENTIFIC RESEARCH </p><p>1981-85 period covered by the literature search did not uncover major trends in authors' locations. 13 While the survey data were from a slightly later period, the absence of temporal trends during the earlier period indicated that such trends were not a general characteristic of this aspect of dental materials research and unlikely to cause major shifts in the distribution of authors' locations. With regard to the exclusion of researchers based in industry, the number of papers with industrial authors in the bibliographic data file was small. (There were 46 articles with industrial authors only and 27 with industrial and other authors.) Their exclusion would not have had a major impact on the overall bibliographic findings. </p><p>It is highly unlikely that these two minor differences in coverage could have accounted for the substantial differences in the volume of data captured by the two data collection approaches (see Table 2). The survey respondents reported 450 journal dental materials journal articles in a single year (1988), twice the average for the five year period (1981-1985) covered in the bibliographic data. While some of this differential might have resulted from growth in the volume of publications, earlier analyses 13 found that rate of growth of dental materials journal articles was neither large nor linear between 1981 and 1985, with the average rate of growth for the period just over two percent per year. The excess volume of articles reported by the survey respondents cannot be attributed to growth alone and, in addition, contradicts the expectation that file sources would lead to greater (more complete) enumeration. </p><p>Several factors could explain the higher number of publications reported in the survey. Respondents may have been overly inclusive in reporting papers for the reference period. Perhaps~ to inaccurate recall of the actual publication dates, earlier publications may have been included. Publications outside of the study's definition of dental materials would also have increased the total number of papers reported, as would those papers published in non-peer reviewed journals not indexed by MEDLINE or the other bibliographic files. </p><p>With the exception of the volume of papers reported, the distribution of authors' locations were very similar for the two data sources. University-based authors contributed 82.6 percent of the papers in the bibliographic file and 91.1 percent of the papers in the survey file. (Adjusting for the exclusion of industry authors in the survey file by eliminating the 46 industry-based authors from the bibliographic file brought these percentages even closer. Dividing the 909 academic publications by the adjusted total, 1055, yielded 86.2 percent.) While papers by government authors were slightly more prominent in the bibliographic file, the percentage of collaborative </p><p>364 Scientomettics 24 (1992) </p></li><li><p>H.H. GARRISON et al.: CHARACTERISTICS OF SCIENTIFIC RESEARCH </p><p>papers with authors from more than one institutional sector was nearly identical in each file. </p><p>Table 2 Researchers' institutional sectors from bibliographic and survey data </p><p>Publications* Publications** Projects (Principial Researchers' (Bibliographic (Researchers Investigators institution Files, 1981-1985) Survey, 1988) Survey, 1988) </p><p>Only government 77 (7.0%) </p><p>Only university 909 (82.6%) </p><p>Only industry 46 (4.2%) </p><p>Government and university 42 (3.8%) </p><p>Government and industry 3 (0.3%) </p><p>Industry and university 21 (1.9%) </p><p>Government-university-industry 3 (0.3%) </p><p>Total 1101 (100.0%) </p><p>14 4 (3.1%) (3.4%) </p><p>410 92 (91.1%) (77.3%) *** *** </p><p>25 2 (5.6%) (1.7%) 1 1 </p><p>(0.2%) (0.8%) 0 18 </p><p>(0.0%) (15.1%) 0 2 </p><p>(0.0%) (1.7%) 450 119 </p><p>(100.0%) (100.0%) </p><p>* U.S.-based authors only. ** Journal articles only. *** Null cell due to the exclusion of industry researchers from survey. </p><p>The distribution of researchers' institutional sectors differed slightly when separately-budgeted projects (rather than publications) became the unit of analysis. The largest differences involved collaboration with industry. For projects, the rate of university-industry collaboration was 15....</p></li></ul>


View more >