Research synthesis methods and library and information

This item is not the definitive copy. The final published version will appear in the Journal of the Association for Information Science & Technology (JASIST). DOI: 10.1002/asi.23499

1

Research synthesis methods and library and information science: Shared problems, limited diffusion

Laura Sheble Center for Health Equity Research (CHER)

University of North Carolina Chapel Hill, NC 27599 [email protected]

ABSTRACT

Interests of researchers who engage with research synthesis methods (RSM) intersect with library and information science (LIS) research and practice. This intersection is described by a summary of conceptualizations of research synthesis in a diverse set of research fields and in the context of Swanson’s discussion of Undiscovered Public Knowledge. Through a selective literature review, research topics that intersect with LIS and RSM are outlined. Topics identified include open access, information retrieval, bias and research information ethics, referencing practices, citation patterns, and data science. Subsequently, bibliometrics and topic modeling are used to present a systematic overview of the visibility of RSM in library and information science. This analysis indicates that RSM became visible in LIS in the 1980s. Overall, LIS research has drawn substantially from general and internal medicine, the field’s own literature, and business; and is drawn on by health and medical sciences, computing, and business. Through this analytical overview, it is confirmed that research synthesis is more visible in health and medical literature in LIS; but suggested that, LIS, as a meta-science, has the potential to make substantive contributions to a broader variety of fields in the context of topics related to research synthesis methods.

Introduction Library and information science (LIS) is a “meta-discipline,” a discipline that interfaces with the

subject matter of other disciplines in ways that have value for society (Bates, 1999). This characteristic of LIS is especially important in the context of research synthesis methods (RSM), a family of methods that has altered practices in, and evaluation of research in medicine, public health, psychology, education, and other fields (e.g., Bastian, Glasziou, & Chalmers, 2010; Cooper & Hedges, 1994; Uthman, Okwundu, Wiysonge, Young, & Clarke, 2013). Document-based research synthesis methods, known as systematic review in the medical and health sciences, and meta-analysis in the case of statistical syntheses, integrate documented knowledge from past research studies to generate new knowledge or understandings. A hallmark of research synthesis methods is the emphasis on comprehensive identification and retrieval of primary research study documents; transparent disclosure of search protocols (Sander & Kitcher, 2006); and systematic methods of integration. To this end, research synthesis methodologists (e.g., Cooper, Hedges & Valentine, 2009; Petticrew & Roberts, 2006) highlight the skills and knowledge of information and library scientists: literature search and retrieval, and knowledge of information resources and scholarly communication systems are essential to the methods. When describing approaches to systematic reviews, Pawson (2006), for example, writes, “the idea is to unearth, dig out and dust down the relevant primary studies that should assist in answering the

2

review question. The skills and strategies required here belong to the domain of the information scientist” (p. 47).

Though the contributions of information and library scientists to the conduct of syntheses is recognized frequently outside the field, research synthesis has been neglected in LIS literatures (Hjørland 2001, 2002). Exceptions include LIS communities associated with health and medical information, and the evidence based librarianship (EBL) movement (see, e.g., Evidence Based Library and Information Practice). White, a contributor to a handbook for research synthesis (1994, 2009), which he describes as his “chief source of extradisciplinary citation” (2002, p. 323), observed that the apparent lack of attention to RSM may be due to the publication venues authors select. LIS authors may be more likely to publish RSM-related work in venues read by those who use RSM in other fields. Hjørland (2002) argues that regardless of whether RSM work by LIS researchers and practitioners is published outside of our disciplinary literature, RSM should be visible in LIS. This paper follows up on the discussion of White and Hjørland by outlining LIS research trajectories that converge with interests of research synthesists; examining the extent LIS researchers engage with RSM; and providing an overview of topics of LIS publications related to research synthesis.

Literature Review This literature review introduces research synthesis as a family of methods and describes a range

of approaches to synthetic research from a select set of diverse perspectives. Additionally, several LIS research trajectories that are coincident with issues central to research synthesis are outlined. Though many of the topics discussed may be extended to focus on data synthesis, this review primarily focuses on synthesis of research findings documented in study reports.

Background: Research Synthesis Methods Research synthesis methods, which include systematic or integrative review and meta-analysis,

have been promoted as an innovative approach to research that has the potential to mitigate weaknesses of traditional scientific review (e.g., Mulrow, 1994) and facilitate consensus formation and cumulation of scientific knowledge (e.g., Glass, 1976; Miller & Pollock, 1994; Rousseau, Manning, & Denyer, 2009). At least three themes recur in discussions of the emergence of systematic approaches to reviewing literature: (1) Pressures associated with increasing numbers of primary research publications (Chalmers, Hedges & Cooper, 2002; Glass, McGaw & Smith, 1981); (2) the roles of reputation and prestige (or “experience and expertise,” Huth, 2009) versus more egalitarian or “fair” evaluations of research findings; and (3) episodic and systemic failures to achieve or estimate consensus, including to communicate “the state of science” and inform policy and practice decisions (Chalmers, Hedges & Cooper, 2002; Glass, McGaw & Smith, 1981; Light & Pillemer, 1984; Light & Smith, 1971; Schulze, 2004).

Systematic review, and meta-analytic techniques in particular, have played a critical role in the transformation of research-related practices in medicine, public health, psychology, and other fields. Development and adoption of RSM has occurred in tandem with the development of complementary statistical techniques; literature search, retrieval, reporting, and appraisal methods; adaptation or interpretation of research synthesis in the context of divergent research traditions (Becker, 1996;


3

Chalmers, Hedges & Cooper, 2002; Noblit & Hare, 1988; Strike & Posner, 1983); and development of novel information resources (e.g., trial registries in medicine). Increasingly, researchers involved with synthesis are focusing on development of data repositories; and, in the practice professions extending synthesis approaches to incorporate viewpoints of a wider spectrum of stakeholders (Lomas, 2005). While quantitative meta-analyses and systematic literature reviews of quantitative research are the most prevalent forms, RSM have been translated to be compatible with other approaches to research, including interpretive methods (e.g., as “meta-ethnography” by Noblit & Hare). More recently, the relationship between RSM and data synthesis has been emphasized (Cooper & Patall, 2009; Sidlauskas, Ganapathy, Hazkani-Covo, Jenkins, Lapp, et al., 2009). Table 1 summarizes the steps in a research synthesis as presented by Cooper and Hedges (1994) and Noblit and Hare. Similarities between (I) Cooper and Hedges and (II) Noblit and Hare may be traced in part to the influence of Cooper and Hedges on Noblit and Hare (Thorne, Jensen, Kearney, Noblit, & Sandelowski, 2005), though conceptual similarities among approaches to RSM are noted in methodological writings of many researchers (e.g., Gough, 2004).

Table 1. The RSM process, as presented by Cooper and Hedges (1994) and Noblit and Hare (1988).

I. Cooper and Hedges (1994): “Research Synthesis”

II. Noblit and Hare (1988): “Meta-Ethnography”

Steps of the research process

1. Problem Formulation 2. Literature Search 3. Data Evaluation 4. Data Analysis 5. Interpretation of Results 6. Public Presentation.

1. Identification of a research interest; 2. Deciding what is relevant to the initial

interest; 3. Reading the studies; 4. Deciding how the studies are related; 5. Translating the studies into one another; 6. Synthesizing translations; 7. Expressing the synthesis.

Frameworks Describe Research Synthesis Approaches and Outcomes In this section, five conceptual frameworks that highlight facets of RSM from a diverse set of

viewpoints are briefly described. The first three are frameworks other researchers have developed to describe and define research synthesis in the context of their fields: evolutionary biology, education, and management and organization science (MOS). The fourth is extrapolated from Swanson’s discussion of “undiscovered public knowledge”; and the fifth discusses the relationship between RSM and research diffusion and integration as these concepts have been presented in the literature of library and information science and cognate fields.

Three conceptual frameworks for research synthesis Sidlauskas and colleagues (2009), writing about synthetic research in evolutionary science,

emphasize the nature of the scientific content that is synthesized. The authors identify the types of inputs synthesized as an important characteristic of research synthesis: A research synthesis study might integrate data, methods, concepts, or results across studies. Sidlauskas and colleagues note, “simple, but

4

still effective syntheses may include only similar elements within single disciplines, whereas more complex syntheses may incorporate heterogeneous elements or span multiple disciplines” (p. 2). To situate research synthesis in the context of evolutionary biology, the authors quote a passage by Simpson (1944, in Sidlauskas, et al.), a mid-twentieth century evolutionary biologist: “Synthesis has become both more necessary and more difficult as evolutionary studies have become more diffuse and more specialized. Knowing more and more about less and less may mean that the relationships are lost and the grand pattern and great processes of life are overlooked” (p. 871). Evolutionary biology is described as a field that is inherently interdisciplinary, drawing on fields such as paleontology, genetics, and systematics. Unlike fields such as ecology (Cadotte, Mehrkens, & Menge, 2012), evolutionary biology research spanned broad time scales and geographies prior to contemporary developments in research synthesis. These and other characteristics of evolutionary biology research inform the authors’ conceptualization of what it means to synthesize research.

Research synthesis may also be differentiated from other processes that involve bringing together reports of primary research studies through consideration of how, for what purposes, and to what effect primary research is brought together. Major and Savin-Baden (2010), writing from an interpretivist stance in the field of education, suggest that secondary uses of primary literature can be viewed along a continuum that ranges from more disjointed and less analytic uses of literature to integrated approaches that emphasize novel understandings. Major and Savin-Baden identify six forms of work along the continuum. The first two (“Information” and “Justification”) are those that collocate and describe primary research in narrative forms such as annotated bibliographies and narrative literature reviews. Here, narrative literature reviews are identified as work presented to position and justify a new research study. The third and fourth forms (“demonstration” and “account”) add the extraction and analysis of evidence. Demonstrations and accounts bring past research together in ways that may or may not be specified by the researcher; and include summaries of findings, including statistical meta-analytic summaries, related to a narrow category of research questions. In the fifth and sixth forms, knowledge derived from primary studies is integrated either to depict how the pieces fit into a whole (“research as part of a whole”), or to find novel interpretations based on prior evidence (“representation”). Like Sidlauskas and colleagues (2009), Major and Savin-Baden (2010) outline a framework that is commensurate with their own approach to research: One that not only legitimizes, but also exalts interpretive syntheses that integrate research to enhance current understandings at the intersection of practice and the academy.

Rousseau, Manning, and Denyer (2008) advocate for a realist approach to research synthesis (e.g., Pawson, 2006) for management and organization science (MOS) to “assemble the field’s full weight of scientific knowledge through syntheses” (p. 475). The authors initially distinguish between the traditional literature review and “systematic research synthesis” by characterizing the former as “often position papers, cherry-picking studies to advocate a point of view” (p. 3). The authors present a typology of syntheses distinguished by purpose: (1) aggregation; (2) integration; (3) interpretation; and (4) explanation. As presented by Rousseau and colleagues, synthesis by aggregation is comprised primarily of meta-analytic studies; integration of those that incorporate two or more methods of data collection (i.e., may include qualitative and quantitative evidence); interpretation those based on


5

relativist epistemologies; and explanation those that draw on a realist perspective and focus on causal mechanisms. The authors suggest that a realist perspective is best suited to the broad, heterogeneous spectrum of research pursued in MOS. Additionally, the authors emphasize use of research synthesis to inform practice, assess research, and make cumulations of knowledge explicit such that what is known and not known, to what extent, and in what circumstances is evident.

Research synthesis and “undiscovered public knowledge” Swanson’s (1986) discussion of “undiscovered public knowledge” provides an ideal framework

to situate concerns and opportunities related to RSM in the context of LIS. Conceptually, ideas encapsulated by Swanson reflect themes related to RSM and relevant to researchers from a broad range of epistemic traditions. At the core of Swanson’s work was the concept that the fragmentation of knowledge is a problem, and that new knowledge can be discovered within what is already public knowledge. While Swanson approached undiscovered public knowledge from what could broadly be described as a positivist or empiricist perspective, researchers with closer associations to other traditions have similarly noted concern about the isolation and fragmentation of research literature. For example, Glaser and Straus (1971), expressed concern that qualitative studies would become deserted “little islands” unlinked to other research, and encouraged researchers to focus on synthesizing findings across research studies, especially for the purpose of contributing to theory development.

If we take as our core concern that literature - or more precisely, findings, explanations, descriptions, and understandings – developed in the context of primary research studies and reported in documents, Swanson (1986) provides insight needed to describe types of questions that may be approached through synthesis of primary research studies. Swanson identified three cases in which retrieval and analysis of information can contribute to novel discoveries: (1) the hidden cumulative strength of individually weak tests; (2) discovery of a hidden refutation; and (3) discovery of a missing link in the logic of discovery. While Swanson emphasized the “hidden” nature of potential knowledge1, when considering research synthesis, it may be helpful to consider generically evidence from studies that has not yet been brought together, analyzed, and integrated. In this context, broad categories of questions that may be posed in a research synthesis study include the following:

1. Questions about the depth of knowledge along a dimension of investigation, such as the extent of a relationship between two variables;

2. Questions about the breadth of the applicability of knowledge across a dimension of investigation (for example, what are the characteristics of contexts in which a particular intervention is successful, and what factors might limit success);

3. Questions that examine conflicting or ambiguous evidence (or understandings) related to questions posed in primary research; and

4. Questions that integrate findings from previous research at novel levels of granularity.

Three other categories of questions, which can be extrapolated from Swanson’s discussion of undiscovered public knowledge and research synthesis literature include:

6

5. Questions that consider the broader context (e.g., temporal context) in which primary studies have been conducted but that is not captured in the primary research reports (Thorne et al., 2004; Twenge, 2011);

6. Questions designed to test theory or competing theories, judged based on primary research study findings2; and

7. Questions about how the processes and practices of research affect the characteristics of knowledge they are used to obtain (meta-science syntheses).3

Dimensions of synthesis: diffusion and integration A primary strength of research synthesis study designs is that researchers are afforded an

approach to integrate explicitly past research in response to specific research questions. As such, when evaluating research, it may be appropriate to assess the extent to which research synthesis studies integrate prior research not only in terms of the research synthesized, as is currently approached in meta-analytic studies through assessment of mediators and moderators (Shadish & Sweeney, 1991), cumulative meta-analysis (Lau, Schmid, & Chalmers, 1995), and meta-regression (Berkey, Hoaglin, Mosteller & Colditz, 1995; Thompson & Higgins, 2002), but also in terms of the dimensions of research contexts across which research is integrated. Since integration may be considered the inverse of diffusion, frameworks developed to investigate dimensions of diffusion, such as that discussed by Boschma (2005), may be appropriate for such tasks. The relationship between research synthesis studies and diffusion may be considered analogous to that of Rafols and Meyer (2010) and Klein (1996) as it relates to interdisciplinarity and integration; and in accord with that of Liu, Rafols and Rousseau (2012) on the relationship between diffusion and integration. An example of a challenge for future research would be to extend studies of relationships between diffusion and integration to examine whether and how knowledge of these relationships in concert with knowledge of the research knowledge base associated with a given research question might be mobilized to assess the potential of study proposals.

If one considers research synthesis methods along typologies such as that of Major and Savin-Baden (2010) and Rousseau, Manning, and Denyer (2008), it might be worth emphasizing that bringing together diffuse ideas could be considered a weak (but still potentially very valuable) form of research synthesis. Synthesis implies the creation of something that is more than the sum of its parts. Parker and Hackett (2012) describe this quality of synthesis using the metaphor of fusion: despite initial tensions that may impede combination, fusion emphasizes the potential for synergistic outcomes when disparate entities are integrated. The emphasis on novelty associated with research synthesis studies suggests a further challenge for future research: to clearly define and estimate novelty and perceived novelty in research contexts. A more fully developed understanding of novelty would be beneficial to the science of research assessment broadly as well as in the assessment of research synthesis studies.

Intersections of Research Synthesis and LIS As noted in the introduction, interests of researchers who use research synthesis methods

intersect with those of LIS researchers and practitioners. Use of research synthesis methods within LIS has been discussed primarily in the context of evidence-based librarianship (EBL) (e.g., Booth & Brice, 2004; Hjørland, 2011). Evidence-based librarianship is part of the broader evidence-based movement in


7

the practice professions (Trinder & Reynolds, 2000), popularized in medicine in the early 1990s (Evidence Based Medicine Working Group, 1992; Pope, 2003). Though RSMs were not recognized as a class of research methods in prior meta-studies of LIS research methods (e.g., Jarvelin & Vakkari, 1990; Hyder & Pymm, 2008), the methods are used to some extent (Grant & Booth, 2009).

Researchers and practitioners contribute to synthesis studies outside of LIS (Beverley, Booth, & Bath, 2003; Harris, 2005; Knight & Brice, 2006; McKibbon, 2006; Shell, Hofstetter, Carlock, & Amani, 2006; Swinkels, Briddon, & Hall, 2006). Additionally, researchers and practitioners are engaged in a broad range of activities that affect, and have the potential to inform, data collection for research synthesis studies. Information retrieval, interactive information retrieval, literature fragmentation, open access, data science, writing and publishing practices, and data repository development and curation influence data availability, accessibility, and management. Information and library scientists may also contribute to research syntheses through the development of data processing and management systems and techniques; and more generally, systems that support collaborative synthetic research projects (e.g., Young & Lutters, 2014).

The nature of research in LIS may pose considerable challenges to widespread adoption of research synthesis methods. It is likely that sub-fields variously embrace or are skeptical of studies that rely on analyses of primary research studies. In part, this may be due to the variety of meta-theoretical positions and approaches of LIS researchers (Bates, 2005; see also Hjørland, 2011). Uneven integration of concepts and methods in heterogeneous disciplines is not unique to LIS. Organization science, for example, has been described as differentially permeable to conceptual and methodological approaches (Bendersky & McGinn, 2010). The modest number of research studies in many areas of LIS, lack of commonly used measures, and inconsistent terminology (McGrath, 1996) suggests compatibility issues and other challenges (Ford, 2000). Bias against secondary research may also impede interest in RSM in LIS. Secondary research can be perceived as having low novelty or uniqueness value (Chalmers, Hedges, & Cooper, 2002). Mone and McKinley (1993), suggest that emphasis on uniqueness and novelty in organization studies is related to a low level of integrative research and a low level of value placed on review work. Though a similar study has not focused on LIS, it is possible that Mone and McKinley’s findings are applicable.

Lack of a comprehensive database (see, e.g., Meho & Spurgin, 2005) with publications indexed to facilitate identification of studies potentially relevant to a given synthesis is also likely to complicate efforts to synthesize past research in LIS. Distribution of LIS meta-science studies across extradisciplinary publication venues may add to this challenge. As noted by researchers in other fields (e.g., Toews, 2011), research synthesis studies become more laborious when literature is fragmented across databases and study attributes important to syntheses are not indexed. Finally, whether current methods to synthesize findings across LIS research are compatible with current and past research practices is an open question, though some approaches have been suggested (e.g., Urquhart, 2011; Ankem, 2005).

8

Open access and retrieval The open access and open source movements are known especially well in LIS as they relate to

open access to journal articles and computer source code, though more recently focus has shifted to also emphasize the importance of access to data (e.g., Piwowar, Vision, & Whitlock, 2011), and more comprehensive conceptualizations may be considered (e.g., Suber, 2012; Willinsky, 2006). It might be argued that transparency in documenting and reporting research and representation of research in document surrogates (Buckland, 1991) are open access issues. Open access, transparency, and representation by document surrogates all relate to the accessibility of intellectual content – though there are important differences in the mechanisms, actors, and organizations that determine levels of openness and accessibility of research content. Issues related to the accessibility of information about, and data from primary studies is central to RSM because these resources constitute the data sources for research synthesis studies: researchers can only perform their analyses if the data have been recorded, can be located, and are accessible. Examples of challenges related to data access for RSM studies include underreporting of primary study results and other elements,, linguistic parochialisms, limitations of document surrogates and database search and retrieval, inability of researchers to reach beyond social circles in order to find unpublished and otherwise unretrieved study documents (Ferguson and Brannick, 2012), and other issues. Many of these problems coincide with the enduring problems of literature fragmentation presented by Swanson (1986).

Research information ethics and bias Attention by research synthesists to the availability of data for synthetic studies has extended to

discussions and studies of systematic biases in research literature. Publication bias (the “file drawer problem,” Rosenthal, 1979) is generally defined as the lack of representativeness of published research compared to all research. Preferential evaluations of some research, most commonly that which is statistically significant (e.g., Fanelli, 2012, 2013; Pautasso, 2010; Schneider, 2013, Sterling, 1959), contribute to this problem. Though publication bias has received the most attention in fields such as psychology (e.g., Rothstein, Sutton, & Borenstein, 2005), other forms of bias are believed to impact publication, evaluation, integration, and synthesis of research (Chavalarias & Ioannidis, 2010). The extent to which biases are recognized and discussed likely varies by research field and values within fields. In addition to publication bias, citation bias, or preferential selection of references to support a perspective or claim (Ferguson & Brannick, 2012; Greenberg, 2009) may be important to library and information science research.

Referencing practices Changes in publication guidelines and standards made in part to accommodate the needs of

researchers who perform syntheses may indirectly affect citation patterns. For example, American Psychological Association (APA) style guidelines were modified to include more explicit instructions about what attributes of primary and meta-analytic studies should be reported, as outlined in the Journal Article Reporting Standards (JARS) and Meta-analysis Reporting Standards (MARS). In part, these standards were intended to facilitate use of reported results and data in research synthesis studies, and to specify how studies synthesized in secondary research should be referenced (e.g., APA Publications & Communications Board Working Group, 2008). According to APA, studies that synthesize up to fifty


9

primary studies should reference the primary studies in the regular reference list and differentiate references to synthesized studies using an asterisk (“*”). If more than fifty studies are synthesized, the studies should be listed in an appendix. Notable variation exists in how synthesized studies are indicated across the literature (see, e.g., Kueffer et al, 2011; Payne et al, 2012). These changing practices and guidelines influence citation metrics.

In some fields, beliefs and conventions related to hierarchies of evidence, which often categorize systematic reviews, meta-analyses, and experimental research (e.g., randomized control trials) as the “highest” levels of evidence based on internal validity, may influence authors’ selection of references. (Bhandari et al., 2004; Montori, Wilczynski, Morgan, Haynes, & the Hedges Team, 2003; Patsopoulos, Analatos, & Ioannidis, 2005); and criticism is levied at research that appears to selectively reference some studies in order to advance a research agenda, including to secure funding (Greenberg, 2009); and less politically, to instruct readers and authors how to evaluate published claims. For example, comparison of effect sizes in highly cited biomarker studies with those presented in meta-analyses of a large number of studies lead Ioannidis and Panagiotou (2011) to advise, “Readers should be cautious when authors cite single studies and not meta-analyses, and authors should be more careful in what they cite” (p. 2208).

Changes in citation patterns From another perspective, research synthesis is important to LIS because widespread use of

research synthesis may contribute to changes observed in patterns in the citation system (Persson, Glänzel, & Danell, 2004; Wallace, Larivière, & Gingras, 2009), including trends towards increasing levels of co-authorship (Morris & Goldstein, 2007; Persson, et al., 2004) and practices that result in the systematic exclusion of primary studies in reference lists of research syntheses (Payne, et al., 2012). In some fields, collaboration is considered beneficial to literature review and synthesis activities because (a) it enables triangulation between researcher evaluations of the relevance and quality of research, (b) the large scale of some research synthesis projects, and (c) needs for multiple types of expertise. This characteristic of research synthesis is very different from traditional literature review. Literature review is traditionally thought of as a genre dominated by expert researchers using the sole authorship model to share perspective as well as research-informed expertise (Peters & van Raan, 1994; c.f., Cooper, 1986) to “shape the literature of a field into a story in order to enlist the support of readers to continue that story” (Myers, 1991, p. 45). Whether and to what extent this is a valid characterization across research fields, however, is an empirical question.

Data access, secondary data analysis, and synthesis In addition to the intersection of research synthesis and library and information science, the

relationship between data science and RSM is important to LIS. Researchers emphasize similarities and differences between document- and data-based syntheses (e.g., Cooper & Patell, 2009; Riley, Lambert, & Abo-Zaid, 2010). Similarly, it may be helpful to emphasize similarities and differences between research synthesis and secondary data analysis. Strictly defined, secondary data analysis can be differentiated from research synthesis of data through comparison of the questions or hypotheses of the primary and secondary studies. If the directions of the questions or hypotheses are commensurate, a

10

secondary study should be considered a synthesis. This strict definition of research synthesis suggests secondary research may be considered along a continuum, from secondary data analysis to research synthesis.

Both synthetic research and secondary data analysis are dependent on data collected in primary studies, which may be reused in a synthesis based on aggregated group results reported in papers; or from accessible, and ideally, curated collections of data. Researchers may rely on hybrid approaches that involve synthesis of both data and reported results (Sidlauskas et al., 2009). Greater understanding of how and for what purpose data is analyzed can enable LIS researchers and practitioners to better facilitate resource interactions; help recognize overlapping and divergent interests of researchers; and promote a holistic perspective of information resource management.

Research synthesis methods and LIS: the current study A systematic bibliometric overview was conducted to assess the visibility of research synthesis

methods in LIS. The following questions guided this analysis:

RQ1: To what extent has engagement with research synthesis methods developed over time in LIS? RQ2: What fields are drawn on to produce publications related to research synthesis methods in LIS; and to what fields do LIS publications related to RSM contribute? RQ3: What topics are addressed in LIS publications related to research synthesis methods? It is evident that LIS researchers have engaged with research synthesis methods at some level,

but the extent of engagement is unclear. Therefore, before describing patterns in more detail, an overview of when and to what extent LIS researchers have engaged with the methods is developed, guided by the first question. This overview is contrasted with engagement across science broadly to contextualize the extent and timing of interest in research synthesis in LIS. Insight into which research fields draw on LIS research related to research synthesis methods, and paths of information flow to and from LIS literature is guided by the second research question. The final question guides an analysis of topics on which RSM related publications are published.

Data Collection and analysis Data was collected from the Thomson Reuters’ Web of Science Expanded Science Citation Index

(SCI) and Social Science Citation Index (SSCI) using the Web of Knowledge (version 5.5) interface using a combination of query and cited reference searches. The indexes were last updated 6 July 2012 at the time of the query term search; cited reference searches were performed on July 10, 16, 17, and August 1, 2012. A review of research synthesis methods literature and preliminary Web of Science searches were performed to identify the most relevant terms and phrases used to refer to research synthesis methods and important methods works. The terms and phrases were translated into the query phrases (Table 2) and used to search the Topic index, which is mapped to the Title, Abstract, Author Keywords, and “Keywords Plus™”4 fields. Query phrase searches were limited to items published between 1976 and 2011. 1976 was selected as a cutoff date based on Glass (1976), the publication in which Gene Glass coined the term “meta-analysis.” Though research synthesis methods predate Glass’s


11

paper, a new era of research synthesis can be traced to this period, and with it, use of a new set of linguistic phrases. Meta-analysis appears to be the first term used to uniquely specify research synthesis methods. Records from all searches were combined, de-duplicated, and limited to those associated with items published before 2011. Records collected across all fields and those limited to the “Information Science & Library Science” (ISLS) category were normalized based on the total number of publications published each year across all results and within the ISLS Web of Science Category (WC), respectively. Data processed with only the above procedure comprise the “unrefined” dataset. This unrefined data was used to provide an overview of the extent to which research synthesis has diffused within library and information science in comparison to diffusion of the methods across science broadly.

Table 2. Query phrases used to search the Topic Index. Query terms used to search the topic index of the Social Science and Science Citation Indexes (S/SCI) using the Web of Science 5.5 interface, on July 9-10, 2012. UNC licensed the Science Citation Index Expanded (SCI-Expanded) – 1955-present and the Social Sciences Citation Index (SSCI) – 1956-present at the time of data collection.

Query phrases "meta[-]stud[y | ies]” OR "meta[-]summar*” OR "meta[-]review*” OR "best-evidence synthes?s” OR "comparative effectiveness review*” OR "systematic review*” OR "systematic [theor* | research | multidisciplinary | method* | literature | evidence | international | critical | clinical | mixed method* | qualitative | narrative | quantitative] review*” OR "research synthes?s” OR "integrative review*” OR "integrative [research | mechanistic | literature] review*” OR Cochrane NEAR/1 review* OR "interpret[ta]tive synthes?s” OR "realist synthes?s” OR "meta[-]ethnograph*” OR "qualitative synthes?s” OR "qualitative evidence synthes?s” OR "meta[-]synthes?s” OR "meta[-]analy*” OR "meta[-]regress*”

Content Analysis: Identifying Publications Related to Research Synthesis Methods Titles, abstracts, and author keywords (if available) of retrieved publications from the ISLS

category were subject to a content analysis to determine whether a publication had a primary, secondary, or no apparent relationship to research synthesis methods. When an abstract was not available via the S/SCI record, the publication was retrieved from the University of North Carolina (UNC) libraries or through UNC inter-library lending (ILL) services. Abstracts available via the full text documents but not in the S/SCI records were added to the dataset. If an item did not have an abstract, an algorithmic summarization of the text was produced via the smmry.com web service and added in place of an abstract. One researcher coded all publication records, and a second coded a random subset, 10% of all

12

publications (Lombard, Snyder-Duch, & Campanella Bracken, 2002). Coders agreed 94.8% percent of the time. Table 3 describes the coding schema used to categorize relationships between publications and research synthesis.

Table 3. Coding categories used code relationships between retrieved publications and research synthesis methods.

Strength of relationship Definition Direct primary relation Publication has a strong primary relationship to research synthesis

or research synthesis methods. A publication has a direct primary relation to research synthesis methods if it reports a research synthesis study, develops methods, tools, or other resources specifically for use in subsequent research syntheses, introduces RSM to a community, studies or evaluates use of the method, discusses the method or a specific application of research synthesis methods, and so on.

Secondary / weak relation only Publication presents a weak, secondary, or tangential link to research synthesis: For example, secondary use of RSM products (e.g., for use in systems, or policy guideline development). Publications that focus on evidence based practice.

No direct relation There is no evidence that the publication directly relates to research synthesis based on the abstract, title, and author keywords.

Limitations to data collection include that only publications indexed in the Citation Indexes and identifiable with the search queries were included. While review of records ensures the probability that retrieved publications relate to research synthesis methods, the dataset likely somewhat understates the prevalence of publications related to research synthesis methods in library and information science. For example, relevant publications from indexed journals such as Small (1986) are excluded from results if they neither cite nor use one of the query phrases, as are relevant items published in books, journals, and other items not included in the indexes (e.g., Bates, 1992; White, 1992).

Bibliometric analysis Each LIS publication with a primary or secondary relationship to RSM in the data set was

analyzed to determine which fields were referenced, and the extent to which each of these fields was referenced. The number of references for each field was then summed across publications. Web of Science Categories (WC) were used as a proxy for fields. A python script was used to parse entries in the cited reference (‘CR’) field of WOS records and attribute entries to Web of Science Categories (‘WC’) based on journal titles. Regular expressions were used to match abbreviated journal title strings in the cited reference (CR) field. Journal title regular expressions were aggregated to match their associated WCs, thus enabling journal titles to be mapped to categories. Only titles indexed in the S/SCI were considered. Books, other resources, and journals not indexed in the S/SCI were ignored.


13

To identify whether and to what extent fields cite LIS publications, the Web of Science Citation Indexes were searched again on 23 October 2013, and a citation report was generated and analyzed within WOS to identify the number of citations by each WC. The relative contribution of LIS publications to research in other fields was estimated as the “field normalized influence”, which was calculated for each field (WC) by dividing the number of papers that cited one or more papers in the LIS set by the total number of publications in the field between 1985 and 2011. This is a very rough estimate of the influence of papers related to research synthesis in LIS in other fields. A better estimate could be derived if factors such as the number of papers referenced in the citing papers were taken into account.

A proportion of publications associated with a given WOS Category are cross-categorized in other categories. Cross-categorizations of the 293 LIS publications were analyzed to specify and map this data set. Category-based mapping was selected for consistency with citing and cited mapping; and to characterize the diversity of publications that comprise the core LIS set. Additionally, cross-categorizations observed in the core data set were compared to the proportional distribution of cross-categorizations of LIS research between 1985 and 2011. The cross-categorization distributions were then used to calculate the levels and categories of cross-categorization that would be expected for any given set of LIS publications given the publication year distribution observed in the core data set. Comparison of observed and expected cross-categorization distributions describes whether and in what ways publications in the core LIS data set differ from research in LIS overall during the study period, and provide detail useful to interpretation of the corresponding bibliometric maps.

Bibliometric Mapping The cited, citing, and core LIS datasets were mapped onto a network projection that represents

citing patterns across science broadly based on publications included in the 2010 Science and Social Science Citation Indexes, and aggregated by Web of Science category following Rafols, Porter and Leydesdorff (2010) using data described by Leydesdorff, Carley, and Rafols (2013). As indicated in the protocol outline by Rafols and colleagues, the relationships between science fields, (i.e., Web of Science Categories), are defined by cosine-normalized citing patterns and visualized in Pajek (Batagelj & Mrvar, 1998) using the Kamada and Kawai (1989) algorithm to project a base network of edges. The frequency with which publications associated with each category was observed in a target dataset (here, the citing, cited, and core LIS sets) were then overlaid on the edge network, represented as proportionally sized nodes. Colors are assigned to nodes based on 19 macro-disciplinary category groups based on factor analysis of the base map (Leydesdorff et al., 2013).

Limitations of this technique specific to this study include the small size of the LIS dataset, the time span across datasets, and the presence of the “Audiology Speech Language Pathology” category, which was not accommodated by the base map. Despite these limitations, the maps provide a visual overview that summarizes a substantial amount of data that would be difficult to adequately describe in text, and which enable viewers to compare the knowledge base that informs a body of work; and subsequent publications that draw on this body of work. Though the practice of associating journals with multiple Web of Science categories unevenly inflates the apparent number of publications overall, and the categorizations used are, in some cases, controversial, this projection provides a visual overview of

14

the approximate extent and diversity of science fields associated with each data set (Leydesdorff & Rafols, 2009). Additionally, through presentation of the full, overlaid map of science rather than a subsection of it, the absence of fields becomes visible.

Topic Modeling A topic model was developed to summarize the content of LIS publications related to research

synthesis methods using the variational Bayes CVB0 implementation (Asuncion, Welling, Smyth, & The, 2009) of the Latent Dirichlet Allocation (LDA) algorithm (Blei, Ng, & Jordan, 2003), through the Stanford Topic Modeling Toolbox (Ramage & Rosen, 2011). As applied here, LDA was used to generate a probabilistic topic model based on word co-occurrences in titles, author keywords (where available), and abstracts using a ‘bag of words’ approach. Where present, labels were removed from structured abstracts. Words were stemmed with the Porter stemmer, and common English stop words and the ten most frequently occurring words were not considered during the modeling process. The number of topics to include in the model was informed by perplexity scores for proposed numbers of topics ranging from 5 to 30, such that local perplexity minima were preferred. Perplexity scores were derived from modeling a given number of topics using half the terms in a document, randomly selected, over 500 iterations, and calculating the subsequent probability that a given document would be assigned a given mixture of topics based on the remaining terms associated with an item.

Topics were named through review of the twenty most common words and their probabilistic frequency of occurrence per topic, and review of documents most closely associated with the given topic. Overall, the approach to topic modeling taken may be described as a quantitatively guided qualitative summary of author-contributed publication meta-data. The document-topic matrix, which indicates the topic mixture distributions across documents, was used to calculate the topic cosine similarity matrix. This matrix was transformed into UCINET format and imported into Gephi (Bastian, Heymann, & Jacomy, 2009) for visualization as a network graph. Within Gephi, the Louvain algorithm (Blondel, Guillaume, Lambiotte, & Lefebvre, 2008) was used to partition topics, nodes were sized based on the proportion of documents associated with each topic, and an edge threshold of 0.1 was applied.

Results and Interpretation Across all Web of Science Categories in the SCI/SSCI, the combined queries retrieved 123,881

unique publication records. The majority (116,613) was retrieved with keyword searches. Cited reference searches yielded 21,757 records, 7,268 of which were uniquely retrieved with this strategy. Across the SCI/SSCI, the number of records retrieved with the RSM search query grew from just a handful of items in the 1970s (less than 10 publications per year until 1979) to 19,994 records in 2011. In LIS, the first publication appeared in 1985; and the number of items published per year rose to 55 in 2011. Figure 1 indicates items retrieved with the query as a percent of all publications by year across SCI/SSCI and limited to LIS. Trends in the proportion of RSM-related publications across science and in LIS were in the same direction, though the magnitude is somewhat less in LIS. The majority of LIS publications related to RSM are located in a small set of journals. The distribution of publications across journal titles approximates the 80/20 rule: About 80 percent of all RSM publications were found in 20% of LIS titles (Figure 2). Journal titles with the greatest number of publications are listed in Table 4. In


15

addition, publications related to research synthesis methods were found in 39 other titles included in the ISLS category. In 2011, the Journal Citation Report (JCR) listed 84 titles in the ISLS category, though a greater number of titles are included in the ISLS category in the Citation Indexes, perhaps because journal titles included in the Citation Indexes has changed over time.

0

3

6

9

12

1980 1990 2000 2010Publication year

RS p

er 1

000

publ

icat

ions

Field

All Fields

ISLS

Figure 1. Overview of RSM publication occurrences. RSM publications per 1000 publications across all fields and within LIS (‘ISLS’).

20

40

60

80

100

0 20 40 60 80 100Titles (cumulative %)

RSM

pub

licat

ions

(cum

ulat

ive

%)

Set

Refined

Unrefined

Figure 2. Concentration of LIS publications related to RSM, by journal title. Approximately 80 percent of RSM publications appear in 20 percent of LIS titles (‘Unrefined” set = publications retrieved with the initial search query; ‘Refined’ set = publications judged to have a primary or secondary relationship to research synthesis).

16

Table 4. Journal titles with the greatest number of publications included in the LIS dataset. Journal

Number of publications

Journal of the American Medical Informatics Association 60 Bulletin/Journal of the Medical Library Association 45 Health Information & Libraries Journal 35 Journal of Health Communications 15 Journal of the American Society for Information Science (& Technology) 12 Information Management 11 Scientometrics 11 MIS Quarterly 10 Library and Information Science Research 7 Journal of Documentation 6 Library Trends 6

The flow of information via LIS publications related to research synthesis methods is visible through examination of fields referenced by LIS publications versus those that cite LIS publications. The 293 LIS papers judged to have a primary or secondary relationship with research synthesis methods collectively reference journal publications that were re-associated with 5175 counts across S/SCI categories. Proportionally, the largest reference fields align with the S/SCI categories General & Internal Medicine (26.26%), followed by Information Science & Library Science (14.42%), Management (5.84%), Public, Environmental & Occupational Health (3.57%), and Operations Research, Management Science (3.21%). Table 5 identifies the ten fields LIS research synthesis methods publications referenced most frequently. The distribution of references across science is presented in Figure 3.

Table 5. Ten fields (WCs) most referenced by LIS publications related to research synthesis. ‘Reference count’ is the number of references to publications associated with the given field, based on WOS WCs. ‘Proportion of references’ is based on the total count of category-reference allocations.

Cited field (WC)

Reference count

Proportion of references

General & Internal Medicine 1359 26.3 Information Science & Library Science 746 14.4 Management 302 5.8 Public, Environ., & Occupational Health 185 3.6 Operations Research, Management Science 166 3.2 Health Care Sciences & Services 132 2.6 Mathematics 109 2.1 Nursing 101 2.0 Education & Educational Research 86 1.7 Medical Informatics 79 1.5


17

Clin Med

Psych Sci

Clin Psych

Math Methods

Social Studies

Econ Poli Geog

Business, Mgt

Infect Disease

Chemistry

PhysicsMaterials Sci

Mech Eng

Envir Sci Tech

Biomed Sci

Agri SciEcol Sci

Geosciences

Hlth Soc Issues

Comp Sci

Clin Med

Psych Sci

Clin Psych

Math Methods

Social Studies

Econ Poli Geog

Business, Mgt

Comp Sci

Infect Disease

Chemistry


Mech Eng

Envir Sci Tech

Biomed Sci

Agri SciEcol Sci

Geosciences

Hlth Soc Issues

Clin Med

Psych Sci

Clin Psych

Math Methods

Social Studies

Econ Poli Geog

Business, Mgt

Infect Disease

Chemistry


Mech Eng

Envir Sci Tech

Biomed Sci

Agri SciEcol Sci

Geosciences

Hlth Soc Issues

Comp Sci

Figure 3a. Citing publicationsthat draw on LIS publications are morediverse than the knowledge base, or those referenced by LIS publications.

Figure 3b. LIS publicationsassociated with research synthesismethods are often cross-categorized inoverlapping fields.

Figure 3c. Referenced publications:The knowledge base of LIS publicationsreference is dominated by ‘Medicine, General & Internal’, followed by LIS (’Information Science, Library Science’).

LIS (ISLS)

LIS (ISLS)

LIS (ISLS)

Figure 3. Relative location of knowledge base and use contexts of LIS research synthesis publications.

18

As of 23 October 2013, the LIS RSM publication set was cited 8927 times by 7617 publications. Table 6 lists citing fields that account for more than 2% of the 7617 citing publications. Not including the ISLS category, the five categories that include publications that cite the LIS RSM publications most (Computer Science, Information Systems; Management; Medical Informatics; Health Care Sciences & Services; and Computer Science, Interdisciplinary Applications) include journals that are cross-categorized with LIS to some extent. Following cross-categorized fields, a broad range of citing fields was observed, including fields from clinical medicine, health sciences, business, computer sciences, and social sciences.

Table 6. Fields that most frequently cited LIS publications related to research synthesis. Field-normalized influence divides the number of publications in a given category in the publication set by the total number of publications in the category, 1985-2011; and is provided to suggest the relative frequency with which the LIS RSM set is referenced given the size of the field/category.

7617 citing publications Field (WC) Records Field influence Comp Science, Information Systems 1897 0.41 Information Science & Library Science 1881 0.69 Management 1006 0.42 Medical Informatics 975 1.85 Health Care Sciences & Services 911 0.62 Comp Science, Interdisciplinary Applications 810 0.22 General & Internal Medicine 577 0.07 Business 499 0.25 Public, Environ., & Occupational Health 444 0.09 Computer Science, Theory & Methods 325 0.06 Operations Research, Management Science 272 0.14 Comp Science, Artificial Intelligence 262 0.05 Health Policy & Services 262 0.24 Education & Educational Research 255 0.10 Communication 229 0.32 Nursing 216 0.20 Computer Science, Software Engineering 191 0.06 Electrical & Electronic Engineering 189 0.01 Psychology Multidisciplinary 176 0.08 Industrial Engineering 173 0.11 Interdisciplinary Social Sciences 141 0.11 Pharmacology & Pharmacy 125 0.01


19

Cross-categorizations within the LIS set were examined to better understand the categories that occur more and less than we would expect; and to better understand the flow of information to and from LIS. As a group, the LIS publications related to research synthesis differed from that which we would expect of any given set of LIS publications with the same distribution of publication years. The LIS set was about four times more likely to be cross-categorized as Computer Science, Information Systems; seven times more likely to be cross-categorized as Computer Science, Interdisciplinary Applications; and twelve times more likely to be cross-categorized as Medical Informatics (Table 7). Over one fifth of the publications (60 items) in the set were published in the Journal of the American Medical Informatics Association (JAMIA), which contributes to these differences. A substantial number of items were published in the Bulletin/Journal of the Medical Library Association and Health Information and Libraries Journal (45 and 35 publications, respectively). These journals are classified only in the LIS (ISLS) category.

Table 7. Expected and observed cross-categorizations in the core LIS set, ordered by number of publications observed

Number of Publications Field (WC) Expected Observed Computer Science, Information Systems 41.20 123 Computer Science, Interdisciplinary Applications 10.79 73 Medical Informatics 4.80 60 Management 7.60 30 Communications 5.40 15 Multidisciplinary Science 14.45 5 Health Care Sciences & Services 0.28 4 Interdisciplinary Social Sciences 2.28 4 Multidisciplinary Humanities 0.90 1 Social Issues 0.10 1 Education 0.92 0 Geography 2.15 0 Physical Geography 1.98 0 Law 2.00 0 Telecommunications 1.98 0

Differences between fields referenced and those that cite LIS publications indicates that overall, there is a net flow from fields such as General & Internal Medicine and Mathematics to those associated with LIS (via cross-categorization), and cognate fields in business and Computer Science. Health science fields such as Health Care Sciences & Services; Public, Environmental & Occupational Health; Health Policy & Services; and Nursing draw on LIS publications related to research synthesis methods more than they inform this work (Table 8). LIS-associated work published in JAMIA, based on the field-normalized influence score, represents an important contribution to the broader Medical Informatics field.

20

Table 8. Fields with the greatest difference between cited and citing publications.

Field (WC)

Cited

Citing

Field-normalized influence

Diff: citing-cited

General & Internal Medicine 1359 577 0.065 -782 Mathematics 109 0 0.000 -109 Food Science & Technology 67 7 0.002 -60 Psychology, Social 71 39 0.053 -32 Veterinary Sciences 51 10 0.003 -41 Computer Science, Hardware & Architecture 69 60 0.023 -9 Interdisciplinary Social Sciences 16 141 0.110 125 Electrical & Electronic Engineering 5 189 0.010 184 Nursing 101 216 0.201 115 Computer Science, Software Engineering 2 191 0.057 189 Communication 61 229 0.315 168 Computer Science, Artificial Intelligence 17 262 0.046 245 Business 63 499 0.245 436 Computer Science, Theory & Methods 9 325 0.062 316 Health Policy & Services 54 262 0.241 208 Public, Environ. & Occupational Health 185 444 0.090 259 Management 302 1006 0.422 704 Comp Science, Interdisciplinary Applications 7 810 0.221 803 Health Care Sciences & Services 132 911 0.616 779 Information Science & Library Science 746 1881 0.694 1135 Medical Informatics 79 975 1.851 896 Computer Science, Information Systems 8 1897 0.410 1889

Overview of Topics and Topic Communities Library and information science work related to research synthesis methods was aligned with

three topical communities across sixteen topics associated with methods, resources, services, and subjects of research synthesis (Figure 4). The most prevalent terms in each topic are identified in the Appendix. The first community consists of the topics “Data Sources”, “Search Filters”, and “Effects”. Search filters are used in the health and medical sciences to identify studies in databases such as PubMed for inclusion in research syntheses. A number of authors focused on search filter development and evaluation. Authors wrote about search more generally with respect to which databases were searched for specific studies, and how; and to review, evaluate and discuss study selection and reporting in research synthesis studies. Content related to data sources focused primarily on literature databases, but also data from primary studies, such as that which might be included in data syntheses (e.g., individual participant data meta-analyses). The strongest link between this and other communities was observed between Data Sources and Search Strategies. An example of a study that is a mixture of these


21

topics is Yoshii, Plaut, McGraw, Anderson, and Wellik (2009), an evaluation study of search protocol reporting practices observed in systematic reviews in the Cochrane Library.

Health

Decisions

Methods

Meta-study

Models & Theories

Data sources

Effects

Systems & Tech

Text analysis

Search strategies

EBP

Patient care

Search filters

HII

Services & Libraries

Tech acceptance

Figure 4. Topic similarity network. Node size represents the relative proportion of documents per topic; edges the similarity between topics based on collocation in documents, normalized with Salton’s cosine. Colors indicate three communities detected with the Louvain algorithm.

The second community is the largest both in terms of the number of topics and document proportions. Topics include “Models & Theories”, “Methods”, “Systems & Technology”, “Meta-Study”, “EBP”, “Text analysis”, “Search Strategies”, “Services & Libraries”, and human-information interactions (“HII”). The text analysis topic includes papers that report machine learning and text analysis techniques developed to reduce the workload associated with research synthesis methods. A goal of machine learning techniques in this context is to reduce the number of potentially relevant studies for synthesis that need to be reviewed manually by researchers. In the health and medical sciences, studies identified as “potentially relevant” to a synthesis may number in the thousands or even tens of thousands though the final dataset will likely include only a small fraction of these studies.

Authors discussed evidence-based practice in the context of libraries and management information science. Papers about methods associated with EBP (e.g., experimental designs and research

22

syntheses) in health and medical sciences and the implications these would have for health and medical librarians were prevalent. For example, in an early paper, Schell and Rathe (1992) introduced readers of the Bulletin of the Medical Library Association to meta-analysis and predicted that medical librarians would play a significant role in the future application of this “tool for medical and scientific discoveries” (p. 219).

The “Methods” topic included texts that introduced research synthesis methods, outlined use or potential implementations of the methods, and raised issues associated with their use in specific circumstances. The “Meta-studies” topic was prevalent in research syntheses within informetrics and studies of MIS research published in the European Journal of Information Systems (EJIS), which were described with the keyword “meta-analysis”. The HII topic was the most difficult to label. This topic includes terms and large proportions of papers related to information behaviors and public availability of knowledge. The “Models and Theories” topic included models of information behavior and the Technology Acceptance Model (TAM). The “Systems and Technologies” topic was related to systems and technologies for and that potentially affect research synthesis methods; and systems and technologies that research synthesis methods were used to study.

The third community consists of the topics “Health”, “Decisions”, “Patient Care”, and “Technology acceptance”. The health topic was originally labeled “Health (campaigns, communities, behavior)”, but later shortened for graphic presentation. As the longer label suggests, this topic is closely associated with studies of campaigns intended to modify health behaviors. The “decisions” topic relates to health behaviors through discussions of patient health decisions. More broadly, the decision topic also includes decisions of health care providers related to patients and information resources; and decision-support systems (DSS). The “Technology acceptance” topic was primarily associated with MIS publications. Technology Acceptance Model (TAM) research has been the basis of an important research trajectory in MIS following Davis (1985). Authors have synthesized findings from subsets of TAM studies in several research synthesis publications. Nine of the ten articles most closely associated with the “Patient Care” topic [topic proportion ranges: 64.0% - 93.9%] were published in JAMIA. These publications reported systematic reviews related to preventative health care provision, treatment effectiveness and cost, systems design to improve health care through error reduction, and related topics.

Associations Between Topics and Subfields A number of topics that emerged during topic modeling appeared to be strongly associated with

research specific to Management Information Systems (MIS) or health and medical information/libraries. In order to view associations between more granular research areas and topics, past research (Baccini & Barabesi, 2011; Cronin & Meho, 2008; Sugimoto, Pratt, & Hauser, 2008; Larsen, Monarchi, Hovorka, & Bailey, 2008; Rainer & Miller, 2005) was reviewed to categorize journals as either: MIS or LIS. Because of the high prevalence of publications with a health or medical focus, journals associated with these topics were categorized separately for LIS (“LIS-Med”). The Information Society was categorized as LIS; and the Journal of Information Technology was categorized as MIS, which is consistent with some but not all past studies. Two journals were categorized as “other” (the Scientist, and Social Science Computer Review). Sub-field categorizations led to 155 publications


23

identified as LIS-Med, published across four journals; 89 LIS publications across 31 journals; 42 MIS publications across 13 journals; and 7 “Other” publications across two journals. The document-topic matrix heatmap, which indicates the topic mixtures within documents, was annotated with a color side bar to indicate the subfield assigned to each publication (Figure 5).

0 0.2 0.4 0.6 0.8

Topic Proportion

LISLIS-MedMISLIS-Other

Sub-field

TopicsTe

ch a

ccep

tanc

ePa

tient

care

Sear

ch fil

ters

Healt

h

Decis

ions

Met

hods

Met

a-stu

dy

Mod

els &

The

ories

Data

sour

ces

Effe

cts

Syste

ms &

Tech

Text

analy

sis

Sear

ch st

rate

gies

EBP

HII

Serv

ices &

Libr

aries

Figure 5. Topic distributions across documents. The color side bar indicates the subfield of each publication: Green = LIS-Med (N=155); Red = MIS (N=42); Blue = LIS (N=89); Purple = Other (N=7).

24

The main color matrix is shaded to indicate the proportion of each topic, light yellow (0) to dark red (1). White space was inserted between rows to identify document row clusters and clarify relationships to subfield annotations.

Work related to health and medical information and libraries (LIS-MED) was most prevalent of all subfields. The topics “Patient Care”, “Search Filters”, “Health”, and “Effects” were almost exclusively associated with publications in the four journals categorized as LIS-Med. To a lesser extent, the same was true for the topics “Data Sources” and “Text Analysis”. One topic that was nearly absent in LIS-Med publications, “Technology acceptance”, primarily consisted of MIS journal publications. The remaining topics have some representation in at least two subfields. Across all topics, the “Theories & Models”, “Methods”, “Systems & Technology”, and “Meta-Study” topics were distributed most evenly across subfields. The evidence-based practice (EBP) topic was also associated with all three subfields.

Conclusion and Future Directions Widespread adoption of systematic approaches to research synthesis has had a profound impact

on how researchers interact with prior research in the medical and health sciences, psychology, education, and at least to an extent in other fields such as ecology, and social work. Though researchers have written about research synthesis extensively in areas of LIS related to health and medical sciences, researchers in LIS more broadly have not. In part, this may be due to a perceived lack of compatibility or value of the methods for LIS research. Whether this is the case and why may be an important consideration for the field (Ford, 2000), and raises questions about how the field functions as a meta-discipline.

Use of research synthesis methods is an important development for library and information science researchers and practitioners because it has affected how researchers in other fields interact with literature, data, and information infrastructures. Some changes associated with use of the methods are visibly manifested in research reporting guidelines and the development of resources, organizations, and tools to support synthetic research. More subtle changes relate to what is included in reference lists, and how research is evaluated and used to support subsequent research. Whether and to what extent use of the methods has contributed to authorship trends and citation metrics is unclear, though these topics have implications for research evaluation, indexing practices, and so on. Other topics of consequence to research synthesis and LIS research and practice broadly include access to research findings in literature and data; and information behaviors that result in perceptions of bias in the literature, including publication bias.

The subset of LIS associated with health and medical information and libraries has made substantial contributions to the practice of research synthesis, as is suggested by the analysis of information flows from LIS via citations in this study, and descriptions of the centrality of library and information scientists to performing research synthesis in methodographies. Health and medical informationists’ research contributions are especially prevalent in the areas related to search strategies, information resources, and search filters. To an extent, use of text analysis and machine learning


25

techniques to support syntheses have been investigated. Additionally, researchers have investigated and described how LIS researchers and practitioners contribute to specific research synthesis projects in the health and medical sciences. Research synthesis is less visible in other areas of LIS. Though topics such as information access, research evaluation, documentary forms associated with knowledge production, and information behaviors and interactions are central to the field, these topics have not been sufficiently problematized in the context of RSM given the prevalence of the methods. If LIS is a meta-discipline that endeavors to interface and support research and information interactions by researchers in other fields, arguably, LIS should engage with use of research synthesis methods to a greater extent; and with respect to a more diverse set of fields.

Footnotes

1. Potential knowledge may be “hidden” because what is known may not (yet) be integrated, studies are unknown, and because of the vast amount of literature. Literature can be considered fragmented in terms of (1) the perspectives of individuals and the groups interested in research in science fields; (2) uneven access to literature through systems; and (3) linguistic isolation.

2. See, for example, Pawson (2006) for a realist approach to theory testing in an evidence-based policy context, and Chamberlin (1890) for discussion of “multiple working hypotheses”.

3. Sandelowski and Barroso (2007) and Thorne, Jensen, Kearney, Noblit, and Sandelowski (2004) do not categorize meta-science studies as a type of research synthesis. While we agree that not all meta-science studies are synthetic, some are or could be classified as such.

4. According to Garfield and Sher (1993), the Keywords Plus™ field includes one, two, and three term phrases algorithmically identified and selected from titles of referenced studies.

Acknowledgements

This work benefitted greatly from the guidance provided by my dissertation advisor and chair, Diane Kelly, and committee members, Bradley M. Hemminger, Joanne Gard Marshall, Lokman I. Meho, and Barbara M. Wildemuth. An earlier version of this work was submitted as a student paper to the 2012 ASIS&T Sig-Metrics Workshop. I am indebted to workshop participants, whose feedback greatly contributed to the development of this manuscript. I would also like to thank the two anonymous reviewers who provided helpful feedback and considerations for both this paper and future work.

26

REFERENCES

Ankem, K. (2005). Approaches to meta-analysis: A guide for LIS researchers. Library & Information Science Research, 27(2), 164-176.

APA Publications & Communications Board Working Group. (2008). Reporting standards for research in psychology: Why do we need them? What might they be? American Psychologist, 63(9), 839-851.

Asuncion, A., Welling, M., Smyth, P., & Teh, Y. W. (2009, June). On smoothing and inference for topic models. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (pp. 27-34). AUAI Press.

Baccini, A. & Barabesi, L. (2011). Seats at the table: The network of the editorial boards in information and library science. Journal of Informetrics, 5, 382-391.

Bastian, H., Glasziou, P., & Chalmers, I. (2010). Seventy-five trials and eleven systematic reviews a day: How will we ever keep up? PLoS Medicine, 7(9), e1000326. doi:10.1371/journal.pmed.1000326

Bastian M., Heymann S., Jacomy M. (2009). Gephi: An open source software for exploring and manipulating networks. International AAAI Conference on Weblogs and Social Media.

Batagelj, V., & Mrvar, A. (1998). Pajek. Program for Large Network Analysis. Connections, 21(2), 47–57.

Bates, M.J. (1992). Rigorous systematic bibliography. In H. D. White, M. J. Bates & P. Wilson (Eds.), For information specialists: Interpretations of reference and bibliographic work (pp. 117-130). Norwood, NJ: Ablex.

Bates, M.J. (1999). The invisible substrate of information science. Journal of the American Society for Information Science, 50(12), 1043-1050.

Bates, M. J. (2005). An introduction to metatheories, theories, and models." In Fisher, K.E., Erdelez, S., and McKechnie, L. (Eds). Theories of Information Behavior (pp. 1-24). Medford, NJ: Information Today.

Becker, B. (1996). Discourse synthesis in meta-analysis. Social Epistemology, 10, 89-105.

Bendersky, C. & McGinn, K.L. (2010). Open to negotiation: Phenomenological assumptions and knowledge dissemination. Organization Science, 21(3), 781-797.

Berkey, C.S., Hoaglin, D.C., Mosteller, F., & Colditz, G.A. (1995). A random-effects regression model for meta-analysis. Statistics in Medicine, 14(4), 395–411. doi:10.1002/sim.4780140406

Beverley, C.A., Booth, A., & Bath, P.A. (2003). The role of the information specialist in the systematic review process: A health information case study. Health Information & Libraries Journal, 20(2), 65-74.

Bhandari, M., Montori, V.M., Devereaux, P.J., Wilczynski, N.L., Morgan, D., & Haynes, R.B. (2004). Doubling the impact: Publication of systematic review articles in orthopaedic journals. Journal of Bone & Joint Surgery, 86-A(5), 1012-1016.


27

Blei, D.M., Ng, A.Y., & Jordan, M.I. (2003). Latent Dirichlet Allocation. Journal of Machine Learning, 3, 993-1022.

Blondel, V.D., Guillaume, J-L, Lambiotte, R., & Lefebvre, E. (2008). Fast unfolding of communities in large networks. Journal of Statistical Mechanices: Theory and Experiment, 2008, 10008 - . doi:10.1088/1742-5468/2008/10/P10008

Booth, A., & Brice, A. (Eds.) (2004). Evidence-based practice for information professionals: A handbook. London: Facet.

Boschma, R. (2005). Proximity and innovation: A critical assessment. Regional Studies, 39(1), 61-74.

Buckland, M. (1991). Information as thing. Journal of the American Society for Information Science, 42(5), 351-360.

Cadotte, M.W., Mehrkens, L.R., & Menge, D.N.L. (2012). Gauging the impact of meta-analysis on ecology. Evolutionary Ecology, 26(5), 1153-1167.

Chalmers, I., Hedges, L.V., & Cooper, H. (2002). A brief history of research synthesis. Evaluation & the Health Professions, 25(1), 12-37.

Chamberlin. (1890). The method of multiple working hypotheses. Science, 15(366), 92-96.

Chavalarias, D., & Ioannidis, J.P.A. Science mapping analysis characterizes 235 biases in biomedical research. Journal of Clinical Epidemiology, 63, 1205-1215.

Cooper, H. (1986). Literature-searching strategies of integrative research reviewers: A first survey. Science Communication, 8(2), 372-383.

Cooper, H., & Hedges, L.V. (1994). The handbook of research synthesis and meta-analysis. New York: Russell Sage.

Cooper, H., Hedges, L.V., & Valentine, J.C. (2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York: Russell Sage.

Cooper, H. & Patall, E.A. (2009). The relative benefits of meta-analysis conducted with individual participant data versus aggregated data. Psychological Methods, 14(2), 165-176.

Cronin, B., & Meho, L.I. (2008). The shifting balance of intellectual trade in information studies. Journal of the American Society for Information Science & Technology, 59(4), 551-564.

Davis, F.D. (1985). A technology acceptance model for empirically testing new end-user information systems: Theory and results. (Unpublished doctoral dissertation). Massachusetts Institute of Technology, Cambridge, MA.

Evidence Based Library and Information Practice: A peer-reviewed, open access journal. University of Alberta Learning Services. http://ejournals.library.ualberta.ca.libproxy.lib.unc.edu/index.php/EBLIP

Evidence-Based Medicine Working Group. (1992). Evidence-based medicine: A new approach to teaching the practice of medicine. Journal of the American Medical Association, 268(17), 2420–2425.

Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891-904.

28

Fanelli, D. Positive results receive more citations, but only in some disciplines. Scientometrics, 94, 701-709.

Ferguson, C.J., & Brannick, M.T. (2012). Publication bias in psychological science: prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods, 17(1), 120-128. doi:10.1037/a0024445

Ford, N. (2000). Improving the ‘darkness to light’ ratio in user-related information retrieval research. Journal of Documentation, 56(6), 624-643.

Garfield, E., & Sher, I.H. (1993). Keywords PlusTM—Algorithmic derivative indexing. Journal of the American Society for Information Science, 44(5), 298-299.

Glaser, B.G., & Strauss, A.L. (1971). Status passage: A formal theory. Chicago: Aldine.

Glass, G.V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5(10), 3-8.

Glass, G.V., McGaw, B., & Smith, M.L. (1981). Meta-analysis in social research (Vol. 56). Beverly Hills, CA: Sage.

Gough, D.A. (2004). Systematic research synthesis to inform the development of policy and practice in education. In G. Thomas & R. Pring (Eds.), Evidence-based practice (pp. 44-62). Buckingham, UK: Open University Press.

Grant, M.J., & Booth, A. (2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26(2), 91-108. doi:10.1111/j.1471-1842.2009.00848.x

Greenberg, S.A. (2009). How citation distortions create unfounded authority: analysis of a citation network. BMJ: British Medical Journal, 339(Jul2013), b2680-b2680. doi:10.1136/bmj.b2680

Harris, M.R. (2005). The librarian's roles in the systematic review process: a case study. Journal of the Medical Library Association, 93(1), 81-87.

Hjørland, B. (2001). Why is meta anlaysis neglected by information scientists? Journal of the American Society for Informaiton Science & Technology, 52(13), 1193-1194.

Hjørland, B. (2002). Meta-analysis should also be visible inside information science. Journal of the American Society for Informaiton Science & Technology, 53(4), 324.

Hjørland, B. (2011). Evidence-based practice: An analysis based on the philosophy of science. Journal of the American Society for Information Science & Technology, 62(7), 1301-1310.

Hider, P. & Pymm, B. (2008). Empirical research methods reported in high-profile LIS journal literature. Library & Information Science Research, 30(2), 108-114.

Huth, E. (2009). The move toward setting scientific standards for the content of medical review articles. Journal of the Royal Society of Medicine, 102(6), 247-251. doi:10.1258/jrsm.2009.09k004

Ioannidis, J.P. (2005). Why most published research findings are false. PLoS medicine, 2(8), e124. doi:10.1371/journal.pmed.0020124


29

Ioannidis, J.P.A. & Panagiotou, O.A. (2011). Comparison of effect sizes associated with biomarkers reported in highly cited individual articles and in subsequent meta-analyses. Journal of the American Medical Association, 305(21), 2200-2210. doi:10.1001/jama.2011.713

Jarvelin , K. & Vakkari, P. (1990). Content analysis of research articles in library and information science. Library & Information Science Research, 12, 395-422.

Kamada, T., & Kawai, S. (1989). An algorithm for drawing general undirected graphs. Information Processing Letters, 31(1), 7–15.

Klein, J.T. (1996). Crossing boundaries: Knowledge, disciplinarities, and interdisciplinarities. University Press of Virginia: Charlottesville, VA.

Knight, T., & Brice, A. (2006). Librarians, surgeons, and knowledge. Surgical Clinics of North America, 86(1), 71-. doi:10.1016/j.suc.2005.11.003

Kueffer, C., Niinemets, U., Drenovsky, R.E., Kattge, J., Milberg, P., Poorter, H., Reich, P.B., Werner, C., Westoby, M., Wright, I.J. (2011). Fame, glory and neglect in meta-analyses. Trends in Ecology and Evolution, 26(10), 493-494. doi:10.1016/j.tree.2011.07.007

Larsen, K.R., Monarchi, D.E., Hovorka, D.S., Bailey, C.N. (2008). Analyzing unstructured text data: Using latent categorization to identify intellectual communities in information systems. Decision Support Systems, 45(4), 884-896.

Lau, J., Schmid, C. H., & Chalmers, T. C. (1995). Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care. Journal of Clinical Epidemiology, 48(1), 45-57. doi:10.1016/0895-4356(94)00106-Z

Leydesdorff, L., Carley, S., & Rafols, I. (2013). Global maps of science based on the new Web-of-Science Categories. Scientometrics, 94(2), 589-593.

Leydesdorff, L., & Rafols, I. (2009). A global map of science based on the ISI subject categories. Journal of the American Society for Information Science & Technology, 60(2), 348-362. doi:10.1002/asi.20967

Light, R.J., & Pillemer, D.B. (1984). Summing up: The science of reviewing research. Harvard, MA: Harvard University Press.

Light, R.J., & Smith, P.V. (1971). Accumulating evidence: Procedures for resolving contradictions among different research studies. Harvard Educational Review, 41(4), 429-471.

Liu, Y., Rafols, I., & Rousseau, R. (2012). A framework for knowledge integration and diffusion. Journal of Documentation, 68(1), 31-44.

Lomas, J. (2005). Using research to inform healthcare managers’ and policy makers’ questions: From summative to interpretive synthesis. Healthcare Policy, 1(1), 55-71.

Lombard, M., Snyder-Duch, J., & Campanella Bracken, C. (2002). Content analysis in mass communication: Assessment and reporting of intercoder reliability. Human Communication Research, 28(4), 587-604.

30

Major, C.H., & Savin-Baden, M. (2010). Introduction to qualitative research synthesis: Managing the information explosion in social science research. London: Routledge.

McGrath, W.E. (1996). The unit of analysis (objects of study) in bibliometrics and scientometrics. Scientometrics, 35(2), 257-264.

McKibbon, K. A. (2006). Systematic reviews and librarians. Library Trends, 55(1), 202-215.

Meho, L.I., & Spurgin, K.M. (2005). Ranking the research productivity of library and information science faculty and schools: An evaluation of data sources and research methods. Journal of the American Society for Information Science & Technology, 56(12), 1314-1331. doi:10.1002/asi.20227

Miller, N., & Pollock, V.E. (1994). Meta-analysis and some science-compromising problems of social psychology. In W.R. Shadish & S. Fuller (Eds.), The social psychology of science (pp. 230-261). New York: Guilford.

Mone, M.A., & McKinley, W. (1993). The uniqueness value and its consequences for organization studies. Journal of Management Inquiry, 2(3), 284-296.

Montori, V.M., Wilczynski, N.L., Morgan, D., Haynes, R. B., & the Hedges Team. (2003). Systematic reviews: A cross-sectional study of location and citation counts. BMC Medicine, 1, 2.

Morris, S.A., & Goldstein, M.L. (2007). Manifestation of research teams in journal literature: A growth model of papers, authors, collaboration, coauthorship, weak ties, and Lotka's law. Journal of the American Society for Information Science & Technology, 58(12), 1764-1782.

Mulrow, C.D. (1994). Systematic reviews: Rationale for systematic reviews. British Medical Journal, 309(6954), 597-599.

Myers, G. (1991). Stories and styles in two molecular biology review articles. In C. Bazerman & J. Paradis (Eds.), Textual dynamics of the professions: Historical and contemporary studies of writing in professional communities (pp. 45-75). Madison, WI: University of Wisconsin Press.

Noblit, G., & Hare, R.D. (1988). Meta-ethnography: Synthesizing qualitative studies. Newbury Park, CA: Sage

Parker, J.N. & Hackett, E.J. (2012). Hot spots and hot moments in scientific collaborations and social movements. American Sociological Review, 77(1), 21-44.

Patsopoulos, N.A., Analatos, A.A., & Ioannidis, J.P. (2005). Relative citation impact of various study designs in the health sciences. Journal of the American Medical Association, 293(19), 2362-2366.

Pautasso, M. (2010). Worsening file-drawer problem in the abstracts of natural, medical, and social science databases. Scientometrics, 85, 193-202.

Pawson, R. (2006). Evidence-based policy: A realist perspective. Thousand Oaks, CA: Sage.

Payne, J.L., Smith, F.A., Kowalewski, M., Krause, R.A. Jr., Boyer, A.G., McClain, C.R., Finnegan, S., Novack-Gottshall, P.M. & Sheble, L. (2012). A lack of attribution: Closing the citation gap through a reform of citation and indexing practices in the sciences. Taxon, 61(6), 1349–1351.


31

Persson, O., Glänzel, W., & Danell, R. (2004). Inflationary bibliometric values: The role of scientific collaboration and the need for relative indicators in evaluative studies. Scientometrics, 60(3), 421-432.

Peters, H.P.F., & van Raan, A.F.J. (1994). On determinants of citation scores: A case study in chemical engineering. Journal of the American Society for Information Science, 45(1), 39-49.

Petticrew, M., & Roberts, H. (2006). Systematic reviews in the social sciences: A practical guide. Malden, MA: Blackwell.

Piwowar, HA, Vision, TJ, & Whitlock, MC (2011). Data archiving is a good investment. Nature, 473 (7347), 285-285. doi:10.1038/473285a

Pope, C. (2003). Resisting evidence: The study of evidence-based medicine as a contemporary social movement. Health: An Interdisciplinary Journal for the Social Study of Health, Illness & Medicine, 7(3), 267-282.

Rafols, I., & Meyer, M. (2010). Diversity and network coherence as indicators of interdisciplinarity: case studies in bionanoscience. Scientometrics, 82(2), 263-287.

Rafols, I., Porter, A.L., & Leydesdorff, L. (2010). Science overlay maps: A new tool for research policy and library management. Journal of the American Society for Information Science & Technology 61(9), 1871-1887

Rainer, R.K., Jr., & Miller, M.D. (2005). Examining differences across journal rankings. Communications of the ACM, 48(2), 91-94.

Ramage, D., & Rosen, E. (2011). Stanford topic modeling toolbox. http://nlp.stanford.edu/software/tmt/

Riley, R.D., Lambert, P.C., & bo-Zaid, G. (2010). Meta-analysis of individual participant data: Rationale, conduct, and reporting. British Medical Journal, 340(c221).

Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638-641.

Rothstein, H.R., Sutton, A.J., & Borenstein, M. (Eds.). (2005). Publication bias in meta-analysis: Prevention, assessment and adjustments. West Sussex, UK: Wiley.

Rousseau, D.M., Manning, J., & Denyer, D. (2008). Evidence in management and organizational science: Assembling the field's full weight of scientific knowledge through synthesis. Academy of Management Annals, 2(1), 475-515.

Sandelowski, M., & Barroso, J. (2007). Handbook for synthesizing qualitative research. New York: Springer.

Sander, L., & Kitcher, H. (2006). Systematic and other reviews: Terms and definitions used by UK organizations and selected databases: Systematic review and Delphi study. National Institute for Health and Clinical Excellence (NICE). http://www.publichealth.nice.org.uk

Schell, C.L., & Rathe, R.J. (1992). Meta-analysis: A tool for medical and scientific discoveries. Bulletin of the Medical Library Association, 80(3), 219-222.

32

Schneider, J.W. Caveats for using statistical significance tests in research assessments. Journal of Informetrics, 7, 50-62.

Schulze, R. (2004). Meta-analysis: A comparison of approaches. Cambridge, MA: Hogrefe & Huber.

Shadish, W. R., & Sweeney, R. B. (1991). Mediators and moderators in meta-analysis: There's a reason we don't let dodo birds tell us which psychotherapies should have prizes. Journal of Consulting and Clinical Psychology, 59(6), 883-893. doi:10.1037/0022-006X.59.6.883

Shell, L., Hofstetter, S., Carlock, D., & Amani, J. (2006). Survivor's guide for the novice: A simplified model for a collaborative systematic review. Journal of Hospital Librarianship, 6(4), 1-12.

Sidlauskas, B., Ganeshkumar, G., Hazkani-Covo, E., Jenkins, K., Lapp, H., McCall, L.W., Price, S., Scherle, R., Spaeth, P.A., & Kidd, D.M. (2010). Linking big: The continuing promise of evolutionary synthesis. Evolution, 64(4), 871-880.

Small, H.G. (1986). The synthesis of specialty narratives from co-citation clusters. Journal of the American Society for Information Science, 37(3), 97-110.

Sterling, T.D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance — or vice versa. Journal of the American Statistical Association, 54(285), 30-34.

Strike, K., & Posner, G. (1983). Epistemological problems in organizing social science knowledge for application. In S.A. Ward & L.J. Reed (Eds.), Knowledge structure and use: Implications for synthesis and interpretation (pp. 45-85). Philadelphia, PA: Temple University Press.

Suber, P. (2012). Open access. Cambridge, MA: MIT Press.

Sugimoto, C.R., Pratt, J.A., & Hauser, K. (2008). Using field cocitation analysis to assess reciprocal and shared impact of LIS/MIS fields. Journal of the American Society for Information Science & Technology, 59(9), 1441-1453.

Swanson, D.R. (1986). Undiscovered public knowledge. Library Quarterly, 56(2), 103-118.

Swinkels, A., Briddon, J., & Hall, J. (2006). Two physiotherapists, one librarian and a systematic literature review: collaboration in action. Health Information & Libraries Journal, 23(4), 248-256. doi:10.1111/j.1471-1842.2006.00689.x

Thompson, S.G., & Higgins, J.P.T. (2002). How should meta-regression analyses be undertaken and interpreted? Statistics in Medicine, 21(11), 1559-1573. doi:10.1002/sim.1187

Thorne, S., Jensen, L., Kearney, M.H., Noblit, G., & Sandelowski, M. (2004). Qualitative metasynthesis: Reflections on methodological orientation and ideological agenda. Qualitative Health Research, 14(10), 1342-1365.

Trinder, L., & Reynolds, S. (2000). Evidence-based practice: A critical appraisal. Oxford, UK: Blackwell Science.

Toews, L. (2011). The information infrastructure that supports evidence-based veterinary medicine: A comparison with human medicine. Journal of Veterinary Medical Education, 38(2), 123-134. doi:10.3138/jvme.38.2.123


33

Twenge, J.M. (2011). The duality of individualism: Attitudes toward women, Generation Me, and the method of cross-temporal meta-analysis. Psychology of Women Quarterly, 35(1), 193-196. doi:10.1177/0361684310395918

Urquhart, C. (2011). Meta-synthesis of research on information seeking behaviour. Information Research-an International Electronic Journal, 16(1).

Uthman, O.A., Okwundu, C.I., Wiysonge, C.S., Young, T., & Clarke, A. (2013). Citation classics in systematic reviews and meta-analyses: Who wrote the top 100 most cited articles? PLoS ONE, 8(10), e78517. doi:10.1371/journal.pone.0078517

Wallace, M.L., Larivière, V., & Gingras, Y. (2009). Modeling a century of citation distributions. Journal of Informetrics, 3(4), 296-303.

White, H.D. (1992). Literary forms in information work: Annotated bibliographies, bibliographic essays, and reviews of literatures. In H. D. White, M. J. Bates & P. Wilson (Eds.), For information specialists: Interpretations of reference and bibliographic work (pp. 131-149). Norwood, NJ: Ablex.

White, H.D. (1994). Scientific communication and literature retrieval. In H. Cooper & L.V. Hedges (Eds.), The handbook of research synthesis. New York: Russell Sage. (pp. 40-55)

White, H.D. (2002). Library and information science (LIS) in aid of meta-analysis. Journal of the American Society for Information Science & Technology, 53(4), 323.

White, H.D. (2009). Scientific communication and literature retrieval. In H. Cooper, L.V. Hedges, & J.C. Valentine (Eds.), The handbook of research synthesis. New York: Russell Sage. (pp. 51-71)

Willinsky, J. (2006). The access principle: The case for open access to research and scholarship. Cambridge, MA: MIT Press. Available at: http://hdl.handle.net/10150/106529

Yoshii, A., Plaut, D.A., McGraw, K.A., Anderson, M.J., & Wellik, K.E. (2009). Analysis of the reporting of search strategies in Cochrane systematic reviews. Journal of the Medical Library Association, 97(1), 21-29. doi:10.3163/1536-5050.97.1.004

Young, A.L., & Lutters, W.G. (2014). Collaborative interpretation in land change science meta-studies. Paper presented at the 17th ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW 2014), 15-19 February 2014, Baltimore, Maryland, USA.

34

APPENDIX

Topic summary: For topics T1-T16, topic labels were subjectively selected based on review of words and most closely associated topics. For each topic, most prevalent words and sum of probable assignment of word occurrences per topic are indicated.

T1: Health 1589.7 T2: Decisions 1173.3 T3: Methods 1662.7 T4: Meta-study 1989.1 health 168.51 decis 69.21 method 84.66 journal 97.06 campaign 35.00 support 53.73 meta-analysi 51.92 articl 71.40 commun 32.64 tool 39.97 paper 42.49 publish 65.50 behavior 30.51 group 38.63 result 41.89 scienc 58.03 design 26.46 qualiti 36.11 literature 32.53 index 44.35 literatur 24.19 impact 32.39 techniqu 28.49 meta-analysi 38.50 literaci 23.91 polici 27.66 type 26.48 public 38.38 public 23.20 aid 25.87 variabl 23.44 librari 35.90 effect 22.27 develop 22.83 applic 21.90 analysi 35.22 articl 21.54 level 17.24 quality 21.80 field 34.34 adopt 20.89 process 16.13 meta-synthesi 20.83 report 30.20 scienc 20.10 facilit 14.28 conduct 18.44 paper 30.00 examin 18.37 system 12.36 common 18.39 relat 25.19 systemat 18.24 organ 11.89 quantit 17.86 between 23.15 intervent 17.93 perform 10.97 methodolog 16.81 methodolog 21.46 promot 17.32 evalu 10.77 quality 16.74 number 20.03 peopl 17.17 appropri 10.55 clear 16.56 author 19.53 dissemin 16.70 when 10.29 approach 16.24 ethic 17.96 visual 15.87 incent 9.97 unit 15.76 year 17.77 diffus 14.92 show 9.93 statist 15.00 issu 17.11

T5: Models & Theories 1652.9

T6: Data sources 1959.6 T7: Effects 1111.8

T8: Systems & Tech 1629.8

model 60.21 systemat 127.00 effect 75.29 issu 38.21 theori 60.14 databas 121.35 patient 58.22 system 29.78 develop 37.88 citat 60.20 advers 34.14 synthesi 28.33 literatur 27.84 sourc 53.97 drug 31.68 articl 27.70 present 25.55 refer 53.72 approach 29.68 literatur 27.23 process 24.37 retriev 50.93 develop 23.63 model 26.10 practic 20.49 identifi 48.09 mean 19.96 develop 25.31 knowledg 19.70 includ 41.23 map 19.77 digit 24.65 critic 18.48 method 34.29 sourc 16.04 data 21.56 system 18.04 abstract 34.26 evalu 15.45 integr 21.36 understand 17.18 relev 32.40 need 14.83 within 20.87 contribut 16.78 literatur 29.15 cancer 14.19 framework 20.84 approach 16.41 health 27.92 symptom 13.99 analysi 19.71 action 15.43 effect 27.40 condit 13.90 manag 18.95 organiz 15.18 medlin 25.28 treatment 13.89 workflow 18.88 concept 14.60 number 24.89 criteria 13.87 challeng 17.23 institut 14.07 strategi 23.88 data 13.37 process 17.08 propos 13.95 index 22.68 medic 13.23 collabor 16.67 build 13.77 record 22.33 model 12.80 scientist 16.49 implic 12.97 differ 22.17 analys 12.49 thei 16.17


35

T9: Text

analysis 1199.8 T10: Search strategies 1759.1 T11: EBP 1851.5

T12: Patient Care 2329.6

perform 57.26 literatur 61.64 evid 124.96 patient 82.66 system 49.92 cochran 39.50 clinic 114.54 care 72.75 systemat 39.29 case 39.31 question 104.66 clinic 61.64 topic 30.10 includ 34.25 evidence-bas 68.94 system 57.71 measur 26.69 provid 33.85 medicin 63.36 outcom 55.95 time 26.30 select 32.76 care 58.15 medic 54.97 articl 22.35 report 29.20 answer 52.93 effect 52.60 data 20.93 medic 25.99 practic 48.15 prevent 47.71 evalu 20.79 reliabl 24.48 librarianship 32.76 evid 44.53 work 20.64 strategi 24.38 resourc 28.16 improv 39.51 classif 20.03 articl 23.71 articl 26.09 impact 33.32 approach 19.81 checklist 20.96 physician 23.79 trial 32.38 autom 18.97 peer 20.61 level 22.92 systemat 29.99 improv 15.94 author 19.52 primari 22.77 health 29.60 valid 15.20 guidelin 18.95 medic 22.41 signific 29.18 classifi 13.94 document 18.76 guidelin 21.40 remind 25.99 result 13.87 book 18.73 sourc 21.19 physician 25.73 train 13.35 year 18.23 how 19.98 evalu 25.07 effici 12.34 thei 18.07 ebm 17.99 control 24.92 text 12.32 kei 17.96 manag 17.68 clinician 24.72

T13: Search Filters 1634.0 T14: HII 1628.4

T15: Services & Libraries 1909.8

T16: Tech acceptance 2000.8

filter 87.96 onlin 39.58 servic 75.96 effect 74.74 sensit 69.94 differ 33.63 systemat 71.46 meta-analysi 67.36 medlin 65.17 behaviour 31.94 librari 70.72 technolog 65.11 strategi 65.13 mai 28.18 librarian 55.64 model 62.54 trial 58.61 how 27.86 evalu 34.70 system 52.47 term 54.12 knowledg 26.41 literatur 34.11 user 47.77 precis 51.13 need 22.45 practic 32.75 factor 43.76 identifi 44.48 find 19.86 need 30.33 success 43.53 retriev 42.46 chapter 16.95 skill 28.82 accept 42.91 clinic 35.52 new 16.77 health 28.56 result 40.17 control 32.73 approach 16.12 provid 27.11 moder 38.91 specif 32.31 gender 15.95 profession 25.13 size 35.52 record 31.64 what 15.69 role 22.91 find 34.60 rct 30.93 thei 15.60 collabor 21.86 implement 31.72 test 28.19 method 15.34 conduct 21.29 task 31.05 random 27.98 seek 13.91 student 19.58 between 28.22 set 23.69 analysi 13.76 develop 18.26 support 24.83 design 22.85 examin 13.18 impact 17.96 empir 24.22 combin 20.51 when 12.88 program 17.07 tam 23.99 pubm 19.10 where 12.75 e-learn 15.96 manag 23.74

Documents

Research synthesis methods and library and information