How Trustworthy Is the Scientiï¬c Literature in Industrial and

Industrial and Organizational Psychology, 6 (2013), 252–268.Copyright © 2013 Society for Industrial and Organizational Psychology. 1754-9426/13

FOCAL ARTICLE

How Trustworthy Is the ScientificLiterature in Industrial andOrganizational Psychology?

SVEN KEPES AND MICHAEL A. MCDANIELVirginia Commonwealth University

AbstractThe trustworthiness of research findings has been questioned in many domains of science. This article callsfor a review of the trustworthiness of the scientific literature in industrial–organizational (I–O) psychologyand a reconsideration of common practices that may harm the credibility of our literature. We note that mosthypotheses in I–O psychology journals are confirmed. Thus, we are either approaching omniscience or ourjournals are publishing an unrepresentative sample of completed research. We view the latter explanation asmore likely. We review structural problems in the publication process and in the conduct of research that islikely to promote a distortion of scientific knowledge. We then offer recommendations to make the I–O literaturemore accurate and trustworthy.

‘‘False facts are highly injurious to theprogress of science for they often endurelong.’’Charles Darwin (1981/1871, p. 385)

In recent years, trust in many scientificareas has come under scrutiny (Stroebe,Postmes, & Spears, 2012). The medicalsciences have received particular atten-tion. For example, some pharmaceuticalcompanies appear to have withheld datato hide results that cast their products ina negative light (e.g., Berenson, 2005;Dickersin, 2005; Goldacre, 2012; Krimsky,2006; Sarawitz, 2012; Saul, 2008; Whalen,Barrett, & Loftus, 2012). The situation isnot much different in other scientific areas,

Correspondence concerning this article should be add-ressed to Sven Kepes.E-mail: [email protected]

Address: Virginia Commonwealth University, 301West Main Street, PO Box 844000, Richmond, VA23284

This focal article has benefited from feedbackprovided by several colleagues.

ranging from the medical sciences, chem-istry, and physics to the social sciences(e.g., Braxton, 1999; Gallup Organization,2008; Goldstein, 2010; LaFollette, 1992;Park, 2001; Reich, 2010; Stroebe, Postmes,& Spears, 2012).

Psychology has not escaped con-cerns about scientific credibility. Forinstance, previously ‘‘established’’ effectsin many areas of psychology such as theMozart effect on spatial task performance(Rauscher, Shaw, & Ky, 1993), socialpriming (Bargh, Chen, & Burrows, 1996),psychic effects (Bem, 2011), verbal over-shadowing (Schooler & Engstler-Schooler,1990), and ‘‘practical’’ intelligence mea-sures purported to yield a general factor thatis substantially different from g (Sternberget al., 2000) have been questioned orrefuted (e.g., Doyen, Klein, Pichon, &Cleeremans, 2011; Francis, 2012; Lehrer,2010; McDaniel & Whetzel, 2005; Ritchie,Wiseman, & French, 2012; Rouder &Morey, 2011; Yong, 2012a). Recently,substantial attention has been focusedon allegations of scientific misconduct in

252

How trustworthy is the scientific literature in I–O psychology? 253

Stapel’s social psychology research (seeStroebe et al., 2012, for a discussion ofStapel’s multiple publications asserted tobe based on fabricated or inappropriatelymanipulated data). As a result of theseand other controversies, some have con-cluded that the reputation of science ingeneral, including the scientific principlesresearchers purport to uphold, has sufferedsevere damage (e.g., Krimsky, 2006).

Most commonly, ‘‘established’’ effectsare not totally discredited, but the mag-nitude of the effects (e.g., bilateral bodysymmetry and health; a test validity; meanracial differences in job performance) maybe different, sometimes substantially, fromthe purported values (Banks, Kepes, &McDaniel, 2012; McDaniel, McKay, &Rothstein, 2006; Van Dongen & Gangestad,2011). Such findings are not likely theresult of fraud but may reflect suppressionof some research findings and othernonoptimal practices of authors and editorsor reviewers.

When we suggest that the magnitudeof effects may be different from the pur-ported values, we rely on evidence frompublication bias analyses that estimates theextent to which the purported values arelikely to be distorted because they are notrepresentative of all research completedon a particular topic (e.g., Banks, Kepes, &Banks, 2012; Banks, Kepes, & McDaniel,2012; Kepes, Banks, McDaniel, & Whetzel,2012; Kepes, Banks, & Oh, in press;McDaniel, Rothstein, & Whetzel, 2006;Renkewitz, Fuchs, & Fiedler, 2011; Sutton,2005). We refer to missing studies as being‘‘suppressed,’’ a term drawn from thepublication bias literature (Kepes, Banks,McDaniel, & Whetzel, 2012; Rothstein,Sutton, & Borenstein, 2005b; Sutton, 2009).This suppression is not necessarily theresult of a purposeful intent to hide results.Rothstein, Sutton, and Borenstein (2005a,p. 3) mention several potential data orresults suppression mechanisms that candistort our cumulative knowledge. Thesemechanisms include the outcome bias (theselective reporting of some outcomes butnot others in primary studies, depending on

the direction and statistical significance ofthe results), the language bias (the selectiveinclusion of samples from studies publishedin the English language in meta-analyticreviews), the availability bias (the selectiveinclusion of samples from studies in ameta-analytic review that are readily avail-able to a researcher), and the familiaritybias (the selective inclusion of samplesfrom studies in meta-analytic reviews fromthe researcher’s own scientific discipline).Thus, most forms of suppression are notlikely to be a function of explicit intent todistort or hide research results.

We offer that it is time to con-sider the trustworthiness of research inindustrial–organizational (I–O) psychol-ogy. We came to this position in twosteps. First, we note that hypotheses inI–O journals are almost always sup-ported, which is consistent with otherdisciplines of psychology (e.g., Sterling,1959; Sterling & Rosenbaum, 1995; Yong,2012a).1 Although substantial support forhypotheses is a phenomenon across allsciences, the percentage of significantresults reported in psychology journals isnoticeably higher than that in most scien-tific disciplines (Fanelli, 2010b; Sterling,1959; Sterling & Rosenbaum, 1995; Yong,2012a). Furthermore, although there isan increase over time in the percentageof confirmed hypotheses, particularly inU.S. journals, the growth is strongest inpsychology (Fanelli, 2012). Second, weconsidered reasons for why hypotheses areconfirmed so consistently. We suggest thatI–O researchers are either approachingomniscience or there are forces at work

1. Some might note that a rejected null hypothesisdoes not necessarily constitute a confirmed alter-native hypothesis. In this article, we use phrasessuch as ‘‘confirmed hypothesis’’ and ‘‘supportedhypothesis’’ to indicate that there was an inferencethat the alternative hypothesis was supported pri-marily based on a finding of statistical significance.In addition, some may assert that the reliance onstatistical significance is waning. We disagree withthis assertion. For example, Leung (2011) reportedthat all 45 quantitative articles published in theAcademy of Management Journal in 2009 involvednull hypothesis testing.

254 S. Kepes and M.A. McDaniel

that cause I–O journal articles to beunrepresentative of all completed I–Oresearch. We favor the latter explanation.

We acknowledge that the issues weraise about the I–O psychology literatureare not unique to I–O psychology. Rather,the evidence that we offer is drawn fromresearch in several scientific disciplines,and we assert that our concerns andrecommendations are applicable to alldisciplines of psychology. As well, ourarticle has relevance to a broad rangeof scientific disciplines. Still, our primaryinterest is in the I–O literature and itsscientific credibility, and we frame thearticle with respect to our literature.

Research-active I–O psychologists aredrawn to the field, at least in part, becausethey wish to advance scientific knowledge.These scholars typically invest substantialtime and effort in this pursuit. Similarly, edi-tors and reviewers are well-accomplishedresearchers who devote considerabletime to evaluating research and offeringsuggestions for improvement, typically ona voluntary basis. These scholars deservesubstantial respect because of their effortsto advance scientific knowledge. Yet, weargue that many practices common to thescientific process, including the editorialreview process, no matter how well inten-tioned and no matter how common, couldbe damaging to the accuracy of our cumu-lative scientific knowledge. We offer thisarticle to highlight our concerns, to suggesta reexamination of common practices, andto encourage constructive debate. We seekto offer light rather than heat, but somecommentary will necessarily invoke both.

In this article, we discuss the chasefor statistical significance (Ferguson &Heene, 2012) that results in a strongtendency to obtain confirmed hypotheses.We then review structural issues in thepublication process that may drive thischase for significance. We identify issuesin the conduct of research that contributeto our specious omniscience. Finally,we offer recommendations for improvingthe trustworthiness of science in I–Opsychology.

Structural Problems in theScientific Process

For decades, we have known that studieswith statistically significant results are morelikely to get published in our journalsthan studies with nonsignificant results(Greenwald, 1975; Orlitzky, 2012; Porter,1992; Rosenthal, 1979; Sterling, 1959).Articles with statistically significant, largemagnitude effect sizes tend to dominate ourjournals (Greenwald, 1975; Orlitzky, 2012;Porter, 1992; Rosenthal, 1979; Sterling,1959; Sterling & Rosenbaum, 1995). Ina review of psychology journals, Sterling(1959) found that 97% of the articlesrejected the null hypothesis. In a replicationmore than 36 years later, Sterling andRosenbaum (1995) reported essentiallyidentical results. We offer these findingsas evidence supporting the inference ofstructural problems in the psychologyscientific process that should also affect I–Opsychology.

Academic journals are in a competitivemarket (Jefferson, 1998). They seek topublish ‘‘hot new findings’’ (Witten & Tib-shirani, 2012) because articles with suchfindings get cited more often than otherarticles, and these citations enhance thereputation of a journal as judged by impactfactors and related indices. In many scien-tific areas, the most heavily cited articlesare concentrated in a small number of jour-nals (Ioannidis, 2006). Relative to articleswith unsupported hypotheses, articles withsupported hypotheses tend to be judgedmore interesting, capture greater attention,and boost the reputation of a journal. This islikely to be particularly true for prestigiousjournals (Murtaugh, 2002). Therefore, thereward structure for academic journals mayinadvertently encourage the acceptanceof articles with supported hypotheses.Relatedly, the reward structure could alsoencourage the rejection of studies that donot find support for hypotheses.

Academic departments in I–O psychol-ogy are also in a competitive market. Thehighest ranked programs are motivatedto stay highly ranked. The lesser ranked


programs are motivated to improve theirranking. Thus, programs tend to encouragetheir faculty and graduate students topublish in high impact journals (Giner-Sorolla, 2012). Some faculty and academicprograms may even hold the norm thatonly publications in elite journals areworthwhile, and researchers are encour-aged to abandon research studies thathave been rejected from such journals.Even in the absence of such norms, facultypublications and the quality of the journalsare emphasized in faculty employmentactions, including tenure, promotion,salary increases, and discretionary funding(e.g., Gomez-Mejia & Balkin, 1992). I–Opsychology graduate students pursuingacademic careers typically seek to find ajob in the best-ranked academic programthat they can. They recognize that theiracademic placement is substantially drivenby their publications. Thus, academicresearchers seek to publish in the most elitejournals and recognize that the probabilityof such publications relies in part on havinghypotheses that are supported.

Given the scientist–practitioner empha-sis of I–O psychology, many I–O psychol-ogists employed in consulting, industry,and government also seek to publish. Theconsultant’s research often relates to theirorganization’s products or services. Industryand governmental researchers often wish todocument the efficacy of an organization’sI–O practices (e.g., selection systems orteam practices). These I–O psychologistsstrive for publications with supportedhypotheses in part to serve the commercialor reputational interests of their organi-zations (e.g., ‘‘our products and serviceswork well;’’ ‘‘our team-based workforce iseffective’’).

Because of the reward structure (Koole& Lakens, 2012; Nosek, Spies, & Motyl,2012), researchers are motivated to ‘‘chasethe significant’’ (Ferguson & Heene, 2012,p. 558) in order to find support for hypothe-ses (Fanelli, 2010a; Jasny, Chin, Chong, &Vignieri, 2011; Koole & Lakens, 2012;Nosek et al., 2012; Wagenmakers, Wetzels,

Borsboom, van der Maas, & Kievit, 2012).2

This chase of the significant can lead toa high rate of false-positive publishedfindings and thus the distortion of science(Ioannidis, 2005, 2012; Sarawitz, 2012;Simmons, Nelson, & Simonsohn, 2011).False-positive results are very ‘‘sticky’’(i.e., they rarely get disconfirmed) becausenull results have many possible causes. Inaddition, failures to replicate false positivesare seldom conclusive (Ferguson & Heene,2012; Simmons et al., 2011). Note that weare not arguing that our literature is entirelycomposed of zero-magnitude populationeffect sizes that are falsely presented asnonzero. Rather, we are arguing that it iscommon for the magnitude of our popu-lation effect sizes to be misestimated, oftenoverestimated (Kepes, Banks, McDaniel,& Whetzel, 2012). Thus, although authorsand journals seek to improve science, ouractions in the ‘‘chase for significance’’may actually damage it. It seems as if theacademic community could be ‘‘rewardingA, while hoping for B’’ (Kerr, 1975, p. 769).

Issues in the Conduct of Researchand the Editorial Process ThatContribute to our SpeciousOmniscience

Researchers have substantial methodolog-ical flexibility (Bedeian, Taylor, & Miller,2010; Simmons et al., 2011; Stroebe et al.,2012) that can be marshaled to obtainsupported hypotheses. In part because ourfield tends to develop several measuresfor the same construct, researchers havethe flexibility to use multiple operational-izations of constructs in order to identifythe measurement approaches that bestsupport their hypotheses (e.g., ‘‘throw lotsof variables against the wall and talk aboutthe variables that stick’’). Researchers canstop data collection, tweak the design or

2. In the next section of this article (Issues in theConduct of Research and the Editorial ProcessThat Contribute to our Specious Omniscience),we describe specific actions authors can take tofacilitate the finding of statistical significance.


measurement, and discard the original dataand collect new data. Researchers can dropoutliers or other observations that diminishthe magnitude of obtained effects. They cancollect additional data needed to increasesample size to move a marginally statisti-cally significant effect size (e.g., p < .06) toa significance level that is more acceptableto journals (e.g., p < .05).3 Researcherscan also terminate data collection oncethe preferred p value is obtained. Theycan alter the analysis method to identifythe analysis approach that best supportsthe hypothesis. They can also abandonthe hypotheses that were not supported.Researchers can then report the results thatbest support the retained hypotheses in anice, neat publishable package and nevermention the discarded variables, analyses,observations, and hypotheses.

If one is to trust Bedeian et al.’s(2010) survey of management faculty4 thatexamined knowledge of methodologicalflexibility in their colleagues during theprevious year, instances of methodologicalflexibility are not rare events. For example,60% of faculty knew of a colleague who‘‘dropped observations or data points fromanalyses based on a gut feeling that theywere inaccurate.’’ Fifty percent of facultyknew of a colleague who ‘‘withheld datathat contradicted their previous research.’’Other questionable practices are also fairlycommon (see Bedeian et al., 2010).

If methodological flexibility does notyield the desired support of a hypothesis,researchers can simply change the hypothe-ses to match the results (HARKing: hypoth-esizing after the results are known; Kerr,

3. We are not arguing against increasing one’s samplesize. We also note that increasing one’s samplesize would not, on average, cause the resultingeffect size to be overestimated. We do argue thatincreasing one’s sample size solely to move amarginally statistically significant effect size intoa state of statistical significance is an example ofmethodological flexibility (Simmons et al., 2011).

4. We note that many I–O psychologists are employedin management departments due to the generallyhigher salaries than can be obtained in psychologydepartments.

1998). The Bedeian et al. (2010) surveyreported that approximately 92% of facul-ties know at least one colleague who hasengaged in HARKing in the last year. Thus,HARKing is likely to be a common practicein I–O psychology.

But it is not only researchers who engagein practices such as HARKing. Reviewersand editors sometimes encourage thesepractices as part of well-intentioned effortsto improve submitted manuscripts (Leung,2011; Rupp, 2011). In essence, editors andreviewers function as gatekeepers (Crane,1967; Rupp, 2011). Thus, it is not unheardof for editors and reviewers to suggesthypotheses or ask authors to drop (or add)analyses, results, or hypotheses to push theeditor or reviewer’s perspective or researchagenda (Leung, 2011; Rupp, 2011), or tosimply make the paper more ‘‘interesting.’’In the most extreme case, if HARKing can-not yield a supported hypothesis, authorscan fabricate their data to be consistentwith the hypotheses (Carey, 2011; Crocker& Cooper, 2011; LaFollette, 1992; Stroebeet al., 2012; Vogel, 2011; Yong, 2012b).

In summary, there is a growing bodyof evidence indicating that I–O-relatedresearch is affected by publication bias (e.g.,Ferguson & Brannick, 2012; Kepes, Banks,McDaniel, & Whetzel, 2012; Rothstein,2012; Rothstein et al., 2005a). That is, ourpublished or readily accessible research isnot representative of all completed I–Oresearch on a particular relation of interest.Thus, the I–O research literature maylikely contain an uncomfortably high rateof false-positive results (i.e., the incorrectrejection of a null hypothesis) and othermisestimated effect sizes. This can havea long-lasting distorting effect on ourscientific knowledge, particularly becauseexact replications tend to be discouragedby our journals (Makel, Plucker, & Hegarty,2012; Neuliep & Crandall, 1990, 1993;Simmons et al., 2011; Yong, 2012a).Because of the lack of such replicationstudies, some scholars believe that oursciences have lost the ability to self-correct and generate accurate cumulativeknowledge (Giner-Sorolla, 2012; Ioannidis,


2005, 2012; Koole & Lakens, 2012;Nosek et al., 2012). The lack of exactreplication studies prevents the opportunityto disconfirm research results and thus tofalsify theories (Kerr, 1998; Leung, 2011;Popper, 1959). This has led to a ‘‘vastgraveyard of undead theories’’ (Ferguson &Heene, 2012, p. 555). We note that we havemany theories, and only a small numberhave been refuted through replication.

Furthermore, even if replication studiesare conducted, it may take a considerableamount of time before such studies becomepublically available (a time lag bias; Banks,Kepes, & McDaniel, 2012; Trikalinos& Ioannidis, 2005). This situation hascaused Lehrer (2010) to question thescientific methods in the social sciences(see also Giner-Sorolla, 2012; Ioannidis,2005, 2012; Koole & Lakens, 2012; Noseket al., 2012; Stroebe et al., 2012; Yong,2012a). In Lehrer’s (2010, p. 52) view,‘‘the truth wears off’’ because facts are‘‘losing their truth: claims that have beenenshrined . . . are suddenly unprovable.’’Relatedly, Ioannidis (2012) asserted thatmost scientific disciplines may be produc-ing wrong or distorted information on amassive scale (see also Ioannidis, 2005).

Results from multiple studies havedocumented likely suppression bias (e.g.,declining to report effect sizes in a pub-lished study or not publishing a study in itsentirety). In the most comprehensive reviewof publication bias in psychology, Fergusonand Brannick (2012) reported that publi-cation bias was present in approximately40% of meta-analyses and that the degreeof this bias was worrisome in about 25%of meta-analyses. This bias has resulted inmisestimating the magnitude of populationeffects in several I–O research domains. Forexample, results were found consistent withthe inference that the validities of somecommercially available employment testsare overestimated, sometimes substantially(McDaniel, Rothstein, & Whetzel, 2006). Inaddition, Renkewitz et al. (2011) reportedthat publication bias exists in the literatureon judgment and decision making. Further-more, Banks, Kepes, and McDaniel (2012)

found evidence consistent with an inferenceof publication bias in the literature onconditional reasoning tests. Similarly,Kepes, Banks, and Oh (in press) analyzedfour datasets from previously publishedI–O meta-analyses and reported that threeof these datasets (work experience andperformance, gender differences in transfor-mational leadership, and Pygmalion inter-ventions) are likely to have been affected bypublication bias. Together, these and otherstudies suggest that at least some literatureareas in our field have a bias against thepublishing of null, contrarian (i.e., effectsizes that are counter to already publishedfindings), and small magnitude effect sizes.5

Having offered evidence that concernsare likely warranted about the trustwor-thiness of the I–O psychology researchliterature, we now offer recommendationson improving the credibility and accuracyof the I–O psychology research literature.

Recommendations

Create research registries. We offer that oneof the most effective ways to overcometrustworthiness concerns in I–O psychol-ogy is the creation and mandatory use ofresearch registries (Ferguson & Brannick,2012; Kepes, Banks, McDaniel, & Whetzel,2012). A research registry is a database inwhich researchers register studies that theyplan to conduct (Banks & McDaniel, 2011;Berlin & Ghersi, 2005). Such registries existacross scientific disciplines, particularly inthe medical sciences (Berlin & Ghersi,2005). When study registration occurs prior

5. We acknowledge that Dalton, Aguinis, Dalton,Bosco, and Pierce (2012) concluded that pub-lication bias is not worrisome in our literature.However, that paper stands alone in that con-clusion. We note that the Dalton et al. (2012)effort differed from all other published publicationbias analyses in that did not examine any specificresearch topic in the literature (Kepes, McDaniel,Brannick, & Banks, 2013). As such, we do not findit an informative contribution to the publicationbias literature (i.e., publication bias is concernedwith the availability of effect sizes on a particularrelation of interest).


to data collection and, thus, before theresults are known, the possibility thatresults are being fabricated or suppressedis minimized (Chalmers et al., 1987; Dick-ersin, 1990; Easterbrook, Gopalan, Berlin,& Matthews, 1991). In addition, when con-ducting a review on a particular relation ofinterest, researchers can search the registryfor information on unpublished samples,which can then be included in the review(Kepes et al., 2012; Rothstein, 2012). Thus,research registries would promote greatercomprehensibility and credibility in thereviews and summaries of our literatures.

Although many registries exist in otherscientific areas, none exist in I–O psychol-ogy and related disciplines in the organiza-tional sciences (Kepes, Banks, McDaniel, &Whetzel, 2012).6 Registries in the medicaland related sciences are typically linked toa particular organization. For example, theClinicalTrials.gov registry is provided by theU.S. National Institutes of Health. Anotherregistry is available through the CochraneLibrary (Dickersin et al., 2002), and even theU.S. Food and Drug Administration oper-ates a registry (Turner, 2004). To encouragethe use of registries, the International Com-mittee of Medical Journal Editors, in 2005,required registration before a clinical trialstudy was conducted as a condition for pub-lication in the participating journals. Thishas led to studies being routinely registered(Laine et al., 2007). We recommend thatjournals in I–O psychology and relateddisciplines require registration of studiesprior to the collection of data. Leading

6. We note, however, the existence of some attemptsto create research registries. For instance, PsycFile-Drawer (http://psychfiledrawer.org) is an archiveof replication attempts in experimental psychol-ogy. The Campbell Collaboration (http://www.campbellcollaboration.org) covers the fields ofeducation, crime and justice, and social wel-fare, and is modeled after the very success-ful and prestigious Cochrane Collaboration inthe medical sciences. One of the most inter-esting attempts is the Open Science Framework(http://openscienceframework.org), currently in itsbeta version, which allows for the documentationand archiving of studies (Open Science Collabora-tion, 2012).

associations and organizations within ourfield, such as the American PsychologicalAssociation, the Society for Industrial Orga-nizational Psychology, or the Academy ofManagement could take a leading rolein the establishment of registries (Kepes,Banks, McDaniel, & Whetzel, 2012). Alter-natively, individual journals or consortiumsof journals could establish and main-tain registries.7 In establishing registries,the Open Science Framework could assist(Open Science Collaboration, 2012). Uni-versities might also require study registra-tion as many do now for institutional reviewboard (IRB) approval when the registry is tocontain data from individuals. Regardless ofwhether individual data are to be releasedinto a registry, IRB policies could requireregistration of a planned study.

If the goal of registering a study priorto data collection is to minimize problemsassociated with HARKing and methodologi-cal flexibility, one can identify a preliminarylist of data elements for a given plannedstudy to be included in a registry. Thesedata elements would include hypotheses,all variables and their operational defini-tions, a power analysis, a statement of theanticipated sample size, and a descriptionof the planned analyses. One would alsoneed data elements to identify the topicareas associated with the research as wellas contact information for the individual(s)who registered the planned study.

Changes in editorial review process. Thecurrent editorial review system is likely partof the reason for the bias against the publish-ing of null, contrarian, and small magnitudeeffect sizes. Journal reviewers and editorsmay inadvertently promote HARKing and

7. We note that, due to the interrelatedness of I–Opsychology with disciplines such as managementand social psychology, registration requirements foronly I–O psychology journals could have adverseeffects on I–O researchers and the field of I–Opsychology. Thus, we recommend that journaleditors from the leading journals in I–O psychologyand related scientific disciplines coordinate theirefforts and act in concert (similar to the effort in themedical sciences with the International Committeeof Medical Journal Editors).


other questionable practices (Leung, 2011;Rupp, 2011), such as requesting authors tochange hypotheses or to delete hypothesesthat are not supported (e.g., ‘‘we recom-mend that you revisit Hypothesis 1 byconsidering the research related to xxx;’’‘‘The lack of support for Hypothesis 3 harmsthe flow of the paper and we recommendits deletion, particularly because we seek areduction in manuscript length’’).8 Review-ers and editors are able to engage in thesenonoptimal behaviors because they see astudy’s result when reviewing a manuscript.Although well intentioned, such practicescan lead to the suppression of researchresults.

To prevent or reduce the frequency ofsuch behaviors, we recommend the imple-mentation of a two-step review process(Liberati, 1992). In the first stage, authorswould submit only part of their manuscriptfor review, specifically the introduction,method section, and the analysis approach.During this stage of the review process,editors and reviewers can make decisionsregarding soundness of the study (e.g., the-ory development, study design, measure-ment, and analysis) and the likely contribu-tion to the literature. Manuscripts that passthis stage would advance to the secondstage in which the complete manuscript,including the results and discussion, wouldbe provided. The editor and reviewerscould then assess whether the results andconclusion sections are aligned with theintroduction, theory, and method sections(i.e., verify that the authors actually didwhat they proposed during the initial sub-mission). Such a two-stage review processcould minimize post hoc rationalizing dur-ing the editorial review process. In addi-tion, authors could have less motivation toengage in HARKing and related inappropri-ate practices because the results and degreeof hypothesis confirmation would have lessimpact on the publication decision.

8. The quotes are offered as illustrative of commentseditors have made to the authors or to colleaguesof the authors. They are not direct quotes fromeditorial reviews.

As part of the second stage of the reviewprocess, we also recommend that authorsbe required to submit the raw data and rel-evant documentation, including syntax anda summary of all measures. This practicebenefits research in three ways. First, it givesreviewers and the editor the opportunityto check the data and syntax for potentialmistakes, a practice that currently is almostnever done (Schminke, 2009; Schminke &Ambrose, 2011). In addition, the researchprotocol and the list of available variablescould be reviewed. If particular variablesare left out of the analysis, reviewers couldask for their inclusion or for a clear state-ment concerning why they were excluded.

Second, this practice would ensure thatthe data are securely stored so that, at somefuture time (e.g., 5 years after the publi-cation date), the journal may publicallyrelease the data. Some would argue thatjournals have the duty to assist the scientificcommunity by permitting the reproductionof published results, an important aspectof the scientific process (Baggerly, 2010).If the raw data and syntax are madeavailable, other researchers could also useit for secondary analyses, thereby likelyadvancing our scientific knowledge. Unfor-tunately, psychologists are very reluctantto share their data (Wicherts, Borsboom,Kats, & Molenaar, 2006; Wolins, 1962).9

We note that the data from these citedstudies are not specific to I–O psychology.Although this is anecdotal evidence, theauthors of this article have often requestedI–O related datasets and many times ourrequests were refused. Thus, we do notbelieve that I–O psychologists tend tobe more generous with their data thanpsychologists in general.

Furthermore, in their reanalysis ofarticles published in two major psychologyjournals, Wicherts, Bakker, and Molenaar

9. The American Psychological Association (2002, p.1071) requires the sharing of data with ‘‘othercompetent professionals who seek to verify thesubstantive claims through reanalysis and whointend to use such data only for that purpose.’’We assert that this is a very restrictive policy thatdoes not actively promote data sharing.


(2011) found that the reluctance to sharedata is associated with weak statistical evi-dence against the null hypothesis and errorsin the reporting of statistical results. Finally,evidence from other scientific disciplinesindicates that the reproducibility of findingsfrom published articles is shockingly low(e.g., Begley & Ellis, 2012; Ioannidis et al.,2009), likely due to incomplete reportingof relevant statistical information in thepublished articles (Baggerly, 2010). Wesuspect that the situation is not muchdifferent in I–O psychology. According toeditors of the journal Science, who promotethe public release of scientific data, ‘‘wemust all accept that science is data andthat data are science’’ (Hanson, Sugden, &Alberts, 2011, p. 649). The sharing of datais thus important for the advancement andcredibility of our literature.

Third, authors would be aware that theirdata are subject to audit immediately by theeditor and reviewers and possibly later byother researchers. This should cause authorsto be more careful in their analysis andmay reduce the number of inappropriateresearch practices. In addition, because sta-tistical analyses are prone to error (Murphy,2004; Strasak, Zaman, Marinell, Pfeiffer,& Ulmer 2007; Wolins, 1962), reanalysescan ensure the accuracy of our scientificknowledge. Otherwise, our cumulativeknowledge can be adversely affected. Werecognize the increased labor demandson the editors and reviewers that a dataaudit would entail. However, with accessto raw data and related documentation,editors and reviewers could use guidelines(Simmons et al., 2011) or checklists (Noseket al., 2012) and inspect the raw data ifthey feel that it is warranted. With access tothe raw data, reviewers may also becomemore tolerant toward ‘‘imperfect’’ resultsbecause they can actually check them.

We encourage the release of datafor meta-analyses in addition to datafrom primary studies. We note that pastcontentious debates concerning differingmeta-analytic results in personality testvalidity (debates in the 1990s) and, morerecently, in integrity test validity, would

have been more clearly and reasonablyresolved if the authors had released theirdata and related documentation. Note thatwe do not wish to underestimate thecontroversies associated with a requirementto release data.

In addition, we recommend that all jour-nals that publish I–O psychology researchaccept and encourage supplemental infor-mation to be submitted. Although severalmedical journals and journals in many otherscientific disciplines permit the submis-sion of supplemental material that is storedindefinitely on the journal websites, I–Ojournals tend not to do this. Supplemen-tal information can contain material thatdoes not fit in a journal article given thepage constraints of the journal. If null effectsor unsupported hypotheses were removedfrom a paper, the deleted results and anal-yses could be placed in the supplementalmaterials. If the web has room for a millioncat videos, it certainly has room for supple-mental information for our research studies.

Encouragement of null effect publicationsand replications. The suppression ofsmall or null effect sizes is likely theprimary cause of research distortion dueto publication bias. Although there havebeen a few attempts by some journals toaddress this effect size suppression (e.g.,the Journal of Business and Psychologyhas a forthcoming special issue on thetopic of ‘‘Nothing, zilch, nil: Advancingorganizational science one null result at atime’’), one special issue is insufficient tocurtail data suppression. Instead, all of ourjournals, particularly our most prestigiousjournals, should actively promote thepublication of scientific findings regardlessof the magnitude of the effect sizes.10 Werecommend that journals reserve space ineach issue for the publication of null results.We note that since 2000, Cancer Epidemi-ology, Biomarkers & Prevention, one of the

10. We are not arguing that journals accept allsubmitted papers. We are arguing that themagnitude of effect sizes should not be a criterionin acceptance decisions.


top oncology and public health journals,contains a section on null results (Shields,2000; Shields, Sellers, & Rebbeck, 2009).

As mentioned previously, in the socialsciences, including I–O psychology, thereis a severe lack of exact replications(e.g., Makel et al., 2012; Pashler &Harris, 2012; Yong, 2012a). For the entirefield of psychology, Makel et al. (2012)estimated that between 1900 and todayonly around 1% of all published articlesin psychology journals are replicationstudies. This is unfortunate because ascientifically ‘‘true’’ effect is one ‘‘whichcan be regularly reproduced by anyonewho carries out the appropriate experimentin the way prescribed’’ (Popper, 1959, p.23). Thus, exact replications are necessaryto determine whether an observed effectis ‘‘true.’’ We therefore recommend thepublication of exact replications in ourI–O journals, including our elite journals,regardless of the results. Replications areconsidered by many to be the ‘‘scientificgold standard’’ (Jasny et al., 2011, p. 1225)because they are essential for the ability of ascientific field to self-correct, which is oneof the hallmarks of science (Merton, 1942,1973). As noted by, Schmidt (2009, p. 90):

Replication is one of the central issuesin any empirical science. To confirmresults or hypotheses by a repetitionprocedure is at the basis of any scientificconception. A replication experimentto demonstrate that the same findingscan be obtained in any other place byany other researcher is conceived as anoperationalization of objectivity. It is theproof that the experiment reflects knowl-edge that can be separated from the spe-cific circumstances (such as time, place,or persons) under which it was gained.

Studies in I–O journals typically fol-low the confirmatory or verification strat-egy, which can impair theoretical progress(Leung, 2011; Popper, 1959; Uchino,Thoman, & Byerly, 2010). This strat-egy seeks to provide confirmatory ratherthan disconfirmatory research evidence for

researcher’s beliefs (i.e., confirmation bias;Nickerson, 1998). The publication of exactreplication studies regardless of their resultscould uncover the potential prevalence ofthe confirmation bias and the publication offalse-positive or otherwise erroneous results(Yong, 2012a) as well as assist the gen-eration of accurate cumulative knowledge(Schwab, 2005).

We thus recommend the publication ofexact replication studies regardless of theresults. Some journals, such as PersonnelPsychology, already have a special sectionfor book reviews. These contributions arerelatively short. We suggest that exactreplication studies would not need morespace because the rationale for the studyand most details of the method and analysisappear in the original article that is thesubject of the replication.

Strengthen the methods-related beliefsystem. One can divide our scientificknowledge into theory-relevant andmethod-relevant beliefs (LeBel & Peters,2011). Although the distinction is notsharp, theory-relevant beliefs concern thehow and why (Sutton & Staw, 1995);that is, the theoretical mechanisms thatcause behaviors and other outcomes. Bycontrast, method-relevant beliefs concernthe procedures and processes by whichdata are measured, collected, and analyzed(LeBel & Peters, 2011). The centrality ofthese beliefs affects how obtained resultsare interpreted. Because psychology tendsto be driven by a theory-relevant beliefsystem, much more than a method-relevantbelief system, researchers tend to interpretconfirmatory results as theory relevant anddisconfirmatory results as method relevant,‘‘with the result that the researcher’shypothesis is artificially buffered fromfalsification’’ (LeBel & Peters, 2011,p. 372).11

11. We suggest that I–O psychology is more theory-oriented than methods-oriented based on researchand assertions in the psychology and managementliteratures (e.g., Hambrick, 2007; LeBel & Peters,2011; Leung, 2011).


Our elite I–O journals emphasize theimportance of ‘‘theoretical contributions,’’and papers are often rejected because theydo not make a strong enough theoreti-cal contribution. Consider the Journal ofApplied Psychology (JAP). In an editorialon the journal’s content and review pro-cess 24 years ago, Schmitt (1989, p. 844)mentioned the word ‘‘theory’’ one time,and in a context that is unrelated to the-ory development (to account for the term‘‘theoretical’’ and related words, the searchstring ‘‘theor*’’ yielded seven results):

We will publish replications, sometimesas short notes. In certain situations,there should be replication, such as areplication of a counterintuitive result orof a result that is inconsistent with theoryor past research. 12

By contrast, the most recent editorialby Kozlowski (2009) on the mission andscope as well as the review process at JAPmentions the word ‘‘theory’’ 36 times (25times excluding the references; ‘‘theor*’’was detected 54 times), often in contextsthat relate to theory development, the-ory extension, and so on. The increase intheory-related language has thus increasedover the years. The situation in other jour-nals is similar. For instance, all articlespublished in the Academy of ManagementJournal ‘‘must also make strong theoreticalcontributions’’ (the phrase ‘‘strong theoret-ical contribution’’ is bold in the original;Academy of Management Journal, 2012;see Leung, 2011).

As Leung (2011, p. 471) phrasedit, ‘‘the zeitgeist of top-notch . . . journalsmandates a huge premium on theoret-ical contributions in determining whatpapers will see the light within theirpages.’’ Hambrick (2007, p. 1346) notedthat the ‘‘blanket insistence on theory,

12. Interestingly, Neal Schmitt’s (1989) editorial expli-citly mentions that the Journal of Applied Psy-chology will publish replications. More recenteditorials have not always made this explicit, high-lighting the predisposition against replications.

or the requirement of an articulationof theory in everything we write, actu-ally retards our ability to achieve ourend: understanding.’’ Still, within the last6 years since Hambrick’s remarks, the ‘‘the-ory fetish’’ (2007, p. 1346) may havebecome stronger instead of weaker inour elite journals. Researchers follow therequirements of these journals, resulting inan emphasis on theory development at thepotential expense of methodological rigorand the unfortunate introduction of HARK-ing and other inappropriate practices to fitthe data to a theory (Bedeian et al., 2010;Ferguson & Heene, 2012; Hambrick, 2007;LeBel & Peters, 2011; Leung, 2011; Rupp,2011; Simmons et al., 2011).

Although we agree with Lewin (1952, p.169) that ‘‘there is nothing more practicalthan a good theory,’’ researchers havedeveloped much theory in the past decadeswithout properly evaluating the theories(Ferguson & Heene, 2012). HARKing andrelated inappropriate practices could havebeen encouraged by the strong emphasis ontheory. We recommend that our journalslessen their ‘‘theory fetish’’ (Hambrick,2007, p. 1346) and free up the journalspace that this stance requires. We suggestthat this journal space could be moreappropriately used for replication studiesof already promulgated theories.

Consistent with a greater emphasis ona method-focused orientation, we rec-ommend that all submitted manuscriptsinclude a section that assesses the robust-ness of the obtained results. For example,if the results section contains analyses withvarious covariates and without outliers, thisseparate section should contain the resultswith outliers as well as without covariates.The reporting of these supplemental resultswould allow a detailed assessment of therobustness of the ‘‘main’’ results. Similarly,results using different operationalizationsof particular variables should be includedin this section. For instance, dispersion,often used in unit-level research to assessconstructs such as team diversity, climatestrength, or pay variation, can be opera-tionalized in multiple ways (Allison, 1978;


Harrison & Klein, 2007), and the chosenoperationalization can affect the obtainedresults (Roberson, Sturman, & Simons,2007). However, our journals rarely reportmultiple operationalizations of a particu-lar construct and the potentially differingresults.

Relatedly, different scales for the sameconstruct may yield differing results (e.g.,personality constructs; Pace & Brannick,2010). Thus, the section on the robustnessof the results should contain the obtainedresults when using alternative operational-izations, scales, or constructs. Sensitivityanalyses may also consider different anal-ysis strategies and approaches. Thus, onewould not want to offer a conclusion thatis not replicable using other appropriateanalysis methods. The reporting of a rig-orous power analysis should also becomemore common in our published journalarticles (Simmons et al., 2011). These prac-tices could greatly reduce concerns relatedto the methodological flexibility problem(Simmons et al., 2011). Although there maynot be anything more practical than a goodtheory (Lewin, 1952), the use of soundmethods as well as the development of newmethods and statistical techniques could beeven more important (Greenwald, 2012),particularly if we are interested in assessingthe robustness of our results and improv-ing the accuracy and trustworthiness of ourliterature. To the extent that sensitivity anal-yses make an article excessively long, thekey findings of such analyses could be men-tioned in the article and the details placed inthe supplemental materials, just as they arein other scientific disciplines (Evangelou,Trikalinos, & Ioannidis, 2005).

The aforementioned recommendationspertain largely to primary studies. How-ever, analogous recommendations apply toreview articles, particularly meta-analytic(systematic) reviews (Kepes et al., 2013).Most meta-analytic reviews fail to assessthe robustness of the obtained results,whether this assessment involves outliers,publication bias, or related phenomena(Aguinis, Dalton, Bosco, Pierce, & Dal-ton 2011; Aytug, Rothstein, Zhou, & Kern,

2012; Banks, Kepes, & McDaniel, 2012;Kepes, Banks, McDaniel, & Whetzel, 2012).In concurrence with previous calls formore rigorous assessments of meta-analyticresults (Banks, Kepes, & McDaniel, 2012;Ferguson & Heene, 2012; Kepes, Banks,McDaniel, & Whetzel, 2012; Kepes, Banks,& Oh, in press; Kepes et al., 2013;McDaniel, McKay, & Rothstein, 2006)and the Meta-analytic Reporting Standards(MARS) of the American PsychologicalAssociation (2008, 2010), we recommendthat journals require such analyses in allmeta-analytic reviews.

Implementation Considerations

We note that there are many challengesfacing those seeking to adopt the recom-mendations presented in this article. Somewill argue that we have overstated theproblems and their opinions will need to beaddressed. Others will agree that changewould benefit our discipline but havelow expectations that change will actuallyhappen. Many of our recommendations,when implemented, are likely to increasethe labor demands during the editorialreview process. Change is hard but is sorelyneeded.

We offer that better practices canbest be obtained through the coordinatedaction of multiple journals.13 For example,change occurred in medical research whena committee of journal editors requiredregistration before a clinical trial study wasconducted as a condition for publicationin the participating journals (Laine et al.,2007). As a result, studies began tobe routinely registered. In addition, mostof the commercial publishers of ourjournals (i.e., publishers not owned byscientific organizations) have experiencein maintaining supplemental information

13. We acknowledge http://editorethics.uncc.edu asa step in this direction. We also refer readers tohttp://psychdisclosure.org/, a website which, on avolunteer basis, provides important methodolog-ical details about recently published articles thatare not included in the journal publication.


because they also publish journals infields in which supplemental informationis a common practice. We note thatAPA journals can accept supplementalinformation, although the practice does notappear to be widely used by the Journal ofApplied Psychology or other I–O relatedjournals. Although we argue that changesare best initiated at the journal level, we areencouraged by the survey findings of Fuchs,Jenny, and Fiedler (2012), which indicatethat research psychologists are generallysupportive of changes in the scientificprocess. We do not seek to underestimatethe challenges faced, but we note that somedisciplines have made substantial progressin rectifying some of the structural problemsin the scientific process.

Although we have made a case for theimprovement of science in I–O psychol-ogy through the implementation of ourrecommendations, there are clear costs.The work load of journal editors and theeditorial boards would increase due to theneed for new policies and procedures. Forinstance, the article review process mayentail more labor. In addition, there isalso substantial labor involved in startingand maintaining a registry. Moreover,there are costs involved in implementingchanges to the way we conduct research.If we stop HARKing and manipulating ourdata and research designs to enhance thelikelihood of publication, it may becomemore difficult to publish. We do not wish tounderestimate any of these costs. However,the question becomes whether we wish tohave a trustworthy science that entails someadditional costs, or should we continuewith our problematic scientific practices?

Conclusion

We offer that a review of the trust-worthiness of the scientific literature inI–O psychology is warranted. Concernwas expressed over the fact that mosthypotheses in our journals are confirmed.We offered that our journals are likely tobe publishing an unrepresentative sampleof completed research in certain literature

areas and that this is damaging our scien-tific knowledge in that population effectsare misestimated, typically overestimated.We reviewed structural problems in thepublication process as well as in the con-duct of research that could be promotingdistortions of our scientific knowledge.We provided recommendations to makethe I–O literature more trustworthy andaccurate.

Just as we doubt that the researchers whopublish in I–O journals are omniscient, weoffer that we do not have all the answersto resolving issues in the scientific processof I–O psychology. However, we arequite sure that researcher practices andour editorial review process need to bealtered in order to build better science andenhance the trustworthiness of our litera-ture. Although we respect our researchercolleagues, reviewers, and journal editors,and recognize that they seek to promotegood science, we offer that some ofwell-intentioned policies, practices, andbehaviors that are common during theresearch and editorial review processescould be damaging to the accuracy of ourscientific knowledge.

References

Academy of Management Journal. (2012). Infor-mation for Contributors. Retrieved fromhttp://aom.org/Publications/AMJ/Information-for-Contributors.aspx

Aguinis, H., Dalton, D. R., Bosco, F. A., Pierce, C. A.,& Dalton, C. M. (2011). Meta-analytic choices andjudgment calls: Implications for theory building andtesting, obtained effect sizes, and scholarly impact.Journal of Management, 37, 5–38. doi: 10.1177/0149206310377113

Allison, P. D. (1978). Measures of inequality.American Sociological Review, 43, 865–880. doi:10.2307/2094626

American Psychological Association. (2002). Ethicalprinciples of psychologists and code of conduct.American Psychologist, 57, 1060–1073. doi: 10.1037/0003-066x.57.12.1060

American Psychological Association. (2008). Report-ing standards for research in psychology: Why dowe need them? What might they be? AmericanPsychologist, 63, 839–851. doi: 810.1037/0003-1066X.1063.1039.1839

American Psychological Association. (2010). Publi-cation manual of the American PsychologicalAssociation (6th ed.). Washington, DC: AmericanPsychological Association.


Aytug, Z. G., Rothstein, H. R., Zhou, W., & Kern, M.C. (2012). Revealed or concealed? Transparencyof procedures, decisions, and judgment calls inmeta-analyses. Organizational Research Methods,15, 103–133. doi: 10.1177/1094428111403495

Baggerly, K. (2010). Disclose all data in publications.Nature, 467, 401. doi: 10.1038/467401b

Banks, G. C., Kepes, S., & Banks, K. P. (2012).Publication bias: The antagonist of meta-analyticreviews and effective policy making. EducationalEvaluation and Policy Analysis, 34, 259–277. doi:10.3102/0162373712446144

Banks, G. C., Kepes, S., & McDaniel, M. A.(2012). Publication bias: A call for improvedmeta-analytic practice in the organizational sci-ences. International Journal of Selection andAssessment, 20, 182–196. doi: 10.1111/j.1468-2389.2012.00591.x

Banks, G. C., & McDaniel, M. A. (2011). The kryptoniteof evidence-based I–O psychology. Industrial andOrganizational Psychology: Perspectives on Sci-ence and Practice, 4, 40–44. doi: 10.1111/j.1754-9434.2010.01292.x

Bargh, J. A., Chen, M., & Burrows, L. (1996).Automaticity of social behavior: Direct effects oftrait construct and stereotype activation on action.Journal of Personality and Social Psychology, 71,230–244. doi: 10.1037/0022-3514.71.2.230

Bedeian, A. G., Taylor, S. G., & Miller, A. N. (2010).Management science on the credibility bubble:Cardinal sins and various misdemeanors. Academyof Management Learning & Education, 9, 715–725.doi: 10.5465/amle.2010.56659889

Begley, C. G., & Ellis, L. M. (2012). Drug development:Raise standards for preclinical cancer research.Nature, 483, 531–533. doi: 10.1038/483531a

Bem, D. J. (2011). Feeling the future: Experimentalevidence for anomalous retroactive influences oncognition and affect. Journal of Personality andSocial Psychology, 100, 407–425. doi: 10.1037/a0021524

Berenson, A. (2005, May 31). Despite vow, drugmakers still withhold data. New York Times,p. A1. Retrieved from http://www.nytimes.com/2005/05/31/business/31trials.html.

Berlin, J. A., & Ghersi, D. (2005). Preventingpublication bias: Registries and prospective meta-analysis. In H. R. Rothstein, A. J. Sutton, & M.Borenstein (Eds.), Publication bias in meta analysis:Prevention, assessment, and adjustments. WestSussex, England: Wiley.

Braxton, J. M. (1999). Perspectives on scholarlymisconduct in the sciences. Columbus, OH: OhioState University Press.

Carey, B. (2011, November 3). Fraud case seen as a redflag for psychology research. New York Times, p.3. Retrieved from http://www.nytimes.com/2011/11/03/health/research/noted-dutch-psychologist-stapel-accused-of-research-fraud.html

Chalmers, T. C., Levin, H., Sacks, H. S., Reitman,D., Berrier, J., & Nagalingam, R. (1987). Meta-analysis of clinical trials as a scientific discipline.I: Control of bias and comparison with large co-operative trials. Statistics in Medicine, 6, 315–325.doi: 10.1002/sim.4780060320

Crane, D. (1967). The gatekeepers of science: Somefactors affecting the selection of articles for

scientific journals. The American Sociologist, 2,195–201. doi: 10.2307/27701277

Crocker, J., & Cooper, M. L. (2011). Addressing scien-tific fraud. Science, 334, 1182. doi: 10.1126/sci-ence.1216775

Dalton, D. R., Aguinis, H., Dalton, C. M., Bosco,F. A., & Pierce, C. A. (2012). Revisiting the filedrawer problem in a meta-analysis: An assess-ment of published and nonpublished correlationmatrices. Personnel Psychology, 65, 221–249. doi:10.1111/j.1744-6570.2012.01243.x

Darwin, C. (1981/1871). The descent of man andselection in relation to sex. London, England:Princeton University Press.

Dickersin, K. (1990). The existence of publication biasand risk factors for its occurrence. Journal of theAmerican Medical Association, 263, 1385–1389.doi: 10.1001/jama.1990.03440100097014

Dickersin, K. (2005). Publication bias: Recognizing theproblem, understandings its origins and scope, andpreventing harm. In H. R. Rothstein, A. J. Sutton,& M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments(pp. 11–34). West Sussex, England: Wiley.

Dickersin, K., Manheimer, E., Wieland, S., Robinson,K. A., Lefebvre, C., McDonald, S., & Group, C. D.(2002). Development of the Cochrane Collabora-tion’s central register of controlled clinical trials.Evaluation & the Health Professions, 25, 38–64.doi: 10.1177/016327870202500104

Doyen, S., Klein, O., Pichon, C.-L., & Cleeremans, A.(2011). Behavioral priming: It’s all in the mind, butwhose mind? PLoS ONE, 7, e29081. doi: 10.1371/journal.pone.0029081

Easterbrook, P. J., Gopalan, R., Berlin, J. A., &Matthews, D. R. (1991). Publication bias in clinicalresearch. The Lancet, 337, 867–872. doi: 10.1016/0140-6736(91)90201-y

Evangelou, E., Trikalinos, T. A., & Ioannidis, J. P.(2005). Unavailability of online supplementaryscientific information from articles publishedin major journals. The FASEB Journal, 19,1943–1944. doi: 10.1096/fj.05-4784lsf

Fanelli, D. (2010a). Do pressures to publish increasescientists’ bias? An empirical support from US statesdata. PLoS ONE, 5, e10271. doi: 10.1371/jour-nal.pone.0010271

Fanelli, D. (2010b). ‘‘Positive’’ results increase downthe hierarchy of the sciences. PLoS ONE, 5,e10068. doi: 10.1371/journal.pone.0010068

Fanelli, D. (2012). Negative results are disappear-ing from most disciplines and countries. Sciento-metrics, 90, 891–904. doi: 10.1007/s11192-011-0494-7

Ferguson, C. J., & Brannick, M. T. (2012). Publicationbias in psychological science: Prevalence, methodsfor identifying and controlling, and implications forthe use of meta-analyses. Psychological Methods,17, 120–128. doi: 10.1037/a0024445

Ferguson, C. J., & Heene, M. (2012). A vastgraveyard of undead theories: Publication biasand psychological science’s aversion to thenull. Perspectives on Psychological Science, 7,555–561. doi: 10.1177/1745691612459059

Francis, G. (2012). Too good to be true: Publicationbias in two prominent studies from experimentalpsychology. Psychonomic Bulletin & Review, 19,151–156. doi: 10.3758/s13423-012-0227-9


Fuchs, H. M., Jenny, M., & Fiedler, S. (2012).Psychologists are open to change, yet wary ofrules. Perspectives on Psychological Science, 7,639–642. doi: 10.1177/1745691612459521

Gallup Organization. (2008). Observing andreporting suspected misconduct in biomedicalresearch. Retrieved from http://ori.hhs.gov/sites/default/files/gallup_finalreport.pdf

Giner-Sorolla, R. (2012). Science or art? How aestheticstandards grease the way through the publicationbottleneck but undermine science. Perspectiveson Psychological Science, 7, 562–571. doi:10.1177/1745691612457576

Goldacre, B. (2012). Bad pharma: How drug compa-nies mislead doctors and harm patients. London,England: Fourth Estate.

Goldstein, D. (2010). On fact and fraud: Cautionarytales from the front lines of science. Princeton, NJ:Princeton University Press.

Gomez-Mejia, L. R., & Balkin, D. B. (1992).Determinants of faculty pay: An agency theoryperspective. Academy of Management Journal, 35,921–955. doi: 10.2307/256535

Greenwald, A. G. (1975). Consequences of prejudiceagainst the null hypothesis. Psychological Bulletin,82, 1–20. doi: 10.1037/h0076157

Greenwald, A. G. (2012). There is nothing sotheoretical as a good method. Perspectives onPsychological Science, 7, 99–108. doi: 10.1177/1745691611434210

Hambrick, D. C. (2007). The field of management’sdevotion to theory: Too much of a good thing?Academy of Management Journal, 50, 1348–1352.doi: 10.5465/amj.2007.28166119

Hanson, B., Sugden, A., & Alberts, B. (2011). Makingdata maximally available. Science, 331, 649. doi:10.1126/science.1203354

Harrison, D. A., & Klein, K. J. (2007). What’sthe difference? Diversity constructs as separation,variety, or disparity in organizations. The Academyof Management Review, 32, 1199–1228. doi:10.2307/20159363

Ioannidis, J. P. A. (2005). Why most published researchfindings are false. PLoS Medicine, 2, e124. doi:10.1371/journal.pmed.0020124

Ioannidis, J. P. A. (2006). Concentration of the most-cited papers in the scientific literature: Analysisof journal ecosystems. PLoS ONE, 1, e5. doi:10.1371/journal.pone.0000005

Ioannidis, J. P. A. (2012). Why science is notnecessarily self-correcting. Perspectives on Psy-chological Science, 7, 645–654. doi: 10.1177/1745691612464056

Ioannidis, J. P. A., Allison, D. B., Ball, C. A., Coulibaly,I., Cui, X., Culhane, A. C., . . . van Noort,V. (2009). Repeatability of published microarraygene expression analyses. Nature Genetics, 41,149–155. doi: 10.1038/ng.295

Jasny, B. R., Chin, G., Chong, L., & Vignieri, S. (2011).Again, and again, and again . . . . Science, 334,1225. doi: 10.1126/science.334.6060.1225

Jefferson, T. (1998). Redundant publication in biomed-ical sciences: Scientific misconduct or necessity?Science and Engineering Ethics, 4, 135–140. doi:10.1007/s11948-998-0043-9

Kepes, S., Banks, G. C., McDaniel, M. A., & Whetzel,D. L. (2012). Publication bias in the organizational

sciences. Organizational Research Methods, 15,624–662. doi: 10.1177/1094428112452760

Kepes, S., Banks, G., & Oh, I.-S. (in press).Avoiding bias in publication bias research: Thevalue of ‘‘null’’ findings. Journal of Business andPsychology. doi: 10.1007/s10869-012-9279-0

Kepes, S., McDaniel, M. A., Brannick, M. T., &Banks, G. C. (2013). Meta-analytic reviews inthe organizational sciences: Two meta-analyticschools on the way to MARS (the Meta-analyticReporting Standards). Journal of Business andPsychology, 28, 123–143. doi: 10.1007/s10869-013-9300-2

Kerr, S. (1975). On the folly of rewarding A, whilehoping for B. Academy of Management Journal,18, 769–783. doi: 10.2307/255378

Kerr, N. L. (1998). HARKing: Hypothesizingafter the results are known. Personality andSocial Psychology Review, 2, 196–217. doi:10.1207/s15327957pspr0203_4

Koole, S. L., & Lakens, D. (2012). Rewarding repli-cations: A sure and simple way to improvepsychological science. Perspectives on Psycho-logical Science, 7, 608–614. doi: 10.1177/1745691612462586

Kozlowski, S. W. J. (2009). Editorial. Journal of AppliedPsychology, 94, 1–4. doi: 10.1037/a0014990

Krimsky, S. (2006). Publication bias, data ownership,and the funding effect in science: Threats to theintegrity of biomedical research. In W. Wagner, &R. Steinzor (Eds.), Rescuing science from politics:Regulation and the distortion of scientific research(pp. 61–85). New York, NY: Cambridge UniversityPress.

LaFollette, M. C. (1992). Stealing into print: Fraud, pla-giarism, and misconduct in scientific publishing.Los Angeles, CA: University of California Press.

Laine, C., Horton, R., DeAngelis, C. D., Drazen, J.M., Frizelle, F. A., Godlee, F., . . . Verheugt, F.W. A. (2007). Clinical trial registration: Lookingback and moving ahead. New England Journalof Medicine, 356, 2734–2736. doi: 10.1056/NEJMe078110

LeBel, E. P., & Peters, K. R. (2011). Fearing the futureof empirical psychology: Bem’s (2011) evidenceof psi as a case study of deficiencies in modalresearch practice. Review of General Psychology,15, 371–379. doi: 10.1037/a0025172

Lehrer, J. (2010). The truth wears off. New Yorker, 86,52–57.

Leung, K. (2011). Presenting post hoc hypotheses as apriori: Ethical and theoretical issues. Managementand Organization Review, 7, 471–479. doi: 10.1111/j.1740-8784.2011.00222.x

Lewin, K. (1952). Field theory in social science:Selected theoretical papers by Kurt Lewin. London,England: Tavistock.

Liberati, A. (1992). Publication bias and the edi-torial process. Journal of the American Med-ical Association, 267, 2891. doi: 10.1001/jama.1992.03480210049017

Makel, M. C., Plucker, J. A., & Hegarty, B.(2012). Replications in psychology research: Howoften do they really occur? Perspectives onPsychological Science, 7, 537–542. doi: 10.1177/1745691612460688

McDaniel, M. A., McKay, P., & Rothstein, H. R.(2006). Publication bias and racial effects on job


performance: The elephant in the room. Paperpresented at the annual meeting of the Society forIndustrial and Organizational Psychology, Dallas,TX.

McDaniel, M. A., Rothstein, H. R., & Whetzel, D. L.(2006). Publication bias: A case study of four testvendors. Personnel Psychology, 59, 927–953. doi:10.1111/j.1744-6570.2006.00059.x

McDaniel, M. A., & Whetzel, D. L. (2005). Situationaljudgment test research: Informing the debateon practical intelligence theory. Intelligence, 33,515–525. doi: 10.1016/j.intell.2005.02.001

Merton, R. K. (1942). Science and technology in ademocratic order. Journal of Legal and PoliticalSociology, 1, 115–126.

Merton, R. K. (1973). The sociology of science:Theoretical and empirical investigations. Chicago,IL: The University of Chicago Press.

Murphy, J. R. (2004). Statistical errors in immunologicresearch. The Journal of allergy and clinicalimmunology, 114, 1259–1263.

Murtaugh, P. A. (2002). Journal quality, effectsize, and publication bias in meta-analysis.Ecology, 83, 1162–1166. doi: 10.1890/0012-9658(2002)083[1162:jqesap]2.0.co;2

Neuliep, J. W., & Crandall, R. (1990). Editorial biasagainst replication research. Journal of SocialBehavior & Personality, 5, 85–90.

Neuliep, J. W., & Crandall, R. (1993). Reviewerbias against replication research. Journal of SocialBehavior & Personality, 8, 21–29.

Nickerson, R. S. (1998). Confirmation bias: A ubiqui-tous phenomenon in many guises. Review of Gen-eral Psychology, 2, 175–220. doi: 10.1037/1089-2680.2.2.175

Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientificutopia: II. Restructuring incentives and practices topromote truth over publishability. Perspectives onPsychological Science, 7, 615–631. doi: 10.1177/1745691612459058

Open Science Collaboration. (2012). An open, large-scale, collaborative effort to estimate the repro-ducibility of psychological science. Perspectives onPsychological Science, 7, 657–660. doi: 10.1177/1745691612462588

Orlitzky, M. (2012). How can significancetests be deinstitutionalized? OrganizationalResearch Methods, 15, 199–228. doi: 10.1177/1094428111428356

Pace, V. L., & Brannick, M. T. (2010). How similarare personality scales of the ‘‘same’’ construct?A meta-analytic investigation. Personality andIndividual Differences, 49, 669–676. doi: 10.1016/j.paid.2010.06.014

Park, R. L. (2001). Voodoo science: The road fromfoolishness to fraud. Oxford, England: OxfordUniversity Press.

Pashler, H., & Harris, C. R. (2012). Is the replicabilitycrisis overblown? Three arguments examined. Per-spectives on Psychological Science, 7, 531–536.doi: 10.1177/1745691612463401

Popper, K. R. (1959). The logic of scientific discovery.Oxford, England: Basic Books.

Porter, T. M. (1992). Quantification and the accountingideal in science. Social Studies of Science, 22,633–652. doi: 10.1177/030631292022004004

Rauscher, F. H., Shaw, G. L., & Ky, C. N. (1993).Music and spatial task performance. Nature, 365,611. doi: 10.1038/365611a0

Reich, E. S. (2010). Plastic fantastic: How the biggestfraud in physics shook the scientific world. NewYork, NY: Palgrave Macmillan.

Renkewitz, F., Fuchs, H. M., & Fiedler, S. (2011).Is there evidence of publication biases in JDMresearch? Judgment and Decision Making, 6,870–881.

Ritchie, S. J., Wiseman, R., & French, C. C. (2012). Fail-ing the future: Three unsuccessful attempts to repli-cate Bem’s ’retroactive facilitation of recall’ effect.PLoS ONE, 7. doi: 10.1371/journal.pone.0033423

Roberson, Q. M., Sturman, M. C., & Simons,T. L. (2007). Does the measure of dispersionmatter in multilevel research? A comparison ofthe relative performance of dispersion indexes.Organizational Research Methods, 10, 564–588.doi: 10.1177/1094428106294746

Rosenthal, R. (1979). The file drawer problem andtolerance for null results. Psychological Bulletin,86, 638–641. doi: 10.1037/0033-2909.86.3.638

Rothstein, H. (2012). Accessing relevant literature. InH. M. Cooper (Ed.), APA handbook of researchmethods in psychology: Vol. 1. Foundations, plan-ning, measures, and psychometrics (pp. 133–144).Washington, DC: American Psychological Associ-ation.

Rothstein, H. R., Sutton, A. J., & Borenstein, M.(2005a). Publication bias in meta-analyses. In H.R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.),Publication bias in meta analysis: Prevention,assessment, and adjustments (pp. 1–7). WestSussex, England: Wiley.

Rothstein, H. R., Sutton, A. J., & Borenstein,M. (2005b). Publication bias in meta-analysis:Prevention, assessment, and adjustments. WestSussex, England: Wiley.

Rouder, J. N., & Morey, R. D. (2011). A Bayes factormeta-analysis of Bem’s ESP claim. PsychonomicBulletin & Review, 18, 682–689. doi: 10.3758/s13423-011-0088-7

Rupp, D. E. (2011). Ethical issues faced by edi-tors and reviewers. Management and Organiza-tion Review, 7, 481–493. doi: 10.1111/j.1740-8784.2011.00227.x

Sarawitz, D. (2012). Beware the creeping cracks ofbias. Nature, 485, 149. doi: 10.1038/485149a

Saul, S. (2008, October 8). Experts concludePfizer manipulated studies. New York Times, p.4. Retrieved from http://www.nytimes.com/2008/10/08/health/research/08drug.html

Schmidt, S. (2009). Shall we really do it again? Thepowerful concept of replication is neglected in thesocial sciences. Review of General Psychology, 13,90–100. doi: 10.1037/a0015108

Schminke, M. (2009). Editor’s comments: The bet-ter angels of our time-Ethics and integrity inthe publishing process. The Academy of Man-agement Review, 34, 586–591. doi: 10.5465/amr.2009.44882922

Schminke, M., & Ambrose, M. L. (2011). Ethicsand integrity in the publishing process: Myths,facts, and a roadmap. Management and Organi-zation Review, 7, 397–406. doi: 10.1111/j.1740-8784.2011.00248.x


Schmitt, N. (1989). Editorial. Journal of AppliedPsychology, 74, 843–845. doi: 10.1037/h0092216

Schooler, J. W., & Engstler-Schooler, T. Y. (1990).Verbal overshadowing of visual memories: Somethings are better left unsaid. Cognitive Psychology,22, 36–71. doi: 10.1016/0010-0285(90)90003-m

Schwab, D. P. (2005). Research methods for organiza-tional studies (2nd ed.). Mahwah, NJ: Erlbaum.

Shields, P. G. (2000). Publication bias is a scientificproblem with adverse ethical outcomes: The casefor a section for null results. Cancer EpidemiologyBiomarkers & Prevention, 9, 771–772.

Shields, P. G., Sellers, T. A., & Rebbeck, T. R.(2009). Null results in brief: Meeting a need inchanging times. Cancer Epidemiology Biomarkers& Prevention, 18, 2347. doi: 10.1158/1055-9965.epi-09-0684

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011).False-positive psychology: Undisclosed flexibilityin data collection and analysis allows presentinganything as significant. Psychological Science, 22,1359–1366. doi: 10.1177/0956797611417632

Sterling, T. D. (1959). Publication decisions and theirpossible effects on inferences drawn from testsof significance—or vice versa. Journal of theAmerican Statistical Association, 54, 30–34. doi:10.2307/2282137

Sterling, T. D., & Rosenbaum, W. L. (1995). Publicationdecisions revisited: The effect of the outcome ofstatistical tests on the decision to publish and viceversa. American Statistician, 49, 108–112. doi:10.1080/00031305.1995.10476125

Sternberg, R. J., Forsythe, G. B., Hedlund, J., Horvath, J.A., Wagner, R. K., Williams, W. M., . . . Grigorenko,E. L. (2000). Practical intelligence in everyday life.New York, NY: Cambridge University Press.

Strasak, A. M., Zaman, Q., Marinell, G., Pfeiffer, K. P.,& Ulmer, H. (2007). The use of statistics in medicalresearch. The American Statistician, 61, 47–55.doi: 10.1198/000313007x170242

Stroebe, W., Postmes, T., & Spears, R. (2012). Scientificmisconduct and the myth of self-correction inscience. Perspectives on Psychological Science,7, 670–688. doi: 10.1177/1745691612460687

Sutton, A. J. (2005). Evidence concerning the con-sequences of publication and related biases. InH. R. Rothstein, A. J. Sutton, & M. Borenstein(Eds.), Publication bias in meta-analysis: Preven-tion, assessment, and adjustments (pp. 175–192).West Sussex, England: Wiley.

Sutton, A. J. (2009). Publication bias. In H. Cooper,L. V. Hedges, & J. C. Valentine (Eds.), Thehandbook of research synthesis and meta-analysis(2nd ed., pp. 435–452). New York, NY: RussellSage Foundation.

Sutton, R. I., & Staw, B. M. (1995). What theory is not.Administrative Science Quarterly, 40, 371–384.doi: 10.2307/2393788

Trikalinos, T. A., & Ioannidis, J. P. A. (2005). Assessingthe evolution of effect sizes over time. In H. R.Rothstein, A. J. Sutton, & M. Borenstein (Eds.),Publication bias in meta-analysis: Prevention,assessment and adjustments (pp. 241–259). WestSussex, England: Wiley.

Turner, E. H. (2004). A taxpayer-funded clinical trialsregistry and results database. PLoS Medicine, 1,e60. doi: 10.1371/journal.pmed.0010060

Uchino, B. N., Thoman, D., & Byerly, S. (2010).Inference patterns in theoretical social psychology:Looking back as we move forward. Social andPersonality Psychology Compass, 4, 417–427. doi:10.1111/j.1751-9004.2010.00272.x

Van Dongen, S., & Gangestad, S. W. (2011).Human fluctuating asymmetry in relation tohealth and quality: A meta-analysis. Evolu-tion and Human Behavior, 32, 380–398. doi:10.1016/j.evolhumbehav.2011.03.002

Vogel, G. (2011). Psychologist accused of fraudon ‘‘astonishing scale.’’ Science, 334, 579. doi:10.1126/science.334.6056.579

Wagenmakers, E.-J., Wetzels, R., Borsboom, D., vander Maas, H. L. J., & Kievit, R. A. (2012). An agendafor purely confirmatory research. Perspectives onPsychological Science, 7, 632–638. doi: 10.1177/1745691612463078

Whalen, J., Barrett, D., & Loftus, P. (2012, July 3).Glaxo sets guilty plea, $3 billion settlement.Wall Street Journal, B1. Retrieved from http://online.wsj.com/article/SB10001424052702304299704577502642401041730.html

Wicherts, J. M., Bakker, M., & Molenaar, D. (2011).Willingness to share research data is related to thestrength of the evidence and the quality of reportingof statistical results. PLoS ONE, 6, e26828. doi:10.1371/journal.pone.0026828

Wicherts, J. M., Borsboom, D., Kats, J., & Mole-naar, D. (2006). The poor availability of psy-chological research data for reanalysis. AmericanPsychologist, 61, 726–728. doi: 10.1037/0003-066x.61.7.726

Witten, D. M., & Tibshirani, R. (2012). Scientificresearch in the age of omics: The good, the bad, andthe sloppy. Journal of the American Medical Infor-matics Association, 1–3. doi: 10.1136/amiajnl-2012-000972

Wolins, L. (1962). Responsibility for raw data. Amer-ican Psychologist, 17, 657–658. doi: 10.1037/h0038819

Yong, E. (2012a). Replication studies: Bad copy.Nature, 485, 298–300. doi: 10.1038/485298a

Yong, E. (2012b). Uncertainty shrouds psycholo-gist’s resignation. Nature. doi: 10.1038/nature.2012.10968

Documents

How Trustworthy Is the Scientiï¬c Literature in Industrial and