COMMENTARY

A “No Prescriptives, Please” Proposal Postscript:When Desperate Times Require Desperate Measures

Daniel H. Robinson & Joel R. Levin

Published online: 15 August 2013# Springer Science+Business Media New York 2013

Abstract We appreciate the thoughtful reactions of our colleagues to the “no prescriptives,please” proposal of Robinson et al. (2013), as well as the opportunity to respond to them. Forthe most part, our colleagues seem to agree that a problem exists in terms of unwarrantedrecommendations for practice appearing too often in educational research journals. At thesame time, we realize that because our proposed policy is rather Draconian—in that it seeksto eliminate all recommendations for practice in all primary research journals—everyresponder mentioned that it was too extreme. This was intentional on our part. If, as Harris(2013) suggested, increased awareness and scrutiny become topics for further discussionamong APA editors, then our modest mission will have been accomplished. In this rejoinder,we attempt to restate and clarify our proposal, along with its entailments, followed by briefcomments on each responder's response.

Keywords Prescriptives . Recommendations . Causality . Policy

For primary research journals in the fields of psychology and education, over the years, thefollowing editorial policy recommendations have been made:

& Traditional statistical hypothesis testing should be banned and replaced by effect-sizeestimates and confidence intervals (e.g., Schmidt 1996).

& The nature of a study's participants must be described more completely with respect tosuch aspects as their demographic, developmental, cognitive, and emotional and behav-ioral characteristics (e.g., Harris 2003).

& Any study's findings that are reported as being “significant” must be preceded by themodifier “statistically” (e.g., Thompson 1996)

& Structured abstracts are more compact and precise than are conventional abstracts and sothe former should replace the latter (e.g., Mosteller et al. 2004).

Educ Psychol Rev (2013) 25:353–359DOI 10.1007/s10648-013-9238-y

D. H. Robinson (*)School of Education, Colorado State University, 1588 Campus Delivery, Fort Collins,CO 80523-1588, USAe-mail: dan.robinson@colostate.edu

J. R. LevinUniversity of Arizona, Tucson, Arizona, USA

& Individual experiments or studies should not be published unless they also include sometype of replication component (e.g., Thompson 1996).

& The high incidence of “underpowered” studies appearing in our journals dictates thatpower estimates must be reported and deemed adequate for the statistical tests that areconducted (e.g., Schmidt 1996).

& Because of documented “prejudice against the null hypothesis,” authors should submittheir manuscripts for review with the Results and Discussion sections removed (e.g.,Walster and Cleary 1970).

& Primary research journals are not an appropriate outlet for authors to provide “prescriptive”statements (i.e., recommendations for practice) to others (Robinson et al. 2013).

Certain of these recommendations have been adopted by various journals and others havenot been. Many of them may sound unduly prescriptive and extreme. Consider, for example,the banning of null hypothesis significance testing in APA journals, an extreme proposal thatwas recommended by highly qualified and well-meaning psychological researchers andmethodologists (e.g., Carver 1993; Schmidt 1996). The proposal was considered and givenserious attention by APA editors and staffers, as well as APA's Publications and CommunicationsBoard and special committees that were assembled to write the “statistical guidelines” for the fifth(2001) and sixth (2010) editions of the APA Publication Manual. In the end, hypothesis testingsurvived scrutiny and continues to be endorsed by the manual as a critical inferential statisticaltool (as it is in other academic disciplines), with researchers' provision of effect sizes andconfidence intervals additionally encouraged.1

We appreciate the thoughtful reactions of our colleagues to the “no prescriptives, please”proposal of Robinson et al. (2013), as well as the opportunity to respond to them. For the mostpart, our colleagues seem to agree that a problem exists in terms of unwarranted recommenda-tions for practice appearing too often in educational research journals. At the same time, werealize that because our proposed policy is rather Draconian—in that it seeks to eliminate allrecommendations for practice in all primary research journals—every responder mentioned thatit was too extreme. This was intentional on our part. If the asking price for a used car is $15,000and you want to pay only $12,000, your counteroffer conversation starts lower than $12,000. Inour case, the price that we are willing to pay is heightened awareness of a problem that will leadto increased editorial scrutiny. If, as Harris (2013) suggested, increased awareness and scrutinybecome topics for further discussion among APA editors, then our modest mission will havebeen accomplished. In this rejoinder, we attempt to restate and clarify our proposal, along withits entailments, followed by brief comments on each responder's response.

An Unfortunate Misrepresentation/Mischaracterization of Our Proposal

Several authors seemed to confuse what we were really proposing with something else. Forexample, Renkl (2013) mentions both practical recommendations and implications. Theseare very different. Our proposal dealt only with the former. A recommendation would besomething like “We recommend that teachers employ physical manipulatives when teachinggeometry to 4th grade students.” A potential implication might be something like “The

1 Although the incidence of effect size reporting has increased dramatically in education and psychologyjournals over the past decade, it is evident that such reporting generally occurs in an unthinking, indiscriminatefashion (Peng et al. 2013). Insofar as no randomized intervention experiment was conducted at the time andthroughout the decade, it is not possible to determine exactly how much of the increase in authors' provision ofeffect size and confidence interval information is attributable to the manual's “encouragement” of them per se.

354 Educ Psychol Rev (2013) 25:353–359

results of this study might eventually inform the way in which math is taught to 4th gradestudents.” Note that the former is very direct and prescriptive, whereas the latter is more“vanilla” and speculative.

The motto of the Royal Statistical Society is Aliis exterendum (let others thrash it out) anddates back to 1834. Thus, our “no prescriptives, please” recommendation may be a bitplagiaristic. All we are suggesting is that primary research articles should stick to reportingthe study's findings. Beyond that, let “others”—many of whom were specifically identifiedby Robinson et al. (2013)—thrash out the possibilities of what to do with the findings.

Bans on smoking in indoor public places have increased in popularity in the USA overthe past 20 years. It is now against the law to smoke in some restaurants that previouslypermitted it. Does this prevent smokers from smoking? Certainly not. Smokers are simplyasked to step outside where the secondary smoke is not as easily ingested by nonsmokers.Similarly, our proposed policy would not prevent educational psychologists (who Alexander,2013, suggests are the perfect thrashers) from thrashing out policy recommendations fromresearch findings. It would only restrict the locations where such thrashing is permitted.

Would the Proposal Eliminate Potentially Useful Practical Information?

Any intimation that our proposed policy would have the effect of preventing educationalpsychologists from influencing practice (e.g., Alexander, 2013; Renkl, 2013; Wainer andClauser, 2013) seems to be more of an over-reaction than a reaction to the policy itself. Again,let us attempt to explain what the policy would do by taking the previous example of arecommendation for practice concerning manipulatives for teaching mathematics to fourthgraders. If we prevent authors from including the recommendation in a research article, thenwhat can they include that would inform practitioners? If they stick to presenting the findings oftheir study, it might be something like this: “Fourth-grade students who were taught geometryusing physical manipulatives correctly solved more problems on a test than did those who weretaught without the manipulatives.” This sentence could be the last sentence of the abstract andof the entire article. If practitioners read it, what is the chance that they would come awaywithout the intended message? Do we really also need to include a recommendation?

In the example that Harris (2013) provides about noise-reducing headphones, we feel thatthe authors should stick to stating the findings of the study—namely, that the headphonesseemed to benefit the learning-disabled students in grades 3–5, whereas they actuallyharmed the performance of the nondisabled students. This is enough. There is no need togo on and recommend that teachers consider using the headphones for some students. Is thatpoint not already obvious? Similarly, if a study finds that shark cartilage tablets benefittedphysically active 45-year-old men in terms of their recovery time, is it necessary torecommend shark cartilage? We prefer that researchers stick to reporting their findings andlet the reader decide. Aliis exterendum—let others thrash it out.

Responses to Selected Author Comments

Renkl (2013) suggested, “Even wrong claims might often have important functions foradvancing scientific knowledge.” That may be true in some cases, but in others, wrongclaims can be catastrophic. Consider the number of children who have died because theirparents chose not to vaccinate them for fear of a potential cause of autism. Renkl also noted,“I think it is fully legitimate when publishing one of the later studies in this branch of

Educ Psychol Rev (2013) 25:353–359 355

research to claim in the discussion that example-enriched tutors are probably the better option thantraditional tutors.” Again, is this claim really necessary, or even justified, based on the results of asingle empirical study? Would not a reader be better served by the author providing a researchsynthesis in the literature review at the beginning of the research article, with the study's findingsnoted as either supporting or not supporting the synthesis conclusions?

We resonate to Harris' (2013) recommendation that professional standards and researchtraining in our field of educational psychology should be improved. Harris also indicatedthat we already have existing standards in place that should prevent bad author and badeditor behavior. Standards may well exist, but we would argue that they are not workinginsofar as authors and editors seem to be ignoring them. Studies, such as the recent one byReinhart et al. (2013), present actual data documenting that journals in our field continue topublish articles that make such inappropriate claims. As far as standards produced by theAmerican Educational Research Association (AERA) are concerned, one must be remindedthat this is the same organization that publishes the Educational Researcher, where an articleonce appeared that promoted the use of poetry within the educational “research” pantheon(Cahnmann 2003)—a situation reminiscent of the fox guarding the hen house. Levin et al.(2003) also provided extended discussion of the major contributory role played by AERA insustaining educational research's “awful reputation” (Kaestle 1993; Sroufe 1997).

Alexander (2013) suggested, “One raison d'être for educational psychology researchers is tospeak directly and meaningfully about learning and development as it unfolds in schools andclassrooms.” Unfortunately, and with educational research's awful reputation still firmly plantedin short-termmemory, speaking directly may also be a raison de ridicule. In other words, becauseeducational researchers are often thought to make a mockery of the educational research process,researchers from other fields mock educational researchers. Alexander goes on, “To tell suchresearchers (and those who fund them) that they are not permitted to speak to the implications oftheir work within their research articles published within primary research journals is illogical. Infact, the failure to speak to the implications of their work for educational practice in a justifiedway and to the degree warranted by their studies and the resulting data represents a failure of thevery mission of such school-based interventions.” This is absolutely not what we arerecommending. Rather, we want these researchers to speak about research-based implicationsin the right venues and we argue that single-study, primary research articles are not a right venue.

Although Wainer and Clauser (2013) provided well-articulated arguments, we believethat their arguments are not accurately framed for our proposal as it is embodied. Weintended to re-present and elaborate that embodiment here, but the following personalcommunication that we received from our proposal coauthor, Erika Patall (to whom weare grateful), captures our sentiments perfectly:

[Wainer and Clauser] seem to be misinterpreting [Robinson et al.'s proposal], assertingthat [it] suggests that researchers should refrain from interpreting their data in singleinvestigations in terms of what it means for theory. This is not the argument…Rather, theargument is that researchers should hold off from making recommendations about whatpractitioners or policy-makers ought to do until an accumulation of high-quality evidencerelevant to that theory is available for interpretation [so that various] practical, social, orfinancial considerations can be weighed into the recommendation…[N]o single investi-gation can capture all the nuances, contexts, samples, etc., to which a theory strives toapply. That is, there is always some amount of speculation and generalization that theresearcher must make when discussing the correspondence between any one study andtheory—the very reason why not to make recommendations for practice based only onthat study. [A] single investigation may support the theory but the strength of a theory is

356 Educ Psychol Rev (2013) 25:353–359

found in an accumulated literature [and not in] a single investigation…Put simply,making recommendations for practice is very different from interpreting data as evidencesupporting or refuting a theory. However, [Wainer and Clauser seem] to be treating thetwo activities as one and the same [by assuming that Robinson et al.] have a similarposition on both (which they do not) (Patall, personal communication (May 14, 2013)).

We applaud Hedges' (2013) creative extension of Campbell and Stanley's (1966) originalexternal validity notions. With Bracht and Glass' (1968) population external validity, one isconcerned with treatment-effect generalization to participants (or other units) similar to thoseincluded in a given study. As Hedges pointed out, from a technical standpoint, this boilsdown to the type of sampling process that was employed to obtain the study's sample. WithBracht and Glass' ecological external validity, concern is with treatment-effect generaliza-tions from an individual study's situational factors (such as the physical context in which thestudy took place, instructional materials and procedures, assessment instruments, and thelike) to other situational contexts—which, from a technical standpoint, involves a samplingprocess that is considerably more challenging than that of drawing a probability sample ofparticipants from a pre-specified population. But then, how many more orders of“challengingness” magnitude, wonders Hedges, are required when generalizing from anexperimental intervention study's treatment effect outcomes to the social or educationalpolicy implications of those outcomes? From that perspective, Hedges' speculations provideadditional support for a proposal that seeks to separate the findings from a single primaryresearch study and blanket recommendations for practice based on those findings.

Robinson et al. (2013) suggested that authors may be inclined to “over”-speculatebecause reviewers and editors often “pressure” them to do so. Vaughn and Fuchs (2013)offered several additional potential reasons for “over-speculation,” including that suchbehavior can be self-serving to authors. We agree and hope that our proposal will targetnot only pressuring reviewers and editors but also self-served authors. Vaughn and Fuchsalso mentioned the new “can of worms” that is opened if we rely on syntheses for practicalrecommendations. Again, we agree that both traditional reviews and meta-analytic synthesesare susceptible to the same selective or misleading reporting and careless errors as arepresent in primary research studies, which would render suspect any recommendations thatflow from them. This point is well taken and provides grist for future discussion. Finally, weappreciate the “narrow conclusions” suggestion by Vaughn and Fuchs, where practicalimplications that emerge from tightly controlled experiments may be appropriate whenplaced in the appropriate context. Indeed, if “sins” of this type were the norm rather thanthe exception, then we would never have been motivated to formulate our proposal.

Concluding Comments

Sometimes when making an argument, one has to go to great lengths. Extreme statements,such as “never” and “always” tend to capture a reader's attention or even raise a reader'shackles. So it is with our “no recommendations for practice in primary research journals”suggestion. Do we believe that primary research journal authors—especially authors ofsingle correlational studies—should refrain from offering platitudinous universal prescrip-tives, such as “Findings from this study suggest the importance of early and continuedintervention by educators all over the world [emphasis added] to help all students [includingpresumably new female parents and consumers of a certain fruit pastry?] maintain positivebeliefs about themselves as mathematically and scientifically competent (see Robinson et al.

Educ Psychol Rev (2013) 25:353–359 357

2013)?” Absolutely, and if those authors cannot or will not refrain, then journal reviewersand editors should do their part either to help either restrain or retrain them. In short, indesperate times when journal authors' irresponsible “research-based” claims are runningamok, desperate measures may be required. And what about primary research authors whohave conducted carefully controlled, replicated, and extended experimental intervention(e.g., randomized controlled trial) studies? Should they be permitted to “hint at” the potentialimplications of their findings for practice, in the spirit of Vaughn and Fuchs' (2013) “narrowconclusions?” Absolutely, and about those authors we cannot really complain.

Finally, we would be remiss if we did not recommend to EPR readers the carefully craftedcommentary by coauthors Schraw and Patall (2013) on aspects of our “no prescriptives,please” proposal that relate to the current evidence-based practice movement in education.

References

Alexander, P. A. (2013). In praise of (reasoned and reasonable) speculation: a response to Robinson et al.’smoratorium on recommendations for practice. Educational Psychology Review. doi:10.1007/s10648-013-9234-2.

Bracht, G. H., & Glass, G. V. (1968). The external validity of experiments. American Educational ResearchJournal, 5, 437–474.

Cahnmann, M. (2003). The craft, practice, and possibility of poetry in educational research. EducationalResearcher, 32(3), 29–36.

Campbell, D. T., & Stanley, J. C. (1966). Experimental and quasi-experimental designs for research. Chicago:Rand McNally.

Carver, R. (1993). The case against statistical significance testing, revisited. The Journal of ExperimentalEducation, 61, 287–292.

Harris, K. R. (2003). Editorial: is the work as good as it can be? Journal of Educational Psychology,95, 451–452.

Harris, K. R. (2013). Disallowing recommendations for practice and policy: a proposal that is both too muchand too little. Educational Psychology Review. doi:10.1007/s10648-013-9235-1.

Hedges, L. V. (2013). Recommendations for practice: justifying claims of generalizability. EducationalPsychology Review. doi:10.1007/s10648-013-9239-x.

Kaestle, C. F. (1993). The awful reputation of education research. Educational Researcher, 22(1), 23–31.Levin, J.R., O'Donnell, A.M., & Kratochwill, T.R. (2003). Educational/psychological intervention research. In

I. B. Weiner (Series Ed.) & W. M. Reynolds & G. E. Miller (Volume Eds.). Handbook of psychology:vol. 7. Educational psychology. New York: Wiley, pp. 557–581.

Mosteller, F., Nave, B., & Miech, E. J. (2004). Why we need a structured abstract in education research.Educational Researcher, 33(1), 29–34.

Peng, C.-Y., Chen, L.-T., Chiang, H.-M., & Chiang, Y.-C. (2013). The impact of APA and AERA guidelineson effect size reporting. Educational Psychology Review, 25, 157–209.

Reinhart, A. L., Haring, S. H., Levin, J. R., Patall, E. A., & Robinson, D. H. (2013). Models of not-so-goodbehavior: yet another way to squeeze causality and recommendations for practice out of correlationaldata. Journal of Educational Psychology, 105, 241–247.

Renkl, A. (2013). Why practice recommendations are important in use-inspired basic research and why toomuch caution is dysfunctional. Educational Psychology Review. doi:10.1007/s10648-013-9236-0.

Robinson, D. H., & Levin, J. R. (2013). A “no prescriptives, please” proposal postscript: when desperate timesrequire desperate measures. Educational Psychology Review. doi:10.1007/s10648-013-9238-y.

Schmidt, F. L. (1996). Statistical significance testing and cumulative knowledge in psychology: implicationsfor training of researchers. Psychological Methods, 1, 115–129.

Schraw, G., & Patall, E. A. (2013). Using principles of evidence-based practice to improve prescriptiverecommendations. Educational Psychology Review. doi:10.1007/s10648-013-9237-z.

Sroufe, G. E. (1997). Improving the “awful reputation” of education research. Educational Researcher, 26(7),26–28.

Thompson, B. (1996). AERA editorial policies regarding statistical significance testing: three suggestedreforms. Educational Researcher, 25(2), 26–30.

358 Educ Psychol Rev (2013) 25:353–359

Vaughn, S., & Fuchs, L. S. (2013). Staying within one's data to make recommendations for practice in primaryeducational research journals. Educational Psychology Review. doi:10.1007/s10648-013-9232-4.

Wainer, H., & Clauser, B. (2013). Reflections on a too extreme idea. Educational Psychology Review.doi:10.1007/s10648-013-9231-5.

Walster, G. W., & Cleary, T. A. (1970). A proposal for a new editorial policy in the social sciences. TheAmerican Statistician, 24, 16–19.

Educ Psychol Rev (2013) 25:353–359 359