Calhoun: The NPS Institutional Archive

Faculty and Researcher Publications Faculty and Researcher Publications Collection

2011-10

Reproducible (operations) research

Nestler, Scott

Operations Research Society of America

OR/MS Today, October 2011, pp.22-28

http://hdl.handle.net/10945/47788

Reproducible (Operations) Research

By Scott Nestler

A primer on

reproducible research

and why the O.R.

community should

care about it.

22

"Let's be clear: the work of science has nothing whatever to

do with consensus, which is the business of politics. What is

relevant is reproducible results. The greatest scientists in

history are great precisely because they broke with the

consensus. "

- Michael Crichton

uried in the middle of physican-author Michael Crichton's quote about the place (or lack thereof) of consensus in science, is an

alliteration - reproducible results - that he claims (and I suspect most would agree) is

relevant to science [1]. The traditional meaning of reproducible results addresses the

verification of a scientific experiment by other researchers using an independent experi­

ment. However, computational science poses new challenges to the scientific tradition

[2]. Science often proceeds by iterative refinements where the works themselves (Le., the

explicit computations) are seldom published, and it is difficult for others to refine or

improve them [3]. As expressed by the Yale law School Roundtable on Data and Code

Sharing, "Generating verifiable knowledge has long been scientific discovery's central

goal, yet today it's impossible to verify most of the computational results that scientists

present at conferences and in papers" [4].

ORIMSTODAY October 2011

c repn cone duce expe anyo reeOJ same coml resea ofre~

tiona thep reprc resea regar medi nal p topic

Evid A

had t have Engle Som! nals, artic! quesl meth th is i l ject i ~

pies f Vitan Time Trust

R, mati( olof;} that t Most ventil

sonal equiv [10]" INFC INFC tical j

said a publi, ical ir other dates RR, ( soluti

~ ,

;

1

II

5

October 10 II

Using the definition of Baggerly and Berry, reproducible research (RR) generally means that conclusions from a single experiment can be repro­duced based on the measurements from that single experiment [5]. Note that it does not mean that anyone conducting the experiment again (but recording different measurements) would get the same results. A more precise definition for use in the computational sciences, might be: Reproducible research refers to the idea that the ultimate product of research is tile paper along with the full computa­tional environment used to produce the results in the paper such as the code, data, etc. necessary for reproduction of the results and building upon the research [6]. While the majority of the discussions regarding RR have been in some intersection of the

Author

1 Database

... Analytic code --+- Presentation code --+- Processing code

•

Reader

medical, bioinformaticslcomputational biology,sig- I figure 1: The research pipeline as a model for reproducible research [16[. nal processing and statistical communities, this is a topic which operations researchers should care about as well.

Evidence of a Problem? A NUMBER OF SCIENTIFIC JOURNALS have recently

had to publish retractions or statements of concern [7]. These have included highly regarded journals, such as: LOl1cet, New Ellgland JOllrnal of Medicille, and Allllois of Tnternol Medicine. Some awareness of this issue extends beyond scholarly jour­nals, due to reporting in the popular media. For example, an article in The New Yorker last year attempted to answer to the question, "Is there something wrong with the scientific method?" While reproducibility (or replicability, as termed in this instance) was not the primary focus of the article, the sub­ject is addressed 18 ]. The author provides a number of exam­ples from medicine (e.g., anti-psychotic drugs, cardiac stents, Vitamin E), psychology and ecology. An article in the New York Times declared: " Reporters Find Science Journals Harder to Trust, but Not Easy to Verify [9]:'

Robert Gentleman, a well-known statistician and bioinfor­matician. points out that in much modern research, "method­ology is often so complicated and computationally expensive that the standard ... journal paper is no longer adequate .... Most statistics papers, as published, no longer satisfy the con­ventional scientific criterion of reproducibility: could a rea­sonably competent and adequately equipped reader obtain equivalent results if the experiment or analys is were repeated? [ 101 " Reading through the various journals published by INFORMS, e.g., Operations Research, Manager/'lell1 Science, INFORMS JOllmal all Computing, etc. or the American Statis­tical Association's Tecllllometrics, it appears the same could be said about many of the articles appearing in our professional publications. While many articles in these journals are theoret­ical in nature, they often include results from simulations or other computational experiments that could be prime candi­dates for RR. if the authors were: ( I) aware of the advantages of RR, (2) familiar with helpful techniques and technological solutions, and (3) committed to (' working reproducibly."

History of and More Details About RR THE FO UNDATIONS OF REPRODUCIBILITY can be

found in Aristotle's Dictum about there being no scientific knowledge in the individual [I I ]. Jon Claerbout, a geophysics professor at Stanford University, observes, '(From Euclid's rea­soning and Galileo's experiments, it took hundreds of years for the theoretical and experimental branches of science to devel­op the standards for publication and peer review that are in use today. Computational science. rightly regarded as the third branch, can walk the same road much faster" [121. Most sources credit Claerbout as being the first to champion RR in the computational sciences [2, 13]. In 1995, Buckheit and Donoho summarized Claerbout's ideas as follows: "An article about computational science in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship. The actual scholarship is the complete software development environment and the complete set of instructions which gen­erated the figures" [14 J. In recent work, David Donoho sug­gests that the word "knowledge" be substituted for "scholarship" [l3 ].

Gentlemen and Temple Lang take these suggestions a step further. Besides the figures in a published document, they advocate that authors should provide «explicit inputs and code that can be used to replicate the results;' i.e., tables, figures, etc. based on computation and data analysis [15]. Figure I shows the "research pipel ine" modellhat Peng and Eckel, who work in bioinformatics, use to describe RR 116] . Some of the key steps are: (I) the pipeline begins with measured data that is transformed by processing code into analytic data; (2) the ana­lytic code turns the analytic data into computational results; (3) these are then summarized by the presentation code into fig­ures and tables in the text. Note that au thors and readers use the pipeline in different directions. Authors start with data, generate analysis and produce the paper itself; readers start at Ihe other end with lhe text of the paper, and if they are inter­ested, examine the analysis by acquiring the data and code that the authors have provided.

www.orms-today.org 23

REPRODUCIBLE O . R .

"

Motivations to Work Reproducibly DONO HO RECENTLY PROVIDED this list of reasons

for working reproducibly [13 I. While his target audience is primarily biostatisticians, many of his suggestions apply to O.R. analysts wi th littl e o r no modification. 1. Improved work products and habits. Because we

know that our scr ipts will be avai lable to others, we will improve them to a higher level of quality than we would if they were only for our own consumption. But, researchers later re turning to their earlier works could also benefit. (I know this is true in my case and suspect it holds for others.)

2. Improved teamwork. When working as part of a team, our colleagues can see what we are doing in a more transparent manner and may be more likely to propose improvements. Additionally, confidence in results produced by other team members will be higher.

3. Greater impact. If we make it easier for other researchers to use the methods we have developed, it should lead to more acknowledgement (by way of increased citations) fro m other researchers using our computat ionally reproducible work.

4. Greater continuity and cumulative impact. For ongoing, longer-term projects, working reproducibly can ease the integration of other researchers and students into the team and better preserve the efforts of team members after they depart the project.

Donoho also points out that taxpayers should want publicly funded research efforts to result in computational reproducibil­ity. Besides providing good stewardship of work product pur­chased with public funds, working reproducibly also: (I) ensures that access to the work continues after the project is over; and (2) increases the availability of publicly-funded spon­sored research to other researchers and the general public. He states in summary, "I believe anyone who understands the process and the benefits (of RR) will eventually be moved to practice it" [131 .

Funding agencies and grant reviewers can also influence reproducibility through a variety of means, to include: ( I) requiring some projects to fu lly implement reproducibility in their workflow and publica tions; (2) funding the creation of tools that better support reproducibility in their field; and (3) others as outlined by the Yale Law School Roundtable on Data and Code Sharing [4 I.

Common Objections to RR RESEA RC HERS GIV E a number of reasons for not

"working reproducibly." Some of these are simply res is­tance to change or "knee-jerk" objections, whi le others are more considered objections and deserve more thoughtful responses. Here are some of the protest s that Donoho and his colleagues have encountered wh en encouraging others to practice RR, along with som e pos­sible responses [ 17 I: 1. It takes extra work. This is indeed true, especially

when starting to do RR, and breaking your old, informal, non-reproducible habits.

2. Nobody else does it. If you actually work reproducibly, and make your code and data available, it will: (a) ge t noticed, (b) get used, and (c) become a reliable tool.

3. My work is too complicated. This is unlikely, but even if true, why should anyone believe what you write and publish? Don't you exercise your computations on test data to verify them? Let o thers see what you have done. Reproducibil ity may even be morc important in this case.

4. It undermines the creation of intellectual capital. Because tools are given away before they ripen, the researcher cannot develop a toolkit over a career. Maybe, but is the purpose of your publication scholarship, personal aggrandizement and/o r financial gain? Repeating prior advice, working reproducibly can improve teamwork and get your work noticed.

5. Legal issues. There are indeed concerns with copyrights, patents and licenses. Victor ia Stodden has several articles on this issue that might be of interest to

Interested Parties and Shared Responsibilities

SCIENTISTS and researchers themselves are not the only ones with important roles in RR; significant responsi bility also rests with others. Editors of scho larly journals p laya key

~ Researchers g ive a nu m ber those with concerns about

. this aspect of RR, such as the of reasons for not "working Reproducible Research

reproducibly." Some are simply Standard (RRS), discussed

resistance to change or "knee- shortly [ 18,191·

jerk" objections; o thers are more • Possible Solutions and

considered objections. .. Ways to Ease the Pain A LITERATE PROGRAM, as

defined by Donald Knuth in 1992, is a document that contains both code and text segments [201. The text provides an expla­nation of what the code actually does when it is executed. Lit­erate programs support two types of transformations, for different audiences. Weaving a literate program creates the doc­ument for a human reader, while tang[hlg the same me hides the text and allows the code to be compiled or evaluated by a

role in establishing reproducibility standards in their fields. They can do this in a number of ways, including: ( I ) imple­menting po licies for the provision of stable URLs for open data and code associated with published papers; (2) requir: ing the replication of computational results prior to publi­ca tion; and (3) requiring appropr iate code and data citations.

ORIMSTODAY O ctober 20 I I

comput on usin com me Leisch, ' Rand tl

As sl in a Lal mat; Uta

extracte weavers ing: odf and Mi StatWea

How. this appl lab-base which h;

supplies have ere reprodUi a datab.,

propose> For use it an add-;,

A COl

called B code inti evidentl of the so website I

Onel which is used for software replicatiJ

One 51, IN A

working and his c 1. Post.

and p alrea(

2. Theo and e differ

3. One-, short-

Some The first could fin. of the set journals I tional res

ltluence ude: (I) ibility in ation of and (3) on Data

fo r not y resis­: o thers ~ more sts that I when ne pos-

y

Ic ibly, ) get lOl. even e and n test

lI1tin

:1. Ie

lIlcial ,Iy

d.

1 has rest to

·ut s the

ed

nd In !AM,as :ontains 1 expIa­ted. Lit­)I1S, for :he doc­Ie hides ted by a

ctober 20 I I

computer. Within the statistical community, the focus has been on using literate programming or close variants. The most common implementation is .Sweave [2 1J, from Friedrich Leisch, which combines the statistical programming language R and the typesetting markup language laTeX.

As shown in Figure 2, "Weaving" a Sweave document results in a laTeX me that can be processed into a PDF or other for­mat; "tangling" the same document yields code that has been extracted for use within R. I am aware of a number of o ther weavers, for use with various programs and languages, includ­ing: odfWeave (for R with OpenOffice documents), R2wd (R and Microsoft Word), SASweave (SAS with L.,TeX), and StatWeave (R, SAS, Stata, and Maple) [6].

However, one need not be a statistician using R to benefit from this approach. Donoho and his colleagues provide extensive Mat­lab-based tools, including the Wavelab and Sparselab packages, which have been developed over a IS-year period [ 17[. leVeque supplies a Python interfuce to Fortran code [23] . Peng & Eckel have created a "elCher" package for R, which enables modular reproducibility by storing results of intennediate computations in a database [16] . Instead of laTeX, anod,er biostatistics researcher proposes using &~ensible Markup Language (XML) with R [24[ . For use in computational biology, Mesirov provides GenePattem, an add-in to Microsoft Office [2] .

A commercial effort called " Inference," from a company called Blue Reference, attempted to integrate Rand Matlab code into Microsoft Office documents. Even though this effort evidently ceased development in 2009, demonstration copies of the software are still available at no cost from the company website [25].

One proposal to assist with RR legal issues is Stodden's RRS, which is similar to the way the GNU Public License (GPL) is used for open-source software. RRS, however, goes beyond software to include all other data and procedures necessary for replicating a computational experiment [18].

One Size Does Not Fit All IN ADDITION TO GIVING EXAMPLES of successes

working reproducibly with Matlab-based toolboxes, Donoho and his colleagues identity three areas where RR has failed [171. 1. Postdocs. Postdocs are typically in a rush to publish

and prefer to work in a manner with which they are already comfo rtable. Do you blame them?

2. Theorists. A theoretical paper with very few d iagrams and calculations may not benefit from or look any d ifferent as a result of application of RR.

3. One-off Projects. Some projects are either so small or short-lived that the addit ional work just isn't worth it.

Some of these exceptions might apply within O.R. as well. The first and third are likely directly transferrable. While one could find instances of theoretical papers fitting the description of the second item on this list, the majority of papers in O.R. journals do include multiple figures and numerous computa­tional results that could benefit from RR.

R CMD Sweave \ pdflatex . lex .pdf

1 J

( .Rnw )

I . I .R ) R CMD Stangle'

I Figure 2: Results of weaving and tangling a Sweave document (221.

One Journal 's Efforts OVER THE PAST COU PLE YEA RS, Biostatistics, a peer­

reviewed journal published by Oxford University Press, has encouraged authors to employ RR. The editors, Peter Diggle and Scott Zeger, recently provided the following explanation of why they chose this route 126]:

"Ollr aim was not to police the technical correct7less of published work bllt rather to recognize that a nontrivial statistical analysis involves ma1/y deci­sions by the mlalyst that are ope" to debate. This is especially true ill more complex mwiyses, for exam­ple, if! a Bayesian analysis lIS;llg Markov chain Mo1lte Carlo that involves choices about the prior, the samplel~ the burn-ill, and the cOflverget'lce crite­ria. All these may affect the inference draw1l from the data, arId the reader would be lVeli served by giv­ing the ability to find Ollt."

The editors further point out that, by making their work reproducible, researchers can render state-of-the-art methods more accessible to others and help preclude the abuse of methods in situations where the assumptions on which they rest are not likely to be satisfied. Similarly, in 0.1<. we often make modeling asslunptions and decisions. While we may state these in words, translating them unambiguously into an implementable form for modeling purposes by other researchers can be difficult.

Searching for Evidence of RR In O.R. My EFFORTS TO DISCOV ER RR in O.R. began with

querying proponents in other disciplines, including several researchers previously mentioned. When that failed, my endeavor moved to the Internet. A search of the INFORMS website for the term "reproducible" yielded eight results. Of these, three were germane. First, the Finance department edi­tor 's statement for Management Science says, "Authors of empi rical and quantitative papers should provide or make available enough information and data so that the results are reproducible" [271. The second result was in an article about Office of Management and Budget Circular A-4 in a Decision Analysis Society Newsletter from 2004 [28]. The last was in the title and abstract for a talk at ICS 2009, the 11th INFORMS Computing Society Conference, held in Cha rleston, sc. The presentation, "GAMSWorid and the Growi ng Demand for Reproducible Computational Experi-

www.orms-today.org lS

REPRODUCIBLE O.R.

26

ments;' attempted to draw the attention of the mathematical programming commun ity away from a myopic focus o n performance testing and benchmarking. The GAMSWorid website (http://www.gamsworld.org) offers "well -focused, tested and mainta ined components (e.g .• model libraries, tools for generating, collecting and analyzing results) to use as building blocks in making reproducible experiments" [29 [. A similar search for the word "replicable" produced II more results; none related to RR. Searches for related terms might yield other instances I did not discover.

Some Examples From O.R. FOR A RELATIVELY RECENT EXAM PLE of work that is

not easily reproducible, look at a paper published in Naval Research Logistics in 2009 on which I was a co-author [30] . This

The previously described tools for producing RR in statistics will be of use to some o.R. analysts, but they clearly are not suf­fic ient for all purposes. In particular, the various methods of combining R and laTeX are of limited use to those in optimiza­tion. However, ubiquitous mathematical modeling languages (e.g., GAMS) and high-quality commercial optimization pack­ages (e.g., IBM's CPLEX, free for academic research) are avail­able and can make models portable, supporting RR. The GAMSWorid site previously introduced appears to be worth consideration for those working in math progrmlffiing or opti­mization. Also, COmputational Infrastructure for Operations Research (COIN-OR) is an initiative for open-source software in o.R.; the items in their mission statement indicate that their projects support RR, without explicitly mentioning the term.

article studies the Shewhart chart of Q statistics proposed for the So What Can We Do About It? detection of process mean shifts in start-up processes and short AS MENTIONED EARLIER, the responsibilities for RR are runs. Exact expressions for the rllil-Iength distribution are shared. If we, as individual researchers, work through the chal­derived and evaluated using an efficient computational proce- lenges involved with making research reproducible, there are dure that Gill be considerably faster than using direct simulation. ways to overcome these obstacles. A few examples are presented While we provided pseudo-code for both Oll r procedure and the here; there are probably more, perhaps more applicable to some direct simulation methods, and included the somewhat standard areas of o.R. but not known to me. Others may yet need to be phrase, " ... are available upon request from the allthors;' I SliSpect developed. If YOll are already using LaTeX and either R or Mat­it would be difficult for other researchers (and perhaps even our- lab for statistical analysis and simulation, the move to using

selves) to replicate the results ~ The In ternet is potential [y a Sweave or a similar option is based on solely on what was pro- .. . . painless. If not, consider an inter-videdinthepaperitself. s ign ificant a id to those WhO~ im step; lise scriptable code,

A much more h ighly refer- desire to make the ir results rather than spreadsheet or other

enced paper(6JS CItes accordmg reproducib[e. GUI-based analysis, which are to Google Scholar) published in inherently less reproducible. Inte1faces in 1995, "Global Supply Chain Management at Digital If you aren't yet convinced, but want to know more about Equipment Corporation;' was volunteered by one of the article's RR, visit Reproducible Research Planet, http://www.rrplan­authors (Gerald Brown) as a poor instance of RR [3 1 J. He et.org. There is an active Google Group on the topic available explains, "There was no way Digital Equipment would release at http: //groups.google.com/grouplreproducible-research. their strategic plan, and we did not think to produce a pilot:' Additionally, the website http://www.reproducibleresearch.org Brown was also part of a team that wrote an ea rlier paper"Oesign contains links to a number of useful resources, including many and Implementation of Large Scale Primal Transshipment Algo- of the papers cited in this article. If you are working in opti­rithms" in 1977 that was much closer to being considered RR. mization, visit the GAMSWorid site, its companion Google This work included statements such as, "a set of standard test Groups site, http://groups.google.com/group/gamsworld, and problems [from NETGEN, 43] that have also been solved by the COIN-OR repository, http://www.coin-or.org. other contemporary codes." ... "The FORTRAN program For those serving on editorial boards of scientific jour­GNET/Depth [6], 1975, is distributed to researchers for a nomi- nals, consider whether the articles that appear in your nal handling charge on an exclusive use basis. For further infor- publication, and your readers, could benefit from authors mation write ... " [32]. He reports that they shipped hundreds of who worked in a reproducible manner. When was the last these, worldwide. So, it appears that this approach can work. time your editorial statement was reviewed and updated?

The Internet is potentially a significant aid to those who Perhaps it is time for a revision that encourages or even desire to make their results reproducible by ma.lting available promises to reward RR. The same applies for organiza­the underlying data and algorithm code. An example of this tions who sponsor sc ientific publications. If INFORMS. can be seen at http://faculty.nps.edu/awashburn/, where the with 12 schola rly journals, were to follow the lead of "Downloads" link includes additional hyperiinks to numerous Biost(llistics, imagine how far reproducibility in O.R. zipped applications and Excel workbooks that accompany could progress in a re latively shor t time. many of Alan Washburn's recent publications [33]. While this doesn't fully meet the definition of RR provided earlier, it is evi­dence of steps toward reproducibility taken by one operations researcher and his colleagues.

That's a Wrap! BAGGERLY AND BERRY CAUTION US, "RR is neces-

sary, but not sufficient, for good science. It needn't contain

ORIMSTOOAY October 20 I I

KNITR diffiCl for ml

Try it [ trial 0

See fo to no~

REPRODUCIB L E O.R.

28

the motivation for what was done, and the motivation may be data·dependent .... perhaps the data we used were 'cleaned' before we got them. These potentially fatal biases wi ll no t be known by someone checking reproducibility, and they may not be known to the primary analyst" [51. David Banks at Duke Un iversity and a former editor of the jOllrnal of the American Statistical Association is skeptical of the RR movement to date. He has expressed, "My own sense is that very few app lied papers are perfectly reprod ucible." Howev· er, he also suggests that a reproducibility standard is a noble aspiration 1341.

The question that Sergey Fome! and Jon Claerbout (the "father" of the RR movement in the past two decades) suggest we ask ourselves before we publish our next paper is, "Have I done enough to allow the readers of my paper to ver ify and reproduce my computational experiments?" [121. If your answer is "no;' or even "not quite;' consider the ideas present­ed here and how they might help you in an effort to make your

ACKNOWLEDGMENTS

The author thanks Professor Emeritus Gordon Bradley, Distinguished

Professor Gerald Brown and Dist inguished Professor Emeritus Alan

Washburn, his colleagues at the Naval Postgraduate School, for review­

ing drafts of this article and providing constructive feedback that led to

sign if icant improvements.

work an exemplar of RR in O.R., or ... Reproducible Operations Research (ROR). IORMS

Scott Nestler is a lieutenant c%nel in the U.S. Army and an

assistant professor in the Operations Research Department

at the Naval Postgraduate School. Previously, he was on the

faculty at the U.S. Military Academy at West Point, served as

the Chief of Stra tegic Assessments, Multi-National Force Iraq

(MNF-I) at the U.S. Embassy in Baghdad, Iraq, and has worked

as an O.R. Analyst on the Army Staff in the Pentagon. Nestler

has a Ph.D. in Management Science from the Robert H. Smith

School of Business, University of Maryland·College Park; he is

also an Accredited Professional Statistician™.

REfERENCES 1. Crichton. M .. speech at California Institute of Technology, Pasadena,

Calif., Jan. 17. 2003. 2. Mesirov, J, -Accessible Reproducible Research.- Science Magazine, Vol. 327, Jan. 22, 2010. 3. Gentleman, R., "Reproducible Research: A Bioinformatics Case Study,· Statistical Applications in Genetics and Molecular Biology, Vol.

4, No.1, pp. 1-23, 2005. 4 . Yale Law School Roundta'ble on Data and Code Sharing.

-Reproducible Research: Addressing the Need for Data and Code Sharing in Computational Science," Computing in Science and Engineering, Vol 5. No.5, September/October 2010. 5. Baggerly, K., and O. Berry, "Reproducible Research: AMSTAT NEWS, January 2011. 6. Reproducible Research Planet!, http://www.rrplanet.com/ reproducible-research/reproducible-research.html.

7. Laine, C. and others. "Reproducible Research: Moving Toward Research the Public Can ReaUy Trust,· Annals of Internal Medicine, pp.

450-454, March 20. 2007. 8. Lehrer. 1.. "The Truth Wears Off,· The New Yorker; Dec 13, 2010. 9. Bosman, J.. "Reporters Find Science Journals Harder to Trust, But Not

Easy to Verify," New York Times, Feb 13, 2006. 10. Green. P., "Diversities of Gifts, But the Same Spirit," The Statistician, pp. 423·438, 2003.

11. Turner, W., "History of Philosophy: Ginn and Company. Ch . 11, 1903. 12. Fomel. S" and J. Claerbout, "Reproducible Research: Computing in Science & Engineering, pp. 5-7, January/February 2009. 13. Donoho, D., "An Invitation to Reproducible Computational Research." Biostatistics, Vol. 11, No.3, pp. 385-388, 2010.

14. Buckheit. J., and D. Donoho, "Wavelab and Reproducible Research : "Wavelets and Statistics." Springer-Verlag, 1995.

15. Gentleman. R .. and D. Temple Lang, -Statistical Analyses and Reproducible Research : Bioconductor Project Working Papers, Paper 2.

2004. 16. Peng, R" and S. Eckel. "Distributed Reproducible Research Using Cached Computations,· Computing in Science & Engineering, pp. 28-

34. January/ February 2009. 17. Donoho, D., and others, "Reproducible Research in Computational

Harmonic Analysis: Computing in Science & Engineering, pp. 8·18. January/February 2009. 18. Stodden, V" -The Legal Framework for Reproducible Scientific Research: Licensing and Copyright" Computing in Science &

Engineering, pp. 35·40, January/February 2009.

19. Stodden. V .. "Enabling Reproducible Research: Licensing for Scientific Innovation," International Journal of Communications Law & Policy. pp. 1-25, winter 2009.

20. Knuth, D., "literate Programming: Center for the Study of language

and Information. Stanford, Calif., 1992. 21. Shotwell, M. BioStatMatt, http://biostatmatt.com/archives/1402.

2011. 22. Leisch. F .• "Sweave: Dynamic Generation of Statistical Reports Using Literate Data Analysis," Compustat 2002 - Proceedings in Computational Statistics, pp. 575-580, 2002. 23. LeVeque, R., "Python Tools for Reproducible Research on Hyperbolic

Problems,- Computing in Science & Engineering, pp. 19·27. January/ February 2009.

24. Temple lang, D .. "Embedding S in Other languages and Environments," Proceedings of the 2nd International Workshop on Distributed Statistical Computing, March 15·17, 2002. 25. Blue Reference, "Inference: A Solution Platform for Business Professionals,· http://inference.us. 2009.

26. Diggle, P., and S. Zeger. "Editorial,· Biostatistics, Vol. 11, No. 3, p. 375,2010.

27. Editorial Statements. Management Science, INFORMS. http://www.informs.org!content/download/15885/182755/file/Depart ment Editorial Statements.pdf, p. 2.

28. Morgan. K .. "Opportunity for DA: Decision Analysis Newsletter, Vol. 23, No. 1, pp. 6-7, March 2004.

29. Dirkse, S .. and others, "GAMSWorld and the Growing Demand for Reproducible Computational Experiments.- Conference Program, ICS2009, 2009.

30. Zantek, P., and S. Nestler, "Properties of Q-Statistic Monitoring Schemes for Start·Up Processes and Short Runs.- Naval Research Logistics, April 2009.

31. Arntzen, B., and others, "Global Supply Chain Management at Digita l

Equipment Corporation: Interfaces, Vol. 25. No. 1. pp. 69·93, January/ February 1995. 32. Bradley, G., and others, "Oesign and Implementation of large Scale

Primal Transshipment Algorithms: Management Science, Vol. 24, No.!. pp. 1·34 , September 1977.

33. Washburn, A., Naval Postgraduate School.

http://faculty.nps.edu/awashburn/. 34. Banks, D., "Reproducible Research: A Range of Response: Statistics, Politics and Policy, Vol. 2, No.1. Art. 4. 2011.

OR/MSTODAY O cto b er 20 I I

Fi n(