NORTH CAROLINA LAW REVIEW

Volume 93 | Number 1 Article 7

12-1-2014

Shaping the Technology of the Future: PredictiveCoding in Discovery Case Law and RegulatoryDisclosure RequirementsChristina T. Nasuti

Follow this and additional works at: http://scholarship.law.unc.edu/nclr

Part of the Law Commons

This Comments is brought to you for free and open access by Carolina Law Scholarship Repository. It has been accepted for inclusion in NorthCarolina Law Review by an authorized administrator of Carolina Law Scholarship Repository. For more information, please contactlaw_repository@unc.edu.

Recommended CitationChristina T. Nasuti, Shaping the Technology of the Future: Predictive Coding in Discovery Case Law and Regulatory DisclosureRequirements, 93 N.C. L. Rev. 222 (2014).Available at: http://scholarship.law.unc.edu/nclr/vol93/iss1/7

Shaping the Technology of the Future: Predictive Coding inDiscovery Case Law and Regulatory Disclosure Requirements*

[Aicquiring preemptive knowledge about emerging technologiesis the best way to ensure that we have a say in the making of ourfuture.

-Catarina Mota, TEDGlobal Fellow'

INTRODUCTION .......................................... 223I. THE RISE OF ELECTRONIC DATA AND AN

INTRODUCTION TO PREDICTIVE CODING ...................... 226A. The Electronic Revolution ............... ...... 226B. Concepts and Terminology Surrounding Predictive

Coding ..................... ............. 229II. PREDICTIVE CODING AND THE LEGAL WORLD ..................... 234III. PREDICTIVE CODING AND RECENT CASES.... .............. 242

A. Da Silva Moore v. Publicis Groupe ....................243B. Cases Following in Da Silva Moore's Footsteps.................249C. Progressive Casualty Insurance v. Delaney: Enforcing

Judicial Predictive Coding Norms.....................253IV. PREDICTIVE CODING AND REGULATORY AGENCIES............257

A. The Department of Justice .................... 259B. The Securities and Exchange Commission ............... 263C. The Federal Trade Commission ............ ..... 266

* @ 2014 Christina T. Nasuti.1. Catarina Mota, Play with Smart Materials, TED (July 2012), http://www.ted.com/

talks/catarinamota-play-with-smart-materials/transcript?language=en.

PREDICTIVE CODING

V. AN OVERVIEW OF INDUSTRY INSIGHTS ANDMISCELLANEOUS FACTORS TO CONSIDER WHENDEVELOPING THE RULES AND REGULATIONSSURROUNDING PREDICTIVE CODING ................... 269

VI. COMPARISONS, DIFFERENCES, AND RECOMMENDATIONSFOR COURTS AND AGENCIES ADDRESSING THE USE OFPREDICTIVE CODING ..................................... 271

CONCLUSION ....................................... ..... 274

INTRODUCTION

The twentieth and twenty-first centuries brought about atechnological revolution for businesses and everyday life. The legalprofession was not left unchanged, as innovations from typewriters tocomputers to the Internet transformed legal practice. 2 Thecorresponding creation and accumulation of electronic data continuesto fundamentally impact legal practice,3 particularly by accounting formassive increases in digital matter available for legal review.'Computer-assisted review technologies, and predictive coding inparticular, permit the legal field to foray into the digital mass thatnow defines the professional world.' It is not surprising to see that

2. See WORKING GRP. ON ELEC. DOCUMENT RETENTION & PROD., THE SEDONACONFERENCE, COMMENTARY ON INFORMATION GOVERNANCE 2 (2013) [hereinafterTHE SEDONA CONFERENCE], available at https://thesedonaconference.org/download-pub/3421 ("We live and work in an information age that is continually-and inexorably-transforming how we communicate and conduct business.").

3. Cf. Darla W. Jackson, Lawyers Can't Be Luddites Anymore: Do Law LibrariansHave a Role in Helping Lawyers Adjust to the New Ethics Rules Involving Technology?,105 LAW LIBR. J. 395, 397-400 (2013) (discussing the evolving technological demands onlawyers and how law librarians can aid practitioners).

4. See John T. Yip, Addressing the Costs and Concerns of International E-Discovery,87 WASH. L. REV. 595, 595-96 (2012).

5. See, e.g., Maura R. Grossman & Gordon V. Cormack, The Grossman-CormackGlossary of Technology-Assisted Review with Foreword by John M. Facciola, U.S.

2014] 223

NORTH CAROLINA LAW REVIEW

these technologies provide important future venues for dealing withthe electronic realities. What is significant, however, is the rapiditywith which these technologies could be incorporated into the currentjudicial and regulatory regimes. Because the legal field already isfeeling the impact of these technologies on its traditional review andconflict-resolution mechanisms, this Comment focuses on howjudicial and regulatory regimes should react to the changinglandscape.

The framework in which these emerging technologies developwill guide how they evolve and whether they can be sufficientlyimplemented in the legal context. As a result, the legal professionmust advocate for courts and regulatory bodies to promptly developand update guidelines intended to address predictive coding andother computer-assisted review technologies. While recent cases haveprovided some transparency into the judicial perspective onpredictive coding and its role in litigation, agencies have generallybeen less transparent in their implementation or acceptance ofpredictive coding in the regulatory context. This Comment arguesthat, to effectuate a meaningful evolution and implementation ofpredictive coding, courts and government agencies should notexperiment in vacuums independent of each other. Rather, bothcourts and agencies should demonstrate the utmost transparencywhen utilizing and addressing this new technology. Increasedtransparency will allow each body to learn from the other'sexperiences and eventually enable both to put predictive coding

Magistrate Judge, 7 FED. CTS. L. REV. 1, 27 (2013) ("A 2012 study (Nicholas M. Pace &Laura Zakaras, Where the Money Goes: Understanding Litigant Expenditures forProducing Electronic Discovery, RAND Institute for Civil Justice (2012)), indicat[ed] thatDocument review accounts for 73% of Electronic Discovery costs, and conclude[ed] that'[t]he exponential growth in digital information, which shows no signs of slowing, makes acomputer-categorized review strategy, such as predictive coding, not only a cost-effectivechoice but perhaps the only reasonable way to handle many large-scale productions.' ").

[Vol. 93224

PREDICTIVE CODING

technology to its highest and best use in appropriate contexts, whilestill maintaining a healthy skepticism regarding the different uses andchallenges posed in each venue. The need for transparency andconsideration of other contexts is especially important whenconsidered in light of normative concerns. Namely, the juxtapositionof these forums raises an important question regarding whethercourts and agencies should consider each other and work in tandem-otherwise, predictive coding technologies could hypotheticallysupport an agency determination of wrongdoing but remainunacceptable for use in a judicial context.

This Comment addresses the origins of, current status of, andfuture possibilities for predictive coding. To that end, Part I addressesthe rise of electronic data and introduces the reader to predictivecoding as well as other terminology and concepts surroundingcomputer-assisted review. Part II surveys the general status ofpredictive coding and the opportunities it presents to the legalprofession. The next two parts, Parts III and IV, address howpredictive coding is applied and addressed in cases by the judiciaryand by regulatory agencies. Part V provides a relatively briefoverview of miscellaneous issues to consider as judges, regulators, andreformers develop rules, regulations, and recommendationsaddressing the legal use of predictive coding. Finally, Part VIcompares and distinguishes the judicial and regulatory treatments ofpredictive coding and offers recommendations for each in order toassure best practices.

2014] 225

226 NORTH CAROLINA LAW REVIEW [Vol. 93

I. THE RISE OF ELECTRONIC DATA AND AN INTRODUCTION TOPREDICTIVE CODING

A. The Electronic Revolution

The recent data revolution resulted in a global shift from hard-copy files and communications to electronic versions.' This shift wasinspired by the newfound ability to create and maintain electronicrecords and communications-a change so fundamentallyrevolutionary that it is sometimes compared to the fifteenth centuryintroduction of the printing press.' Companies and individual usersare not alone in this electronic shift. The government is a substantialforce in the move, as the executive branch has mandated thatagencies embrace the digital reality.! As a result of the nearly

6. See, e.g., Judge Herbert B. Dixon, Jr., Automating the Search and Review of ESI,JUDGES' J., Summer 2012, at 36, 36 ("[Vlarious estimates [claim] that from 90 percent to97 percent of today's business and personal records are created and maintainedelectronically, and that as little as 3 percent of information is printed on paper...Grossman and Cormack define electronically stored information ("ESI") as a term

[ulsed in Federal Rule of Civil Procedure 34(a)(1)(A) to refer to discoverableinformation "stored in any medium from which the information can be obtainedeither directly or, if necessary, after translation by the responding party into areasonably usable form." Although Rule 34(a)(1)(A) references "Documents orElectronically Stored Information," individual units of review and production arecommonly referred to as Documents, regardless of the medium.

Grossman & Cormack, supra note 5, at 157. See Working Grp. on Elec. Document Retention & Prod. & Search & Retrieval

Scis. Special Project Team, The Sedona Conference, Best Practices Commentary on theUse of Search and Information Retrieval Methods in E-Discovery, 8 SEDONA CONF. J. 189,197 (2007) [hereinafter The Sedona Conference Working Group] ("This 'digitalrealm' ... resulted in as fundamental a shift in the way information is shared as that whichoccurred in 1450 when Johannes Guttenberg invented the printing press.").

8. See Joseph Marks, White House Overhauls Electronic Records Requirements,NEXTGOV (Aug. 24, 2012), http://www.nextgov.com/cio-briefing/2012/08/white-house-overhauls-electronic-records-requirements/57651/?oref=govexec-todayni ("Federal

2014] PREDICTIVE CODING 227

universal move towards electronically stored information ("ESI"),the sheer amount of data now available compounds generic concernsregarding information and data production and thus poses greaterconcerns for litigation in the digital age."o To visualize the vastamount of data facing modern litigators and regulators, consider thisfact: as of "2011, the digital universe [had] expanded to over 1800exabytes, enough data to fill 57.5 billion 32GB Apple iPads.""

As an unsurprising result, electronic discovery ("e-discovery")represents a crucial but overwhelming part of litigation budgets.12 Fornumerical orientation, e-discovery costs surrounding discovery and

agencies have until the end of 2019 to adopt systems that store and manage all electronicrecords in formats that will keep them safe and searchable.... Agencies have until the endof 2016 to store all email in electronic formats and until Nov. 15, 2012, to appoint a seniorofficial responsible for beefing up their electronic records management programs . . .. ").

9. For a definition of electronically stored information ("ESI"), see supra note 6.10. See, e.g., Yip, supra note 4, at 595 ("[T]he rapidly increasing volume of ESI has

substantially increased the costs of e-discovery for producing parties."). Because of theseserious financial concerns, and their impact on litigation, Zubulake v. UBS WarburgL.L.C, 216 F.R.D. 280 (S.D.N.Y. 2003), and its progeny evolved to permit "shiftfing]some of the e-discovery costs from the responding party to the requesting party." Yip,supra note 4, at 595. Significantly for the purposes of this Comment, predictive coding andrelated technologies promise reduced costs for both parties. This holds potential both tofree parties from oppressive litigation by decreasing the financial burden even if shiftingdoes not occur and to encourage other parties to engage in frivolous suits, as even if thecourt shifts the financial burden of their requests, the overall cost of production is farlower than that of litigation historically. For a discussion of the current state of the lawregarding the allocation of e-discovery costs in litigation, see generally JacquelineHoelting, Note, Skin in the Game: Litigation Incentives Changing as Courts Embrace a"Loser Pays" Rule for E-Discovery Costs, 60 CLEv. ST. L. REV. 1103 (2013) and Emily P.Overfield, Note, Shifting the E-Discovery Solution: Why Taniguchi Necessitates a Declinein E-Discovery Court Costs, 118 PENN ST. L. REv. 217 (2013).

11. Yip, supra note 4, at 595 (emphasis added).12. See Hoelting, supra note 10, at 1105; see also Grossman & Cormack, supra note 5,

at 15 (defining electronic or e-discovery as "[tihe process of identifying, preserving,collecting, processing, searching, reviewing, and producing Electronically StoredInformation that may be [r]elevant to a civil, criminal, or regulatory matter").

NORTH CAROLINA LAW REVIEW

document production independently comprised an astounding "$2.8billion in 2009,"'1 with continued projected increases as "[t]he amountof electronically stored information in the United States doublesevery 18-24 months, and 90 percent of U.S. corporations are currentlyengaged in some kind of litigation." 4 In a case that eventuallypermitted the use of computer-assisted review, the original electronicrecords would have required "10 man-years of billable time" simplyto adequately locate relevant documents."

These astounding costs represent the irreconcilable chasmbetween traditional data review and the vast digital prowess of themodern era. In response, human ingenuity and the marketplacedeveloped a solution: computer-assisted review and predictive codingtechnologies." These advances promise fiscal benefits and increasedefficiency." In the aforementioned scenario, utilization of codingtechnologies would only require "two weeks to cull the relevant

13. E.g., Yip, supra note 4, at 595.14. Overfield, supra note 10, at 217, see also Dixon, supra note 6, at 36 ("[T]he RAND

report estimates that human review of documents as part of responding. to discoveryrequests consumes about 73 cents of every dollar spent on the production of ESI.").

15. Dixon, supra note 6, at 36-37 (citing Defendants' Memorandum in Support ofMotion for Protective Order Approving the Use of Predictive Coding at 5, GlobalAerospace Inc. v. Landow Aviation, L.P., Consolidated Case No. CL 61040 (Va. Cir. Ct.Apr. 9, 2012)).

16. For a helpful summary of the evolution of predictive coding beginning in 2008, seeCharles Yablon & Nick Landsman-Roos, Predictive Coding: Emerging Questions andConcerns, 64 S.C. L. REV. 633,637-38 (2013).

17. Ronni Solomon, Are Corporations Ready to Be Transparent and Share IrrelevantDocuments with Opposing Counsel to Obtain Substantial Cost Savings Through the Use ofPredictive Coding?, METROPOLITAN CORP. COUNS., Nov. 2012, at 26, 26, available athttp://www.metrocorpcounsel.com/pdf/2012/November/26.pdf(describing the potential forover a million dollars in savings by adopting predictive coding in Da Silva Moore v.Publicis Groupe, 287 F.R.D. 182 (S.D.N.Y. 2012)).

228 VYol. 93

PREDICTIVE CODING

documents" at a fraction of the cost for traditional methods.' As aresult, humans matched. the electronic revolution with potentialelectronic solutions--computer-assisted review technologies andpredictive coding-that hold the potential to revitalize legal reviewdespite massive increases in ESI.

B. Concepts and Terminology Surrounding Predictive Coding

Despite its rapid emergence and strong potential, predictivecoding brings with it a host of confusion and concerns for modernattorneys-ranging from gaining the technological expertisenecessary to understand its promises to learning how to apply the newtechnology. Even more simply, however, one must learn to speak thelanguage of these new technologies. This Comment provides a briefoverview of some of the essential terms necessary to understand thescholarship and debates surrounding predictive coding. However,parts of this terminology lack uniformity, and new technologicaladvances may change the terminology. As a result, interested readersmust dedicate constant self-study to the rapid revisions.19

As an initial matter, technology-assisted review ("TAR") andcomputer-assisted review ("CAR") are broad terms that encompass anumber. of technologies, including predictive coding as well as less

18. Dixon, supra note 6, at 36-37 ("[B]y use of predictive coding, it would take lessthan two weeks to cull the relevant documents at roughly 1/100 the cost of using humansto review every document in the database." (citing Protective Order Approving the Use ofPredictive Coding for Discovery at 10, Global Aerospace Inc. v. Landow Aviation, L.P.,Consolidated Case No. CL 61040 (Va. Cir. Ct. Apr. 23, 2012))).

19. For a comprehensive and helpful description of many terms surroundingcomputer-assisted review and predictive coding technologies, as well as the technicalprocess and slight distinctions between different technologies at this level, see generallyGrossman & Cormack, supra note 5.

2014] 229

NORTH CAROLINA LAW REVIEW

advanced but similar technologies, such as keyword searching.20

However, because there is no single lexicon governing computer-assisted review technologies, the terminology distinctions betweenTAR, CAR, and predictive coding technologies are not followed inmuch of the literature discussing and comparing predictive coding,technology-assisted review, and computer-assisted review.2' Thecommon example of a square and rectangle can be applied to clarifythe distinction here. While predictive coding-the so-calledsquare-is a type of computer- or technology-assisted reviewtechnology, CAR and TAR-the so-called rectangles-are broaderterms that include other less automated technologies as well. ThisComment seeks to define these terms for the reader as well as to usetheir most precise forms in order to alleviate confusion.

At its most technical, predictive coding is "[an industry-specificterm generally used to describe a Technology [or Computer]-AssistedReview process involving the use of a Machine Learning Algorithmto distinguish Relevant from Non-Relevant Documents, based onSubject Matter Expert(s)' Coding of a Training Set of Documents."22

In plain English, predictive coding matches human judgment andhands-on training with computer learning and iterative skill to teachsoftware to quickly and accurately search and categorize documents,

20. See generally, e.g., Sharon D. Nelson & John W. Simek, Predictive Coding: A Roseby Any Other Name, LAW PRAC., July-Aug. 2012, at 20 (arguing that all similar forms ofanalytical technology that have similar features but do not qualify as predictive codingtechnology are referred to as "TAR [technology-assisted review] or CAR [computer-assisted (or computer-aided) review]").

21. See, e.g., id. at 20; cf. Grossman & Cormack, supra note 5, at 13, 32 (showing theterminology overlap between content-based advanced analytics ("CBAA") andtechnology-assisted review, which also overlaps with computer-assisted review andpredictive coding).

22. Grossman & Cormack, supra note 5, at 26.

[Vol. 93230

2014] PREDICTIVE CODING 231

much like human-only review.2 3 Pandora Internet radio provides ahelpful analogy for this process: the computer program learns fromusers' positive or negative feedback-the equivalent of a Pandora"thumbs up" or "thumbs down" of a song-to predict future outputs,such as a desired song to play next, and the users' preferences.2 4 Forpredictive coding, the initial learning process occurs as humans code aprimary seed set of documents to teach the program what constitutesrelevancy and privilege for the overall document set; the training isthen repeated using sets of documents until the machine reaches apre-determined accuracy in self-categorizing documents."

A number of other terms are especially helpful for understandinghow predictive coding is described, how it works, and how to graspthe legal significance and potential for its technological components. 26

First, the repeated interactive process 27 between the machine software

23. See Yablon & Landsman-Roos, supra note 16, at 638-42.24. See About Pandora, PANDORA, http://www.pandora.com/about (last visited Nov.

19,2014).25. See Yablon & Landsman-Roos, supra note 16, at 638-40; see also Nelson & Simek,

supra note 20, at 22 (defining predictive coding as requiring all of the following:"Integrated, keyword-agnostic analytics to quickly generate accurate seed sets[;] Languageand keyword-agnostic machine-learning technology to accurate find relevant documentsduring the 'training' process[;] A sound and well-documented workflow[;] Integratedsampling to verify results to a statistical certainty before, during and after review[;] Acompletely integrated, purpose-built system to ensure results are consistent throughoutthe entire process, every time").

26. For a definition of predictive coding, see supra note 22 and accompanying text.27. This process is known as iterative training. See Grossman & Cormack, supra note

5, at 20 ("Iterative Training: The process of repeatedly augmenting the Training Set withadditional examples of Coded Documents until the effectiveness of the Machine LearningAlgorithm reaches an acceptable level. The additional examples may be identified throughJudgmental Sampling, Random Sampling, or by the Machine Learning Algorithm, as inActive Learning.").

232 NORTH CAROLINA LAW REVIEW [Vol. 93

and the human teachers is known as active learning.28 This processemploys a training set of documents coded by humans to teachpredictive coding programs how to evaluate the relevance of futuredata.29 Predictive coding' and similar technologies are able to learnfrom their human teachers because of their ability to "emulate humanjudgment"-a characteristic of their artificial intelligence.3' Amachine's ability to use its artificial intelligence to properly engage inthe learned coding process, known as machine learning," is measuredby its accuracy." Accuracy levels are determined by the program's

28. Id. at 8 ("Active Learning: An Iterative Training regimen in which the TrainingSet is repeatedly augmented by additional Documents chosen by the Machine LearningAlgorithm, and coded by one or more Subject Matter Expert(s).").

29. Id. at 32-33 ("Training Set: A Sample of Documents coded by one or moreSubject Matter Expert(s) as Relevant or Non-Relevant, from which a Machine LearningAlgorithm then infers how to distinguish between Relevant and Non-Relevant Documentsbeyond those in the Training Set."). Note also that the first training set is known as theseed set. See id. at 29.

30. Coding is a short-hand term that refers to the human and/or "automated" processof "[1]abeling a [diocument as Relevant or Non-Relevant." Id. at 11.

31. Id. at 9 ("Artificial Intelligence: An umbrella term for computer methods thatemulate human judgment. These include Machine Learning and Knowledge Engineering,as well as Pattern Matching (e.g., voice, face, and handwriting recognition), robotics, andgame playing.").

32. Id. at 22 ("Machine Learning: The use of a computer Algorithm to organize orClassify Documents by analyzing their Features. In the context of Technology-AssistedReview, Supervised Learning Algorithms (e.g., Support Vector Machines, LogisticRegression, Nearest Neighbor, and Bayesian Classifiers) are used to infer Relevance orNon-Relevance of Documents based on the Coding of Documents in a Training Set. InElectronic Discovery generally, Unsupervised Learning Algorithms are used forClustering, Near-Duplicate Detection, and Concept Search.").

33. Id. at 8 ("Accuracy: The fraction of Documents that are correctly coded by asearch or review effort. Note that Accuracy + Error = 100%, and that Accuracy = 100% -Error. While high Accuracy is commonly advanced as evidence of an effective search orreview effort, its use can be misleading because it is heavily influenced by Prevalence.Consider, for example, a Document Population containing one million Documents, ofwhich ten thousand (or 1%) are Relevant. A search or review effort that identified 100%

2014] PREDICTIVE CODING 233

responsiveness? Responsiveness, a measure of how well the programreturns relevant 5 documents, can be determined by the particularinformational or legal need at hand and is often based on theproportion36 of relevant documents returned versus non-relevantdocuments." In order to measure accuracy, users employ a control setof random documents to assess the sufficiency of the system's codingabilities at that point." Scholars often measure this machine-learningautomated process against pre-determined accuracy levels, to ensurequality control," as well as against the results that human reviewerswould achieve under manual review.' These comparisons allow

of the Documents as Not Relevant, and, therefore, found none of the RelevantDocuments, would have 99% Accuracy, belying the failure of that search or revieweffort."); cf. Ralph C. Losey, Predictive Coding and the Proportionality Doctrine: AMarriage Made in Big Data, 26 REGENT U. L. REV. 7, 21-24 (2013) (describing themachine-learning process, with a focus on predictive coding and the relevancy rankings).

34. Grossman & Cormack, supra note 5, at 28 ("Responsiveness: A Document that isRelevant to an Information Need expressed by a particular request for production orsubpoena in a civil, criminal, or regulatory matter.").

35. Id. ("Relevance / Relevant: In Information Retrieval, a Document is consideredRelevant if it meets the Information Need of the search or review effort.").

36. Id. at 26 ("Proportion: The fraction of a set of Documents having some particularproperty (typically Relevance).").

37. See Losey, supra note 33, at 21-24.38. Grossman & Cormack, supra note 5, at 13 ("Control Set: A Random Sample of

Documents coded at the outset of a search or review process that is separate from andindependent of the Training Set. Control Sets are used in some Technology-AssistedReview processes. They are typically used to measure the effectiveness of the MachineLearning Algorithm at various stages of training, and to determine when training maycease.").

39. Id. at 27 ("Quality Control: Ongoing methods to ensure, during a search or revieweffort, that reasonable results are being achieved."); see also id. ("Quality Assurance: Amethod to ensure, after the fact, that a search or review effort has achieved reasonableresults.").

40. Id. at 22 ("Manual Review: The practice of having human reviewers individuallyread and Code the Documents in a Collection for Responsiveness, particular issues,privilege, and/or confidentiality."); see, e.g., Losey, supra note 33, at 13-14 (describing the

NORTH CAROLINA LAW REVIEW

scholars not only to assess whether technology is sufficientlyadvanced to achieve an accuracy level sufficient for the micro-levelcoding tasks, but also to discuss on a macro policy level whether thistechnology is sufficiently advanced to complement, or even replace,traditional human review. While a number of more technical aspectsand terms surround predictive coding and the mathematical decisionsgoverning acceptable quality control levels, this explanation providesthe rudimentary lexicon necessary for a discussion of predictivecoding in the judicial and regulatory spheres.

II. PREDICTIVE CODING AND THE LEGAL WORLD

To elaborate on predictive coding's role in the legal world, thisPart proceeds with five main themes: (1) predictive coding's legalapplications; (2) solutions to problems posed by ESI and discussionby predictive coding's advocates; (3) attorneys' roles in relation topredictive coding; (4) predictive coding's main legal advantages; and(5) predictive coding's main disadvantages. Finally, it provides anintermediate conclusion to resolve these disparate pieces of acomplicated technology in the broad legal realm before discussingspecific case applications.

Predictive coding has numerous applications beyond the legalworld. However, within the legal world, it is often referred to inconnection with e-discovery, which is defined as "[tihe process ofidentifying, preserving, collecting, processing, searching, reviewing,and producing Electronically Stored Information [ESI] that may be[r]elevant to a civil, criminal, or regulatory matter."4' The FederalRules of Civil Procedure automatically deem ESI to be available as

traditional human review process and the challenges it faces in the ESI-era in a sectionthat provides a contrast to document review that employs predictive coding).

41. Grossman & Cormack, supra note 5, at 15.

[Vol. 93234

PREDICTIVE CODING

potential evidence in lawsuits,42 consequently rendering almost anymedium as fair game for discovery. 43 Because the discovery processwas created around traditional review mediums and processes,conventional methods unsatisfactorily address the electronic world.'

Predictive coding provides a solution to the problems createdwhen incorporating ESI into the legal arena: "(1) volume andduplicability, (2) persistence, and (3) dispersion."" For example, themultitude of emails produced in an investigation is compounded whenthe emails are produced in duplicates, as a new copy of a document isincluded for each sender or recipient. This unnecessarily requiresevaluating attorneys to classify the same document multiple times.With problems like these, which can also include assessing incoming

42. Nicholas Barry, Note, Man Versus Machine Review: The Showdown BetweenHordes of Discovery Lawyers and a Computer-Utilizing Predictive-Coding Technology, 15VAND. J. ENT. & TECH. L. 343, 346-47 (2013) ("An amendment to the FRCP [FederalRules of Civil Procedure] in 2006 explicitly made all electronic files discoverable. Theamended rules [, however,] did not make ESI discoverable for the first time; courts hadlong held that electronic files were discoverable even without a specific grant in therules.").

43. Id. at 347 ("[El-discovery has grown exponentially and now includes, inter alia,emails, word-processing files, spreadsheets, databases, video files, MP3 files, and virtuallyevery other file now stored on computers and other electronic devices (such as PDAs, cellphones, flash drives, DVDs, etc.)."). The new role for ESI also creates an enhanced rolefor "big data" in the legal profession. See Jobst Elster, Big Data for Law Firms: Hype,Reality, Myth, or Legend, LEGAL MGMT., Oct.-Nov. 2013, at 35, 37. See generallyTHOMSON REUTERS, 50 STATE STATUTORY SURVEYS: CIVIL LAWS: CIVIL PROCEDURE:ELECTRONIC DISCOVERY (2014), available at 0020 SURVEYS 4 (Westlaw) (providingcitations to forty-nine states' statutes and applicable federal statutes addressing electronicdiscovery).

44. See The Sedona Conference Working Group, supra note 7, at 198-99.45. Barry, supra note 42, at 347 (citing WORKING GRP. ON ELEC. DOCUMENT

RETENTION & PROD., THE SEDONA CONFERENCE, THE SEDONA PRINCIPLES: BESTPRACTICES RECOMMENDATIONS & PRINCIPLES FOR ADDRESSING ELECTRONICDOCUMENT PRODUCTION I (2d ed. 2007), available at https://thesedonaconference.org/download-pub/81).

2014] - 235

236 NORTH CAROLINA LAW REVIEW [Vol. 93

data, shifting production costs, or searching one's own documents forresponsive materials,' ESI's sheer financial bulk-document reviewcosts comprise "nearly 75 percent of the eDiscovery budget"4 7

renders predictive coding a necessary advancement in the modernlegal profession.48

Advocates of predictive coding champion its centrality torevitalizing an efficient modern legal practice. They often concedethat that the technology is not fully equivalent to human review,49instead arguing that predictive coding works best in mundanecontexts, characterized by the easy relevancy or privilegedeterminations on a large number of documents, whereas humans arebetter at making close calls.50 Despite this concession, a number ofarguments support predictive coding's efficacy in the legal profession.

46. See generally Craig D. Ball, About Predictive Coding: The "Not Me" Factor (ALl-ABA Continuing Professional Education, July 2013), WL CVOO1 ALI-ABA 573(discussing big data's significance for the legal profession in a variety of contexts, rangingfrom document production to firms' business development).

47. Elster, supra note 43, at 38.48. As the Sedona Conference is a very helpful source to get a sense of what practices

and technologies are used and should be used in this area, see generally The SedonaConference Working Group, supra note 7, for further research.

49. See Ball, supra note 46, at 575.50. See id. ("[Allthough predictive coding isn't better at dealing with the swath of

documents that demand careful judgment, it's every bit as good (and actually much, muchbetter) at dealing with the overwhelming majority of documents that don't require carefuljudgment-the very ones where keyword search and human reviewers fail miserably.").Ball goes on to argue that most documents under review are easy calls-either "obviouslyrelevant" or "obviously irrelevant"-making predictive coding a far more cost-effectiveand consistently efficient alternative in these contexts, while humans thrive with "thejudgment call documents." See id. at 575-76 (citations omitted). But see The SedonaConference Working Group, supra note 7, at 203 (describing "[r]esistance by [somemembers of] the [I]egal [p]rofession" who challenge predictive coding based on groundsincluding superior human capabilities, insufficient foundation for these technologies in thecourtroom, and simple ignorance about how to best use automated technologies to theirfull potential). The decision of when predictive coding should be employed, rather than

2014] PREDICTIVE CODING 237

Numerous factors-such as readability, the area(s) of law at issuein a particular piece of litigation, and the professional judgmentsmade by the attorneys who effectively teach the program a particularissue"-impact how well predictive coding works on differentdocuments and implicate its relative success in certain areas of law.52

The initial "teachers" are often "attorneys with knowledge about theresponsiveness of those documents."" While the coding programprovides the ultimate review for responsiveness, it learns from theattorneys who make the initial structural responsivenessdeterminations and classifications.54 However, the mechanical natureof the predictive coding programming still leaves room for humanjudgment. After the automated processes are designed in compliancewith the particular request, "attorneys must then decide whichdocuments to produce."" Attorneys' professional judgment is thennecessary to determine what to do with the mechanized data and thecorresponding documents: (1) manual review of sufficiently

when it can be employed, consequently remains an important area for discussion andresearch in the legal community. However, because it provides a meaningful analyticaltool in many cases, it is important to understand the overall potential that predictivecoding holds for the legal field in general before reviewing its current status with thejudiciary and administrative bodies.

51. See Yablon & Landsman-Roos, supra note 16, at 638 ("This shift [to technology-assisted review] was especially true in securities cases where relevant documents weremore readable-and less frequent in antitrust and intellectual property cases in which thedocument population was more technical and varied.").

52. See id. at 637-38 ("The use of technology-assisted review began around 2008....The underlying technology, called machine learning, had been available for decades, butonly in about the last five years has the legal profession considered its use.... This shiftwas especially true in securities cases where relevant documents were more readable-andless frequent in antitrust and intellectual property cases in which the document populationwas more technical and varied." (citation omitted)).

53. Id.54. Id. at 639-40.55. Id. at 641.

NORTH CAROLINA LAW REVIEW

responsive documents;5 6 (2) a combination of production, culling, andreview depending on relevancy benchmarks;s" or (3) production andculling based solely on relevancy benchmarks.ss More lawyers willsoon face these choices as predictive coding gains increased tractionamong judges, agencies, and the profession.

For instance, pre-existing legal doctrines may push towards wide-scale implementation of predictive coding protocols due to the .fiscalbenefits available from implementation. Under the doctrine ofproportionality, parties are excused from retrieving and sharing ESIshould it not be cost-effective." Because predictive codingsignificantly reduces the cost of e-discovery," it renders ESI moreaccessible to litigants. The cost savings provide positive and negativeimpacts for a variety of parties: (1) it may cause businesses to releaseunfavorable data in response to disclosure or discovery requests, butthat same data may help challengers who would otherwise be stymiedby informational inequities; (2) it may allow businesses to complywith stringent regulatory requirements at lower costs to the bottom-line; and (3) it may simply revolutionize all parties' access to the

56. This most closely maintains the traditional approach to ESI: "thlistorically, themost commonly used discovery search process involves human reviewers looking at eachitem of ESI to determine whether it is relevant and, if so, whether it is privileged." DavidJ. Waxse & Brenda Yoakum-Kriz, Experts on Computer-Assisted Review: Why FederalRule of Evidence 702 Should Apply to Their Use, 52 WASHBURN L.J. 207, 208 (2013).

57. See Yablon & Landsman-Roos, supra note 16, at 641-42.58. See id.59. See Grossman & Cormack, supra note 5, at 26-27 ("Proportionality: Pursuant

to Federal Rules of Civil Procedure 26(b)(2)(B), 26(b)(2)(C), 26(g)(1)(B)(iii), and otherfederal and state procedural rules, the legal doctrine that Electronically StoredInformation may be withheld from production if the cost and burden of producing itexceeds its potential value to the resolution of the matter. Proportionality has beeninterpreted in the case law to apply to preservation as well as production."); see alsoYablon & Landsman-Roos, supra note 16, at 663-72 (addressing "predictive coding andproportionality review").

60. See supra notes 12-18 and accompanying text.

238 [Vol. 93

PREDICTIVE CODING

amount of information available-either by promoting negotiation orby rendering more cases more suitable for trial due to the influx ofevidence.6 1 Intuitively, predictive coding helps producing partiessatisfy the opposing sides' requests at a decreased cost. 62 Thesesavings are particularly helpful when agencies request documents, ascompanies, and possibly individuals, wish to fully comply with therequests at a minimal fiscal cost. 63 Scholars also recognize, however,that automated learning and coding technologies "can be equallyvaluable for analyzing incoming document productions" because theyallow for an incoming review triage: the protocols "rank documentsby degree of responsiveness so attorneys can home in on the mostimportant documents quickly."' Consequently, the concept ofproportionality does not just change whether predictive coding can be

61. See The Sedona Conference Working Group, supra note 7, at 198-99 ("Lawyersof all stripes therefore have a vital interest in utilizing automated search and retrieval toolswhere appropriate. The plaintiff's bar has a particular interest in being able to efficientlyextract key information received in mammoth 'document' productions, and in automatedtools that facilitate the process. The defense bar has an obvious interest in reducingattendant costs, increasing efficiency, and in better risk-management of litigation(including reducing surprises). All lawyers, clients, and judges have an interest inmaximizing the quality of discovery, by means of using automated tools that produce areliable, reproducible and consistent product."); cf. Solomon, supra note 17, at 26(discussing three cases in which the parties were willing to "share irrelevant documentsduring the predictive coding training process to achieve cost savings").

62. Solomon, supra note 17, at 26.63. See, e.g., Matthew Nelson, Predictive Coding & the "Risk-A verse" Attorney: Top 3

eDiscovery & Compliance Use Cases (Part 2), CORPORATE COMPLIANCE INSIGHTS (July2, 2013), http://www.corporatecomplianceinsights.com/predictive-coding-the-risk-averse-attorney-top-3-ediscovery-compliance-use-cases-part-2/. Interestingly, while parties mayhave negotiated alternatives in litigation and companies may be able to use discovery costsas a tool, individuals and companies are at a relative bargaining disadvantage withgovernment agencies, which hold greater leverage.

64. Id.

2014]1 239

NORTH CAROLINA LAW REVIEW

mandated;6 s rather, it also impacts who has access to the fruits bornfrom this technology.

Those convinced by the advantages of predictive coding mustface another large hurdle: determining the accuracy and effectivenesslevels that predictive coding must achieve before it gains wideracceptance. The legal field promotes cautious but vigorous advocacyfor clients.66 Before fully adopting these relatively newmethodologies, the legal field must be certain that predictive codingrepresents an improvement on or, at the very least, a supplement tocurrent practices. A number of studies support the assertion thatpredictive coding methodologies are at least equal to, if not moreaccurate than, traditional human review." Should these studies provean acceptable foundation for the profession's ethical standards,predictive coding may gain a substantial foothold in the future of thelegal profession.

Overall, given the symbolic Mount Everest posed by the ever-growing amount of ESI in all legal spheres, predictive coding providesthe metaphorical climbing poles and oxygen allowing attorneys totrek to the top "to assess cases faster and more efficiently[,] mak[ing]

65. Nevertheless, the potential for mandatory predictive coding in the judicial contextis a necessary theme in current jurisprudence and this Comment. See infra Part III.

66. See MODEL CODE OF PROF'L RESPONSIBILITY pmbl. 11 2,5 (2012).67. See Maura R. Grossman & Gordon V. Cormack, Inconsistent Responsiveness

Determination in Document Review: Difference of Opinion or Human Error?, 32 PACE L.REV. 267, 267-68 (2012); Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient ThanExhaustive Manual Review, 17 RICH. J.L. & TECH. 11 1 61 (2011) [hereinafter Grossman& Cormack, Technology-Assisted Review]; Herbert L. Roitblat, Anne Kershaw, & PatrickOot, Document Categorization in Legal Electronic Discovery: Computer Classification vs.Manual Review, 61 J. AM. SOC'Y FOR INFO. SCi. & TECH. 70, 79 (2010); Ellen M.Voorhees, Variations in Relevance Judgments and the Measurement of RetrievalEffectiveness, 36 INFO. PROCESSING & MGMT. 697,714-15 (2000).

[Vol. 93240

2014] PREDICTIVE CODING 241

case preparation easier, more comprehensive, and, less expensive."IWhether it is employed in the courtroom6 9 or when interacting withadministrative agencies,70 predictive coding is a new path fortraditional data management and dispute resolution. What is not anoption, at this point, is simply ignoring the data mountain in the room,as ESI grows exponentially and automated technologies increase insignificance for all attorneys."

Proponents of court-based acceptance of predictive coding pointto traditional principles, such as the Federal Rules of Civil Procedure'semphasis on "balanc[ing] costs and completeness" in* discovery, insupport of their claims.72 Predictive coding promises a sharp increasein these existing values as it promotes both efficiency and fiscalresponsibility. As such, it promises a level of continuity by pursuing

68. Nelson, supra note 63.69. See infra Part III.70. See infra Part IV.71. See, e.g., Nelson & Simek, supra note 20, at 20 ("There is a great quote from

a Forbes blog post by Barry Murphy that indicates why all lawyers need to understand abit about predictive coding: 'A lawsuit can really knock a company for a loop. Imaginebeing sued and asked to produce all responsive information, only to find that means siftingthrough 10 TB of emails. The process is complicated and it can be very costly. After all,the company must somehow determine with confidence whether each and every one ofthose emails is relevant to the lawsuit and/or subject to attorney-client privilege. Thisprocess has become much more manageable using technology to assist the reviewprocess.' " (quoting Barry Murphy, 2012: The Year Of Technology-Assisted Review IneDiscovery, FORBES (Jan. 17, 2012, 2:12 PM), http://www.forbes.com/sites/barrymurphy/2012/01/17/2012-the-year-of-technology-assisted-review-in-ediscovery/)).

72. Barry, supra note 42, at 365 ("Predictive coding can meet the FRCP standard ifparties can show it is more accurate, efficient, and responsive than manual review." (citingGrossman & Cormack, Technology-Assisted Review, supra note 67, 5)).

73. See supra notes 17-18 and accompanying text; see also, e.g., Barry, supra note 42,at 365-66 ("The [TREC Legal Track] study concluded that 'technology-assisted reviewcan achieve at least as high recall as manual review, and higher precision, at a fraction ofthe review effort, and hence, a fraction of the cost.' " (quoting Grossman & Cormack,Technology-Assisted Review, supra note 67, 55).

NORTH CAROLINA LAW REVIEW

traditional judicial ideals. Advocates include Judge Andrew Peck,7who has asserted that, in his "opinion, computer-assisted codingshould be used in those cases where it will help 'secure the just,speedy, and inexpensive' determination of cases in our e-discoveryworld.""

At the same time, however, proponents and critics alikerecognize the fundamental choices inherent in the move from manual,human-based review to artificial intelligence-based review throughthe use of automated predictive coding processes."6 It is in thenecessary re-education and re-orientation process that the link to thisComment's thesis lies: because predictive coding complementsexisting judicial and litigation values, including efficiency, andbecause it will require an intellectual overhaul, it is very important tostudy current case law trends in addition to administrative and agencyresponses. Studying these trends will determine if the future ofpredictive coding supports existing legal values and, if it does not, willguide what changes can be made during this re-shaping of the legalconsciousness to ensure that the new framework supports healthydevelopment in accordance with those generally accepted principles.

III. PREDICTIVE CODING AND RECENT CASES

Recent cases demonstrate judicial experimentation as courts gaina tentative footing in how to address and incorporate predictive

74. Judge Peck presided over the seminal predictive coding case, Da Silva Moore v.Publicis Groupe, 287 F.R.D. 182 (S.D.N.Y. 2012), discussed in infra Part III.A.

75. Andrew Peck, Search, Forward, LAW TECH. NEWS, Oct. 2011, at 25, 29 (quotingFED. R. Civ. P. 1).

76. Barry, supra note 42, at 365 ("The transition to newer, more efficient, and moreaccurate models of document review will require the legal field, as a whole, to undergo aprocess of education.").

242 [Vol. 93

PREDICTIVE CODING

coding in existing discovery and litigation norms.n Thisexperimentation with predictive coding provides more than simplyuseful data points that demonstrate the significance of ESI. Rather,these cases show that predictive coding can and does work formodern legal issues and that courts are responding to these issuesthrough traditional norms intertwined with an embrace ofrevolutionary technology." To that end, this Part discusses key recentcases to show not only that predictive coding has been used anddiscussed by the judiciary, but also to demonstrate with meaningfulexamples that case law is creating an initial framework through whichto evaluate and respond to predictive coding. Finally, this Part usesthe review of these cases to get a sense of how that frameworkcompares to similar responses from regulatory agencies in this time oflegal and technological flux.

A. Da Silva Moore v. Publicis GroupeThe Southern District of New York's 2012 decision in Da Silva

Moore v. Publicis Groupe" arguably represents predictive coding'smost well known foray into case law. The gender discrimination casegave rise to a class action suit,' which provided data ripe for codingtechnologies. The plaintiffs' case rested on allegations that there was

77. Am. Bar Ass'n, Computerized Review on Trial, A.B.A. J., Apr. 2013, at 30, 30(2013) ("[W]e return ... with a study showing [that] TAR and predictive coding aregetting increased attention in America's court system. Kroll Ontrack's 2012 analysis of 70state or federal judicial opinions affecting electronically stored information ... found ninepercent of the total opinions discuss[ed] either predictive coding ... or TAR.").

78. See, e.g., id. (illustrating the advocacy by proponents, some of whom claim that"many notable e-discovery trends emerged ... but none promise[] to change the statusquo more than the line of opinions approving the use of technology-assisted review"(internal quotation marks omitted)).

79. 287 F.R.D. 182 (S.D.N.Y. 2012).80. Id. at 183.

2014]1 243

NORTH CAROLINA LAW REVIEW

" 'systemic, company-wide gender discrimination against female PRemployees.' "I' Because this employment complaint was based not onthe actions of a select group of individuals, but rather on acomprehensive practice, a number of electronic documents anddata-including hiring and promotion practices along gender lines,email communications between employees and supervisors,communications regarding promotions, and any companypolicies-comprised the relevant discovery material.8 2 At the initialstages of this case, the parties sought to address the " 'electronicdiscovery protocol' ... [for] approximately three million electronicdocuments... ."8

Judge Peck, who served as the magistrate judge in Da SilvaMoore and whose scholarship addresses emerging litigationtechnology, recognized the potential that predictive coding andrelated technologies possess for enhancing the efficiency of reviewand production processes during discovery.' His discussionhighlighted that the technologies allow companies to fully respond tothe plaintiff's legal requests while engaging in a timely and cost-feasible retrieval process." In this early-stage decision, Judge Peckalso acknowledged the numerous legal challenges inherent inintroducing a new technology to court practice, including the role

81. Id. (quoting Complaint at 3, Da Silva Moore, 287 F.R.D. 182 (No. 11-cv-01279),2011 WL 655226).

82. See id. at 183-84.83. See id. at 184 (quoting Transcript of Dec. 2, 2011 Trial Conference at 7, Da Silva

Moore, 287 F.R.D. 182 (No. 11-cv-01279), available at http://www.itlawtoday.com/files/2014/05/DaSilvaMoorePublicisTR_12-2-1 1.pdf).

84. See id. at 182-83; see also Peck, supra note 75, at 26, 29.85. See id. at 183, 189-91, 193 (noting that computer-assisted discovery technologies

are especially useful when there is a large amount of data and the parties want tomaximize cost and time effectiveness).

244 [Vol. 93

PREDICTIVE CODING

Daubert v. Merrell Dow Pharmaceuticals6 would play in theadmissibility of evidence derived from this new, expert-driventechnology." Finally, he advocated a uniform manner in addressinghow parties should signal their intent to use predictive coding in aparticular case: "The best approach-.. is to follow the SedonaCooperation Proclamation model ... [and] [a]dvise opposing counselthat you plan to use computer-assisted coding and seek agreement; ifyou cannot, consider whether to abandon predictive coding for thatcase or go to the court for advance approval.""

By layering his academic suggestions into a judicial opinion,Judge Peck provided a logical bridge between a previouslyhypothetical and purely academic discussion-predictive coding'srole, if any, in the courtroom-and the day-to-day efficiency andevidentiary challenges faced by modern judges. He demonstratedmore than just the promise of predictive coding, which his academicresearch had already elucidated. Rather, through this opinion, JudgePeck provided a very real and practical example of coding'sapplications9 and moved predictive coding from academic intrigueinto the ever-growing array of litigators' tools.

86. 509 U.S. 579 (1993). For the remaining prongs of the Daubert standard, seeKuhmo Tire Co. v. Carmichael, 526 U.S. 137, 147-53 (1999) and Gen. Elec. v. Joiner, 522U.S. 136, 142, 146 (1997).

87. Da Silva Moore, 287 F.R.D. at 184. For a more thorough analysis surrounding howpredictive coding technologies implicate the Federal Rules of Evidence and Daubert, aswell as an argument in favor of folding coding technologies into the existing frameworksfor emerging technologies under the evidentiary rules, see Waxse & Yoakum-Kriz, supranote 56, at 207 ("This article examines [Federal] Rule [of Evidence] 702 and concludesthat it and the Daubert standard should be applied to experts who testify or otherwiseprovide evidence before the court on discovery disputes involving these ESI searchmethods.").

88. Da Silva Moore, 287 F.R.D. at 184 (quoting Peck, supra note 75, at 29).89. In light of this practicality, Judge Peck noted that while the technology is now a

viable option in some cases, it is "not going to be perfect. [Rather,] [t]he idea is to make it

2014] 245

NORTH CAROLINA LAW REVIEW

Da Silva Moore "recognize[d] that computer-assisted review isan acceptable way to search for relevant ESI in appropriate cases."'When accepting predictive coding on the facts of the case, Judge Peckoutlined five main factors in support of his decision:

(1) the parties' agreement, (2) the vast amount of ESI to bereviewed (over three million documents), (3) the superiority ofcomputer-assisted review to the available alternatives (i.e.,linear manual review or keyword searches), (4) the need forcost effectiveness and proportionality under [Federal] Rule [ofCivil Procedure] 26(b)(2)(C), and (5) the transparent processproposed by [the defendant]."

He also recognized Da Silva Moore's significance and the"lessons [it held] for the future."' They included the importance of''cooperation among counsel," the need to train the automatedprograms and to validate the results through quality controlprocesses, the importance of triaging document review based onrelevancy in order to minimize costs, and the need for activeparticipation in "court hearings" by "the parties' e[-]discoveryvendors."" All of these individual "lessons" illustrate the legalimplications courts need to address in order to embrace the ready andpractical opportunities presented by predictive coding.9 4

significantly better than the alternatives without nearly as much cost." Id. at 187 (internalquotation marks omitted).

90. Id. at 183 (footnote omitted).91. Id. at 192.92. Id.93. Id. at 192-93.94. Id. at 193 ("What the Bar should take away from this Opinion is that computer-

assisted review is an available tool and should be seriously considered for use in large-data-volume cases where it may save the producing party (or both parties) significantamounts of legal fees in document review. Counsel no longer have to worry about beingthe 'first' or 'guinea pig' for judicial acceptance of computer-assisted review.").

[Vol. 93246

PREDICTIVE CODING

Interestingly, even Judge Peck, a strong advocate ofincorporating coding technologies, recognized in initial discoveryconferences that predictive coding, at least in its current form, is notan automatic answer for electronic discovery in all cases.95 JudgePeck's approach, in the parties' initial conference, signaled a need toweigh the benefits and costs of technology, particularly in its untestedstate, and to determine what forms should be mixed and matched tocreate the most appropriate option in a particular case. 96 The issuesraised in this case exemplify areas of the law that are and willcontinue to be subject to tension when incorporating predictivecoding technologies into "typical" litigation. For instance, eventhough predictive coding makes vast amounts of data relatively more'accessible, it does not justify carte blanche judicial access to alldocument caches. In Da Silva Moore, the parties had to agree onwhat data sources would be searched and subjected to the automatedprotocol." Additionally, even though these parties shared a generalagreement about predictive coding and the necessary confidencelevels,98 they disagreed on next steps and, in particular, how thetrained system should be used, where the cut-off for manual reviewshould lie, and how that should be balanced with the potential for

95. Id. at 185 (" '[Piredictive coding should be used in the appropriate case. Is this theappropriate case for it? You all [should] talk about it some more. And if you can't figure itout, you are going to get back in front of me [, Judge Peck].' " (quoting Transcript of Dec.2, 2011 Trial Conference at 20, Da Silva Moore, 287 F.R.D. 182 (No. 11-cv-01279),available at http://www.itlawtoday.com/files/2014/05/DaSilvaMoorePublicisTR_12-2-11.pdf)).

96. Cf. id. ("Is this the appropriate case for [predictive coding]? . .. Key words,certainly unless they are well done and tested, are not overly useful. Key words along withpredictive coding and other methodology, can be very instructive." (internal quotationmarks omitted)).

97. Id. at 185-86.98. The parties established "a 95% confidence level (plus or minus two percent) to

create a random sample of the entire email collection." Id. at 186.

2014] 247

248 NORTH CAROLINA LAW REVIEW [Vol. 93

responsive, but unproduced, documents. 9 This dilemma presentedthe question of whether "document production is [truly] completeand correct as of the time it was made."" In a concrete application ofhis academic scholarship, Judge Peck also resolved the evidentiaryrules and Daubert issues presented by these concerns and determinedthat the rules would not apply to the actual search methodology butwould remain relevant if particular pieces of evidence are presentedat trial' 01

This initial acceptance signals an expedition among the judicialcommunity into a new world of predictive coding.'02 Previously,judges were certainly aware of its existence, but had not providedguidance on its admissibility or potential value in the judicialsystem. 03 For now, the intellectual revolution remains in its infancy,

99. Da Silva Moore, 287 F.R.D. at 185 ("Rather, plaintiffs took issue with[defendant's] proposal that after the computer was fully trained and the results generated,[defendant] wanted to only review and produce the top 40,000 documents, which itestimated would cost $200,000 (at $5 per document). The Court rejected [defendant's]40,000 documents proposal ... [and] explained that 'where [the] line will be drawn [as toreview and production] is going to depend on what the statistics show for the results,' since'Iplroportionality requires consideration of results as well as costs. And if stopping at40,000 is going to leave a tremendous number of likely highly responsive documentsunproduced, [the proposed cutoff] doesn't work.' " (alterations to internally quotedmaterial in original) (citations omitted)).

100. Id. at 188 (citation omitted) (internal quotation marks omitted) (addressingFederal Rule of Civil Procedure 26(b)-(c)).

101. Id. at 189 ("The admissibility of specific emails [or other pieces of ESI] at trial willdepend upon each email [or item] itself ... , not how it was found during discovery. Rule702 and Daubert simply are not applicable to how documents are searched for and foundin discovery."). Judge Peck went on to state that questions regarding relevancy are bestdecided at a later date. Id.

102. Id. at 191 ("Computer-assisted .review appears to be better than the availablealternatives, and thus should be used in appropriate cases.").

103. Id. at 182-83 ("To my knowledge, no reported case (federal or state) has ruled onthe use of computer-assisted coding.... Until there is a judicial opinion approving (oreven critiquing) the use of predictive coding, counsel will just have to rely on this article as

PREDICTIVE CODING

and judges, proponents, and critics of coding alike must recognize theactive role they play in shaping the future of automated technologiesin the legal system.

B. Cases Following in Da Silva Moore's Footsteps

A number of cases followed the predictive coding trail begun inDa Silva Moore. One such case, decided mere months later, is GlobalAerospace Inc. v. Landow Aviation, L.P." Extending Da SilvaMoore's initial permit for predictive coding, Global Aerospaceanswered the next logical question: If one party is opposed to the useof predictive coding, can and, arguably, should a court mandate itsuse anyway? 1 s As a matter of positive law, Global Aerospacemandated "the use of predictive coding for purposes of the processingand production of electronically stored information," while reservingthe opposing party's right to later object to the coding methodologyitself.1" However, the normative question-should a court force thisnew and unfamiliar methodology on parties who may, either forpersonal, strategic, or intellectual reasons, prefer manual review-remains unanswered. Judges, and perhaps legislatures, must address

a sign of judicial approval." (quoting Peck, supra note 75, at 29) (internal quotation marksomitted)).

104. No. CL 61040, 2012 WL 1431215 (Va. Cir. Ct. Apr. 23, 2012).105. A case in the northern district of Illinois also touched on these issues. See Kleen

Prods. L.L.C. v. Packaging Corp. of Am., No. 1:10-cv-05711, 2012 WL 4498465, at *5-6(N.D. Ill. Sept. 28, 2012) (noting an attempt by the plaintiffs to compel the use of content-based advanced analytics to find responsive documents).

106. Global Aerospace, 2012 WL 1431215, at *1 ("[I]t is hereby ordered Defendantsshall be allowed to proceed with the use of predictive coding .... This is without prejudiceto a receiving party raising with the court an issue as to completeness or the contents ofthe production or the ongoing use of predictive coding.").

2014] 249

250 NORTH CAROLINA LAW REVIEW [Vol. 93

this normative issue because predictive coding is now a potentialpiece of litigation and discovery."o

Only a few months later, In re Actos (Pioglitazone) ProductsLiability Litigation'" arrived. It too built upon the shift towardspredictive coding technologies. First, Da Silva Moore established thata party could use predictive coding.'" Then, Global Aerospaceshowed that a court could require a dissenting party to engage inpredictive coding-based discovery." 0 Finally, Actos demonstrated howpredictive coding worked in discovery."' Actos presentedfoundational and technical practices helpful to parties seeking toimplement predictive coding in a case management order concerningESI production." 2 These foundational practices included presentingthe sources and the likely custodians that would have been helpful toachieving a comprehensive data foundation as part of the casemanagement process." 3

Most significantly, however, Actos provided a summary of its"search methodology proof of concept to evaluate the potential utility

107. A parallel issue can arise with respect to regulatory agencies-if an agency beginsto allow companies to utilize predictive coding with respect to document requests, can orshould that agency require that all companies do so? A requirement to use predictivecoding would provide fiscal benefits and increase overall efficiency. However, it raisessimilar normative concerns as well as potential delegation issues. See infra Part IV.Essentially, the question of whether predictive coding can have an impact on legal mattersis already answered-it has arrived and can be tailored to legal concerns. However, justbecause the potential has been recognized does not mean that predictive coding should beutilized in every situation-rather, policy makers and players in each realm need toevaluate predictive coding in light of the unique considerations of each context to seewhether, even though it is a possible answer, it is the right one.

108. MDL No. 6:1 1-md-2299, 2012 WL 7861249 (W.D. La. July 27, 2012).109. Da Silva Moore v. Publicis Grp., 287 F.R.D. 182, 191 (S.D.N.Y. 2012).110. Global Aerospace, 2012 WL 1431215, at *1.111. Actos, 2012 WL 7861249, at *1-4.112. Id. at *1.113. See id. at *1-3.

PREDICTIVE CODING

of advanced analytics."ll 4 This methodology, essentially laying outhow these parties would employ predictive coding for electronicdiscovery, presents both an example of the parties' initial agreementon the details and provides a salient picture of the technicalities thatneed to be resolved by courts-whether there is, or should be, anofficial methodology adopted by the judiciary or agencies. Literaturespawned in the aftermath of these opinions urged judges to developunified and defined methodologies for automated technologies."' Inthis way, like the previous cases, Actos plays a dual role as an exampleof and as a spur to future judicial action. Each role occupies a rung inthe logical ladder pushing predictive coding from academichypotheticals to industry possibility to legal reality to, ideally, a fullyand adequately integrated part of the discovery and litigationschemes.

As a final point, the line between governmental uses orresponses to predictive coding and its status in case law is not as clearas it may seem. For instance, National Day Laborer OrganizingNetwork v. U.S. Immigration and Customs Enforcement Agency" 6

involved a Freedom of Information Act ("FOIA") request fordisclosure of information from the United States Immigration andCustoms Enforcement Agency."' Under FOIA requests, courtsdetermine whether a search is adequate by evaluating "the methods

114. Id. at *3.115. See, e.g., Elle Byram, The Collision of the Courts and Predictive Coding: Defining

Best Practices and Guidelines in Predictive Coding for Electronic Discovery, 29 SANTACLARA COMPUTER & HIGH TECH. L.J. 675, 693-701 (2013) (arguing that "the casesreveal the disagreement and uncertainty that exists for determining when and how[automated technologies] should be used" and that "[cilarifying standards will assistparties in reaching agreement earlier in the case and more easily, allowing for discovery toproceed more smoothly with less court interference").

116. 877 F. Supp. 2d 87 (S.D.N.Y. 2012).117. Id. at93.

2014] 251

NORTH CAROLINA LAW REVIEW

used to carry out the search.""' Consequently, the court's willingnessto endorse automated technologies represents an important steptowards the normalization of predictive coding in FOIA cases."' Thecourt's decision presents, like before, the normative question ofwhether these technologies should be endorsed in this situation.Additionally, if the parties meet the normative requirements, thedecision requires the secondary evaluation of predictive coding'smethodological features to ensure that they are sufficient to meet theFOIA search burden faced by the agency. As such, this judicialobservation about machine learning creates a signal for those whointerface with government agencies under FOIA, suggesting that thefuture will likely involve automated systems as a natural feature ofagencies' data retrieval' 20

By showing how agencies may need to use predictive coding onthe production-side, rather than receipt-side, of ESI production anddiscovery, National Day Laborer illustrates the overlapping nature ofpredictive coding for government agencies, which are impacted both

118. Id. at 96.119. Id. at 109 ("[Bleyond the use of keyword search, parties can (and frequently

should) rely on latent semantic indexing, statistical probability models, and machinelearning tools to find responsive documents.... [Tihese methods (knownas... 'predictive' coding) allow humans to teach computers what documents are and arenot responsive to a particular FOIA or discovery request and they can significantlyincrease the effectiveness and efficiency of searches." (emphasis added) (footnoteomitted)).

120. Maureen E. O'Neill, Obtaining ESI from the Government: Legal and PracticalGuidance from National Day Laborer Organizing Network, Presentation at the 2013National Conference on EEO Law 6 (2013), available at http://www.americanbar.org/content/dam/abalevents/Iabor_1awl2013/04/nat-conf-equal-empl-opp-law/13_oneill.authcheckdam.pdf ("For parties seeking information from the government, takeheed of Judge Scheindlin's directive to work cooperatively with the agency to deviseeffective, efficient ways to find the documents you seek. Keep in mind that somegovernment agencies are beginning to incorporate predictive coding... into their routinediscovery programs. "

[Vol. 93252

PREDICTIVE CODING

in the courtroom and in their regulatory duties. The overlap points toa logical necessity: courts and agencies should consider each otherwhen creating the schemes in which to envelop predictive coding.Their uses and concerns are not incompatible, and each will be betterserved when making the difficult choices necessary to develop thistechnology properly by learning from the other's mistakes andsuccesses. This would allow for the creation of a framework bestpoised for the development of a legal system that maximizespredictive coding's potential while avoiding its dangers, includingoverreliance on technology and frameworks that misunderstand howthe technology fits with existing legal obligations.

C. Progressive Casualty Insurance v. Delaney: Enforcing JudicialPredictive Coding Norms

In the years since the initial predictive coding cases, courts haveissued decisions focusing on the pragmatic concerns that continue toilluminate how they have addressed predictive coding in discovery.One such example is Progressive Casualty Insurance v. Delaney,12 1

which, similarly to National Day Laborer, involved private parties in asuit involving a government agency.122 In this case involvingunderlying causes of action based on banks taken over by theFDIC,123 the parties agreed to use keyword searching, a computer-

121. No. 2:11-cv-00678-LRH-PAL, 2014 WL3563467 (D. Nev. July 18,2014).122. This Comment points out that the respective positions of agencies and courts on

predictive coding may interact in cases like this and National Day Laborer, thus furtherpromoting the need for collaboration and transparency to create a more efficient overalldynamic. On the other hand, the purpose of this discussion of Progressive CasualtyInsurance is to show how a court reacted to the behind-the-scenes use of predictive codingwithout prior approval. However, see infra Part IV for a discussion of agencies' similarrequirements of up-front approval before parties may employ coding technologies in dataretrieval.

123. Progressive Cas. Ins. Co., 2014 WL 3563467, at *1.

2014]1 253

NORTH CAROLINA LAW REVIEW

assisted review technology, to cull the 1.8 million potentiallyresponsive documents to a supposedly manageable mass of 565,000documents to review for responsiveness.124 Progressive, the reviewingparty, facing efficiency and fiscal concerns, decided to further reviewthe documents using predictive coding.'25 Significantly, it made thisdetermination "without seeking leave of the court to amend the[parties' previously agreed upon] ESI Order," 26 or even informingthe opposing party of its intention.127 In addition to making thesedeterminations without consulting the court or opposing party,Progressive also violated the agreed-upon ESI Order protocol byfailing to produce the responsive documents "on a rolling basis." 28

In response to this behavior and corresponding motions by theopposing party, the court discussed predictive coding in theaggregate,'129 describing it as an "accurate means of producingresponsive ESI in discovery," particularly as compared to "ineffectivetools" like "manual human review[] or keyword searches... ."'" Inthis discussion, the court highlighted that it is an adherent to thepotential for predictive coding in certain ESI cases--even noting thatif parties "agree[] at the onset of this case to a predictive coding-based ESI protocol, [it] would not hesitate to approve a transparent,mutually agreed upon ESI protocol."'"' This juxtaposition with the

124. Id. at *2, *6.125. Id. at *2.126. Id.127. Id. at *4 ("In this case, Progressive unilaterally developed the predictive coding

methodology and implemented it without input or consultation from the FDIC-R'scounsel.").

128. Id. at *7.129. See id. at *8.130. Id. The court also cited a number of publications to support the assertion that

"[s]tudies show [that predictive coding] is far more accurate than human review orkeyword searches[,] which have their own limitations." Id.

131. Id. at *9.

254 [Vol. 93

PREDICTIVE CODING

court's hypothetical support for the use of predictive coding highlightsthe significance of its disapproval of what happened in this case. Here,the parties only agreed to search term or manual review for documentretrieval.13 2 In emphasizing the problem of unilateral action in thiscase, the court pointed to literature showing the judicial trendtowards a need for "unprecedented .. . transparency and cooperationamong counsel" when using computer-assisted review since judges"typically ... give deference to a producing party's choice of searchmethodology and procedures in complying with discoveryrequests."' The problem in this case was not that predictive codinghad been used; rather, the problem was that Progressive used thetechnology without adhering to the necessary cooperative andtransparency requirements "for a predictive coding protocol to beaccepted by the court ... as a reasonable method to search for andproduce responsive ESI."34 In response to Progressive's violation ofthese new predictive coding norms, the court required it to turn overthe entirety of the 565,000 documents originally located throughkeyword searching, subject only to privilege restrictions.' This harshresult illustrates how seriously the court took the need for above-board behavior when employing coding technologies in discovery.

Significantly, the court showed a remarkable acceptance ofpredictive coding as something that could be appropriate and evenviewed positively, as compared to other review mechanisms, in ESIcases. However, this acceptance came with strict methodologicalstrings attached, emphasizing a focus on cooperation andtransparency. Interestingly, although courts and agency responseshave developed separately, these same strands appear in regulatory

132. Id.133. Id. at *10.134. Id. at *11.135. Id. at *11-12.

2014] 255

256 NORTH CAROLINA LAW REVIEW [Vol. 93

agencies' limited publications.'36 Because similar themes emerge ineach context, increased cooperation between courts and .agenciesregarding the specifics-such as how much transparency is needed(e.g., should a party only show its methodology or should it showevery document classified as responsive or non-responsive?); howmuch agreement is required versus if a court or agency couldunilaterally order its use; what confidence intervals are appropriate inwhich contexts; and more-will allow each to develop a strongprotocol. These strong protocols will enforce and regulate the use ofpredictive coding to ensure that it is a positive development andstrictly regulated by the enforcing body to avoid any exploitation,such as what happened in Progressive."'

136. See infra Part IV.137. Interestingly, some cases have begun to address these specifics. For example, in In

re Biomet M2a Magnum Hip Implant Products Liability Litigation, the court addressed aparty's use of predictive coding along with other technologies to see if it sufficiently met itsdiscovery requirements. No. 3:12-MD-2391, 2013 LEXIS 84440 (N.D. Ind. Apr. 18, 2013),at *5-6. There, the court held that cooperation did not reach so far as to "requir[e] counselfrom both sides to sit in adjoining seats while rummaging through millions of files thathaven't been reviewed for confidentiality -or privilege." Id. at *6. For individualsresearching the metrics other courts facing predictive coding may use when evaluatingthese concerns, the role Sedona Conference publications play in this opinion is instructive.The court's discussion here is particularly helpful because it signals that the SedonaConference may play a role in establishing judicial standards measuring how automatedreview technologies are employed. Cf. id. at *6--8 (discussing the parties' citations to theSedona Conference and measuring the approach in this case against the SedonaConference reports). While the Sedona Conference could serve as an official vehicle forcourts' transparent publication of their standards for predictive coding, it remains to beseen whether there would, or could, be a universal agency vehicle to do the same. Inkeeping with the modest gains recognized so far for automated technologies in both ofthese areas, this Comment focuses on the potential transparency and metrics of individualagencies. See infra Part IV.

PREDICTIVE CODING

IV. PREDICTIVE CODING AND REGULATORY AGENCIES

While the jurisprudence addressing predictive coding remains inits infancy, its relative youth is offset by increased usefulness forscholars and attorneys due to its fairly broad accessibility. Whatjudges say in these cases is not a secret. Future judges and lawyers,who will argue based on and for these new judicial rules, can see whathas and has not been applied and, at the same time, what works andwhat does not work in the predictive coding jurisprudence. Thistransparency allows the profession to understand where points ofcontention will arise, what policy choices must be made, and whatsolutions are available to resolve these questions.

Unfortunately, government agencies lack full-scale transparencyin their interactions with predictive coding-including both theirindependent use of the technology and their receptiveness to dataretrieved with automated methods. This Comment argues thatincreasing transparency regarding agencies' procedures, uses, andconcerns would allow for an information exchange to createcomprehensive understandings of the potential for, and challengessurrounding, predictive coding in agencies, and regulatory law. Moreimportantly, greater transparency would also allow for a more holisticcomparison between regulatory uses for predictive coding and its usesin the judicial system. Significantly, this sort of meaningfulcomparison would create a feasible environment for both courts andagencies to adapt their policies to best address the issues posed byautomated technologies and ESI in the legal world.13 8

138. See Tonia Hap Murphy, Mandating Use of Predictive Coding in ElectronicDiscovery: An Ill-Advised Judicial Intrusion, 50 AM. Bus. L. 609, 651 (2013) ("The 2012predictive coding cases suggest reason for concern about cost, delay, andgamesmanship."). See generally Andrew Gallo & Sarah Kim, Predictive Coding: Processand Protocol, Bos. BUS. J., Fall 2013, at 22, 22 ("This article provides an overview ofpredictive coding and highlights issues likely to arise when negotiating such a protocol.");

2014] 257

258 NORTH CAROLINA LAW REVIEW [Vol. 93

Much like discovery and litigation, ESI's exponential growthcreates a mass of documents subject to potential regulatoryinquiries.' Similarly, predictive coding's promises of lower cost andincreased efficiency make it a revolutionary option to facilitatecompanies' and individuals' responses to regulatory inquiries. Thefuture benefits, which are even greater in regulatory contexts, maycreate incentives for cooperative methodologies and emphasize ashared end-goal between companies and agencies: the opportunity tomanually review fewer documents, resulting in monetary andmanpower savings.'4 ' Moreover, despite a lack of comprehensive andtransparent information, evidence indicates that at least someagencies-particularly the Department of Justice, the Securities andExchange Commission, and the Federal Trade Commission-are

Yablon & Landsman-Roos, supra note 16 (providing helpful summaries of what predictivecoding entails, current case law and court-ordered responses, and the issues that can arisetechnically and doctrinally in implementation).

139. See THE SEDONA CONFERENCE, supra note 2, at 18-19 (discussing "predictiveanalytics and compliance").

140. See Nelson, supra note 63 ("Regulatory inquiries by ... federal agencies might notbe cause for celebration, but using predictive coding technology to respond to governmentinquiries more effectively and with minimal risk may be a silver lining."). However, theconverse may also be true, particularly in investigations with criminal undertones-anindividual's biases could (un)intentionally taint the training process and thus returninaccurate documents as relevant or exclude relevant documents, both of which wouldharm the investigation. Cf. Tania Mabrey, Conquering Postindictment Discovery in theDigital Age, CRIM. JUST., Summer 2012, at 51, 52 (2013) ("A drawback to this technologyis that the intelligence draws from the perspective of only one or two humanreviewers-so it is important that the subset is reviewed by the person who has thegreatest knowledge of the case details and the types of documents that will make thestrongest impact as exhibits.").

141. See Nelson, supra note 63 (highlighting the unique party relationship that is "lessadversarial than typical litigation," as many corporations want to maintain good termswith government agencies and those same agencies appreciate the opportunity to decreasethe number of documents they must eventually review).

PREDICTIVE CODING

allowing predictive coding to be used to respond to inquiries-although sometimes merely "on a 'case-by-case' basis."l 42

A number of factors support a deliberative approach whenconsidering the use of predictive coding in regulatory contexts. Forexample, one observer highlighted that "[w]ading through [a] virtualavalanche of data can be intimidating in civil litigation, but effectivelysorting through ESI in a government investigation is even moredaunting, where one potentially exculpatory document may changethe nature of a case."' 43 Given that regulatory agencies share many ofthe same potential benefits and concerns surrounding predictivecoding, it is helpful to see how agencies address automatedmethodologies to provide a meaningful contrast, and possiblycomparison, with courts as both entities must reformulate theirexisting frameworks to adapt to machine-learning technology'sgrowing influence. To that end, this Comment explores how threeagencies-the Department of Justice ("DOJ"), the Securities andExchange Commission ("SEC"), and the Federal Trade Commission("FTC")-are responding to, regulating, and incorporating predictivecoding technologies.

A. The Department of Justice

The Department of Justice, established in 1870, serves under theAttorney General as "the central agency for [the] enforcement offederal laws."'" Because the Department is quite large, it is

142. Id.; see also William Kolasky, Antitrust Litigation: What's Changed in Twenty-FiveYears? ANTITRUST, Fall 2012, at 9, 15, 17 n.85 ("The antitrust agencies have also indicatedthat they are open to discuss the use of predictive coding in certain cases as well.").

143. Mabrey, supra note 140, at 51 (examining "some of the most effective ways toleverage e-discovery technologies in a government investigation and criminal litigation:data culling, concept searching, and predictive coding strategies").

144. Office of the Attorney General: About the Office, U.S. DEP'T OF JUSTICE,http://www.justice.gov/ag/about-oag.html (last visited Nov. 19, 2014).

2014] 259

260 NORTH CAROLINA LAW REVIEW [Vol. 93

subdivided into smaller divisions that have their own particularizedmissions.145 One example is the Antitrust Division, which "promote[s]economic competition through enforcing and providing guidance onantitrust laws and principles."I4 6 This Comment focuses on theDepartment's approach to predictive coding technologies through thelens of the actions and perspectives taken by the Antitrust Division.

The Department recognizes that electronic discovery. providesnew challenges for companies, which are now required to search andproduce more documents.'47 In response, it is among the agencies thathave already permitted predictive coding to be used to satisfyrequired document production in some situations.148 The Antitrust

145. See Department of Justice Agencies, U.S. DEP'T OF JUSTICE,http://www.justice.gov/agencies/index-list.htmi (last visited Nov. 19, 2014).

146. Antitrust Division: About the Division, U.S DEP'T OF JUSTICE,http://www.justice.gov/atr/aboutlindex.html (last visited Nov. 19, 2014).

147. TRACEY GREER, ELECTRONIC DISCOVERY AT THE ANTITRUST DIVISION: ANUPDATE 1 (2012), available at http://www.justice.gov/atI/public/electronic- discovery/281388.htm ("[E]lectronic productions have grown exponentially.... Today, we havesingle investigations that involve over 8 terabytes of data [and,] [r]ecently, the Division hascompleted several investigations with productions [around] ... one million records."). Insome of her material published on the DOJ's website, Greer, a senior litigation counsel,states that "[t]he views presented ... are [her] own, [and] do not reflect those of theDepartment of Justice or the Antitrust Division." E.g., id. However, given that thematerial is published by the DOJ on its website under the heading "Electronic Discovery,"see Electronic Discovery, U.S. DEP'T OFJUSTICE, http://www.justice.gov/atr/public/electronicdiscovery/ (last visited Nov. 19, 2014) (noting that "materials pertain to civilinvestigations only[,] ... are merely representative[,] ... [and] are subject to change" andinstructing the viewer to "consult Division staff for matter-specific guidance beforebeginning any collection of electronic documents for production to the division"(emphasis omitted)), this Comment will operate under the logical assumption that if theDOJ sees fit to publish this paper on its website under this heading, the views in the paperare actually endorsed by the DOJ.

148. Renata B. Hesse, Deputy Assistant Att'y Gen., Antitrust Div., U.S. Dept. ofJustice, IP, Antitrust and Looking Back on the Last Four Years, Presentation at theGlobal Competition Review 2nd Annual Antitrust Law Leaders Forum 13 (Feb. 8, 2013),available at http://www.justice.gov/atr/public/speeches/292573.pdf ("Another innovation

PREDICTIVE CODING

Division is an example of a division that needs access to largeamounts of documents and financial information in order todetermine whether any criminal wrongdoing has occurred. However,before disclosing relevant information, companies may need to firstperuse broad caches of data to gain a sense of what is relevant.Illustratively, the Antitrust Division permitted the use of predictivecoding to determine relevance in the "proposed merger of Anheuser-Busch InBev NV and Mexico's Grupo Modelo SAB." 49

In doing so, the Department recognized the mutual benefitsavailable when properly employing coding technologies. For example,these technologies allow the Department to "reduce the documentreview and production burden on parties while still providing the[Department] with the documents it needs to fairly and fully analyzetransactions and conduct."'s Predictive coding technologies alsoallow for a prioritized allocation of resources so that "only the 'really'relevant documents [are] produced.""' These benefits mirror thefiscal and efficiency gains that the same technology promised inlitigation and discovery.

However, predictive coding is not a perfect solution to theimbalances between companies or individuals and governmentagencies. Many of the same concerns raised in the case law also occurwith regulatory inquiries: in order for predictive coding to workwithin the existing dynamic, parties must employ "a high degree ofcooperation and transparency about the implementation and

we have been testing over the past several years to help streamline our process is allowingparties to use predictive coding in their document productions.... [W]e have allowedparties to use predictive coding in some matters already.").

149. Nelson, supra note 63.150. Hesse, supra note 148, at 13.151. GREER, supra note 147, at 4.

2014] 261

NORTH CAROLINA LAW REVIEW

structure of the predictive coding process."l5 2 Notably, thesetransparency requirements apply only to the producing party, not tothe agency. The Department's concern mirrors Judge Peck'sdiscussion with the parties in the discovery conferences in Da SilvaMoore, where he emphasized collaboration's centrality to themeaningful application of predictive coding.1 3 Questions also arise inthe regulatory context as to who is qualified to determine adocument's relevance and whether intentional or unintentional biascould taint the retrieval process.'54 Because of these concerns and thelack of sufficient data, the Department currently requires "writtenmodification" to the informational request when using the artificialintelligence potential for predictive coding and emphasizes the needfor "cooperation, transparency, time, and hard work," along withreview of the training set and the qualitative samples and the creationof a mechanism for supplementary document retrieval.' Theseconcerns show the likely direction for any predictive codingagreements permitted by the Department.'16

Significantly, despite the potential issues, the Department doesnot proscribe potential use of predictive coding in these inquests. Itswillingness to at least consider using coding in particular casesbolsters the need for open informational exchanges about predictive

152. Hesse, supra note 148, at 13.153. See supra note 93 and accompanying text.154. GREER, supra note 147, at 4-5 ("Relevance ... is in the eye of the beholder.").155. Id. at 5.156. Id. at 5 ("As an initial framework,... a plan [where the Division reviews the

training set and the qualitative samples and a mechanism is created for supplementarydocument retrieval] could serve as the basis for a meaningful negotiation between theDivision and the producing party."). However, without consistent publication of theseagreements and increased guidance from agencies, parties seeking to create a plan willhave to rely more on individual experience and give-and-take with the agency based onthese concerns, rather than following clear precedents.

[Vol. 93262

PREDICTIVE CODING

coding so that all agencies can put it to its highest use. Given theDepartment's emphasis on transparency to be sure that respondingparties adequately employ coding technologies, 5 1 it only makes sensethat the Department's similar transparency about its evaluationswould allow for the development of a broad-based, efficientframework.

B. The Securities and Exchange Commission

The Securities and Exchange Commission has a comprehensivemandate "to protect investors, maintain fair, orderly, and efficientmarkets, and facilitate capital formation.""' In order to effectuate itsduties, the SEC requires extensive disclosures from companies tocreate an informational balance for investors.'59 To produce requiredinformation to the SEC, companies may have to sort throughextensive caches of information and communications. As such,predictive coding may provide a more efficient retrieval system,allowing these companies to fully comply with the Commission'srequirements at the lowest possible cost.

157. See Allison C. Stanton, DOJ Director Talks About Investigations and E-DiscoveryTechnology, METRO. CORP. COUNSEL (Feb. 25, 2013), http://www.metrocorpcounsel.com/articles/ 22 623/doj-director-talks-about-investigations-and-e-discovery-technology("Transparency goes to the company's and counsel's preparation for discussions with thegovernment.... If a company used predictive coding and advanced analytics beforeproducing information, for instance, it can hurt a company's credibility if they don't tell usup front that they are planning to use these technologies."); Template, Dep't of Justice,Request for Additional Information and Documentary Material Issued to WeebyeweCorporation 10 (March 2012), available at http://www.justice.gov/atr/public/220239.pdf.

158. The Investor's Advocate: How the SEC Protects Investors, Maintains MarketIntegrity, and Facilitates Capital Formation, U.S. SEC. & ExCH. COMM'N,http://www.sec.gov/about/whatwedo.shtmi (last visited Nov. 19, 2014).

159. Id. ("[T]he SEC requires public companies to disclose meaningful financial andother information to the public. This provides a common pool of knowledge for allinvestors to use .... The result of this information flow is a far more active, efficient, andtransparent capital market . . . .").

2014] 263

NORTH CAROLINA LAW REVIEW

Similar to the Department of Justice and Federal TradeCommission, the Securities and Exchange Commission does not havea blanket ban on predictive coding for data production. Rather, itrequires specific disclosure by the requesting company and SECapproval before the company can use the technology to satisfy itsregulatory guidelines." What sets the Commission apart, however,and represents the next logical progression in regulatory treatment ofpredictive coding is the fact that it is beginning -to use predictivecoding software within its own systems and review processes inaddition to merely accepting information selected through automatedsystems.' 6' The Commission provides a perfect example of howagencies hold the potential to shape the way private parties usepredictive coding: by accepting the fruits of these technologies whenthe technologies are properly employed per publically accessibleguidelines, which still require Commission approval. However, thefact that a body as concerned with data and accurate production asthe SEC is signaling acceptance of coding technologies provides animportant foundation for private parties, as they now have anexternal standard by which to measure their interest in using codingin private cases. Additionally, this example could provide a yardstickby which courts could measure their own standards. This potential isstrengthened by the fact that regulatory bodies, like the Commission,

160. U.S. SEC. & EXCH. COMM'N, DATA DELIVERY STANDARDS 1 (Rev. Jan. 17,2013), available at http://www.sec.gov/divisions/enforce/datadeliverystandards.pdf ("Anyproposed production in a format other than those identified below, the proposed use ofPredictive Coding, computer-assisted review or technology-assisted review (TAR), or theuse of de-duplication during the processing of documents, must be discussed with andapproved by the legal and technical staff of the Division of Enforcement (ENF) and themethodology must be disclosed in the cover letter.").

161. See Ari Levy, Recommind Lands SEC as Software Client, SFGATE (Feb. 3, 2013),http://www.sfgate.com/default/article/Recommind-lands-SEC-as-software-client-4247554.php.

[Vol. 93264

2014] PREDICTIVE CODING 265

will learn how to address coding technologies through repeatedinteraction with it on the other side of the table as well as through itsactual use.162

Significantly, this potential for insider familiarity with thetechnology and knowledge about its strengths, weaknesses, andpossibilities could allow the Commission, along with other regulatorybodies, to use that familiarity to create effective guidelines fordisclosures."' The benefit of actual knowledge is without parallel andillustrates the next logical step in regulatory bodies' acceptance ofcoding technologies.'" Moreover, this knowledge can be put to evengreater use if the employing agencies are willing to be transparent andshare their experiences, successes, and trials not only with otheragencies, but also with the courts. This transparency would alloweffective rules to develop across the spectrum and also permitsoftware providers to adopt more effective methodologies."s

162. O'NEILL, supra note 120, at 6 ("Keep in mind that some government agencies arebeginning to incorporate predictive coding and other analytical software tools into theirroutine discovery programs (for example, the SEC recently acquired a license to use theRecommind predictive coding software).").

163. Cf. Murphy, supra note 138, at 657 ("Groups such as the Sedona Conference andthe Seventh Circuit Electronic Discovery Program... might also encourage greatertransparency regarding predictive coding and services, development of empirical dataregarding its effectiveness, and further improvement in predictive coding technology,which may lead to wider voluntary use of that technology and perhaps more 'just, speedy,and inexpensive discovery.' " (quoting FED. R. Civ. P. 1)).

164. But see id. at 654-55 ("A broader question may be whether a judge or indeed anygovernment actor should or can effectively promote public acceptance of newtechnology.... One scholar recommends that rather than mandating new technologies,government 'should identify what are the key goals or problems it is trying to address andthen not discriminate against any technologies that can help achieve those statedobjectives.' " (quoting Gary E. Marchant, Sustainable Energy Technologies: Ten Lessonsfrom the History of Technology Regulation, 18 WIDENDER L.J. 831, 856 (2009))).

165. Cf. Murphy, supra note 138, at 657 (citation omitted). But see Gary E. Marchant,Sustainable Energy Technologies: Ten Lessons from the History of Technology Regulation,18 WIDENDER L.J. 831, 845 (2009) ("If consumers are unwilling to accept or pay for a new

NORTH CAROLINA LAW REVIEW

C. The Federal Trade Commission

The Federal Trade Commission is tasked with "prevent[ing]business practices that are anticompetitive or deceptive or unfair toconsumers; ... enhance[ing] informed consumer choice and publicunderstanding of the competitive process; and . . . accomplish[ing] thiswithout unduly burdening legitimate business activity."l66 Similar tothe Department -of Justice's Antitrust Division, the Commission'smandate necessitates perusal of large amounts of data to determine ifany inappropriate behavior has occurred.' 7 As such, companiestargeted by the Commission for disclosure are subject to the sameburdens as those faced under obligations to the other agencies andduring litigation.

In a move similar to those of the Department of Justice and theSecurities and Exchange Commission, the Federal Trade Commissionshows preliminary acceptance of predictive coding through itsresponse letters that encourage the use of coding in disclosuresprocured by informational subpoenas.16 s Interestingly, however, theCommission moved beyond merely permitting predictive coding

technology, that technology is unlikely to prosper. Therefore, policies that attempt to'push' a new technology onto unreceptive or even uninterested consumers are particularlyprone to fail.").

166. About the FTC, U.S. FED. TRADE COMM'N, http://www.ftc.gov/about-ftc (lastvisited Nov. 19,2014).

167. Cf. A Brief Overview of the Federal Trade Commission's Investigative and LawEnforcement Authority, U.S. FED. TRADE COMM'N, http://www.ftc.gov/about-ftc/what-we-do/enforcement-authority (last updated July 2008) (laying out the breadth of the FTC'sinvestigative powers, per statutes and regulations, and illustrating the broad powers it hasto demand large quantities of information during its investigations).

168. See, e.g., Letter from Donald S. Clark, Secretary, U.S. Fed. Trade Comm'n, toSeth Silber & Douglas H. Meal, Attorneys 8 (Apr. 11, 2012) [hereinafter Letter from Sec'yClark to Silber & Meal], available at http://www.ftc.gov/sites/default/files/documents/petitions-quash/wyndham-hotels-resorts-lc/120411 wyndhamletter.pdf.

[Vol. 93266

PREDICTIVE CODING

(which raises the tensions and concerns previously discussed'6 1) andinstead moved towards requiring companies to, at the very least,consider predictive coding when attempting to decrease theirdisclosure responsibilities. 0 For example, in an April 11, 2012,response to a motion to quash or limit civil investigative demand, theFFC made a passing reference, when critiquing the petitioner's costanalysis, endorsing the affordability of predictive coding technologiesin these contexts."' The letter critiqued the cost estimate, stating thatit failed to "account for factors that may reduce the cost and time ofproduction . . . [because] Petitioners have not sufficiently addressedthe availability of e-discovery technology, such as advanced analyticaltools and predictive coding, to enable fast and efficient search,retrieval, and production of electronically stored information . ."172

Lest this appear to be an isolated incident, in another responseletter, the Commission addressed the correlation between ESI and anincreased burden, asserting that parties need to include "affirmativesuggestions [that] could include ... predictive coding."" Taken inconjunction with each other, responses like these illustrate theCommission's relative acceptance of coding technologies. This stephas logical parallels to the progression described in the case law.Automated technologies' mandatory acceptance, however, raises thesame needs for transparency and interactive learning with otherentities, given predictive coding's growing support in the regulatorycommunity.

169. See supra Part IV.A.170. A similar logical shift occurred in the case law. See supra Part III.171. See Letter from Sec'y Clark to Silber & Meal, supra note 168, at 8.172. Id.173. Letter from Donald S. Clark, Secretary, U.S. Fed. Trade Comm'n, to Mark W.

Nelson, Attorney 6 (May 23, 2011), available at http://www.ftc.gov/sites/default/files/documents/petitions-quash/w.l.gore-associates/1 10523quashgoreletter.pdf.

2014] 267

NORTH CAROLINA LAW REVIEW

Even more importantly, the Commission has begun the processof moving predictive coding decisions from informal determinationsinto the actual regulatory structure.174 This shift represents asignificant step in the progressive ladder towards full-scale acceptanceand implementation of predictive coding. In early 2012. theCommission proposed rule changes to 16 C.F.R. Parts 2,Nonadjudicative Procedures, and 4, Miscellaneous Rules."' Inexplaining these proposed changes, the Commission addressed the"Need for Reform of the Commission's Investigatory Process" andcited three key reasons in support of the use of automatedtechnologies: (1) "information is no longer accurately measured inpages, but instead in megabytes," (2) "ESI[] is widely dispersedthroughout organizations," and (3) "because ESI is broadly dispersedand not always consistently organized... , searches, identification,and collection all require special skills and, if done properly, mayutilize one or more search tools such as. . . predictive coding, andother advanced analytics.""' Notably, only the explanatory section inthe Federal Register specifically mentions predictive coding."However, the specific reference in the Federal Register section stillprovides helpful insights into how the agency views predictive codingand similar technology. Additionally, and most importantly, it signalsthat at least one regulatory body is willing to move beyond theintermediate stages of agency responses to meaningful rule-makingthat seriously considers predictive coding technologies.

174. See 16 C.F.R. H§ 2.2, 2.4,2.6,2.7,2.9-2.11,2.13,2.14, 4.1 (2014).175. See 77 Fed. Reg. 3191 (proposed Jan. 23, 2013) (codified at 16 C.F.R. §§ 2.2, 2.4,

2.6, 2.7, 2.9-2.11, 2.13, 2.14, 4.1 (2014)).176. Id.177. See id.

268 [Vol. 93

PREDICTIVE CODING

V. AN OVERVIEW OF INDUSTRY INSIGHTS AND MISCELLANEOUSFACTORS TO CONSIDER WHEN DEVELOPING THE RULES AND

REGULATIONS SURROUNDING PREDICTIVE CODING

Much of the review and analysis in this Comment, and much ofthe available academic literature,"' presupposes that private entities,or the attorneys representing them, will be the ones actuallyemploying predictive coding software in response to disclosureinquires, whether in litigation or in regulatory inquests. Interestingly,the SEC's use of coding technologies, when considered in conjunctionwith National Day Laborer, which also touches on this issue to anextent,179 poses an additional normative question: whethergovernment bodies should also be allowed to use predictive coding inresponse to document requests. Government bodies are held todifferent standards than private parties," and thus something thatmay be appropriate for a corporation or individual to employ may notbe sufficient to meet certain higher burdens held by agencies. On theother hand, because the government has similar interests in producingrelevant documents at low cost to personnel and budgets,"' predictivecoding provides parallel possibilities for increased speed, quality of

178. See, e.g., Barry, supra note 42, at 364-72 (arguing that "[clourts [sihould [a]dopt[p]redictive [c]oding" and providing solutions for the implementation of predictive codingin courts).

179. See supra notes 116-20 and accompanying text.180. For example, they are bound to uphold the Constitution in their actions. U.S.

CONsT. art. VI, cl. 3.181. See MARY GAY WHITMER, NASCIO, SEEK AND YE SHALL FIND? STATE CIOs

MUST PREPARE NOW FOR E-DISCOVERY! 3 (2007), available at http://www.nascio.org/publications/documents/nascio-ediscovery.pdf ("What's at stake? If a state is involved inlitigation, the outcome of the case could hinge upon the... retrieval of electronicinformation. In the event that the State CIO cannot ... locate and retrieve discoverableinformation, the state could be penalized ... [and,] [ujltimately, a negative litigationoutcome could cost substantial amounts of taxpayer dollars that might be spent on morepressing priorities." (emphasis omitted)).

2014] 269

NORTH CAROLINA LAW REVIEW

review, and fiscal efficiency. However, the ways in which states andother branches of government do or do not use predictive coding willlikely be influenced by the examples set by the courts and regulatoryprotocols. As such, increased transparency and awareness of theramifications of these decisions are crucial for the properdevelopment of predictive coding in a number of venues.

Another serious question is whether these technologies shouldbe permitted in criminal cases.182 Given that the Department ofJustice is already accepting data produced through automatedreview,'83 this concern is far more than academic. It also bringstogether regulatory and judicial concerns-hypothetically, predictivecoding technologies could be sufficient when an agency determineswrongdoing has occurred and pursues regulatory action against anentity but the court may not deem it acceptable for use in a judicialcontext. Additionally, the higher moral and punitive stakes incriminal cases particularly implicate the margin of error fromautomated retrieval. If the one document to prove innocence ismissed because it is within the margin of error, is that acceptable in acriminal case? Should it be? Should an attorney be able to make thatcall for a client or should explicit understanding and consent berequired from defendants? These are issues that connect regulatoryand judicial action and raise salient issues that touch more thanprocedural concerns and may call for public commentary in additionto legal recommendations.

182. See generally Andrew D. Goldsmith & John Haried, The New Criminal ESIDiscovery Protocol: What Prosecutors Need to Know, U.S. Arr'YS' BULL., Sept. 2012, at 3,available at http://www.justice.gov/usao/eousalfoia-reading-room/usab6005.pdf (discussingcriminal ESI discovery protocol disseminated by the Department of Justice); Mabrey,supra note 140 (suggesting ways to employ predictive coding and automated technologiesin a criminal government investigation).

183. See supra Part IV.A.

[Vol. 93270

PREDICTIVE CODING

VI. COMPARISONS, DIFFERENCES, AND RECOMMENDATIONS FORCOURTS AND AGENCIES ADDRESSING THE USE OF PREDICTIVE

CODING

A review of the seminal cases and limited information availableabout regulatory bodies' behavior reveals a number of parallel trends.Most fundamentally, predictive coding solves a common problem:ESI results in massive caches . of data that are fundamentallyincompatible with traditional human review but can be more readilyand affordably accessed with technology. Moreover, among bothcourts and regulatory bodies, there is clear support for heightenedtransparency and cooperation between the parties who seek toemploy predictive coding. Furthermore, shared areas of tension alsoarise. For example, can an unwilling party be mandated to engage in,or at the least consider engaging in, predictive coding? Is it better fordecisions to be made by individuals or bodies with personalexperience working with predictive coding? The latter question isparticularly poignant given that while agencies may gain actualexperience using, rather than regulating, coding, it is less likely thatjudges will do so. Additionally, actual use will help to enforceaccurate expectations of what the technology is capable of but mayalso create other biases, including those in favor of vendors or infavor of decreased transparency by parties, who may fear thattransparency could implicate privilege or attorney-work productconcerns.'" It also creates an imbalance between judges and agencies,

184. See Progressive Cas. Ins. Co. v. Delaney, No. 2:11-cv-00678-LRH-PAL, 2014 WL3563467, at *10 (D. Nev. July 18, 2014) ("[Llitigators are loathe to reveal theirmethodological decisions [when training an automated system] for various reasonsincluding assertions that: methodological decisions reveal work product; discovery aboutdiscovery exceeds the scope of Rule 26 of the Federal Rules of Procedure; revealingdocuments non-responsive to discovery requests exposes the producing party tounnecessary litigation risks; and the Federal Rules of Civil Procedure only require partiesto conduct a reasonable search for responsive documents.").

2014] 271

NORTH CAROLINA LAW REVIEW

as only agencies really have the power to gain this hands-onexperience, since.courts act as mediators while agencies can be partiesto disputes involving actual data. Moreover, it raises the question ofwhether there should be differences between litigation and regulatoryuse. Finally, both contexts place significant emphasis on the need forproper training in order to create unbiased results.

There are also a number of differences between litigation andregulatory contexts. First, thus far, litigation has involved privateparties and the potential for information and monetary imbalances,'whereas regulatory bodies can require private parties to produceinformation regardless of the individuals' finances and comfort.186

This dynamic relegates the greater incentives to private parties, ratherthan government agencies, for incorporation of this newtechnology.88 However, other factors make agencies seem to be themore attractive candidates for incorporating coding technologies. Forexample, agencies simply have better opportunities to gain experiencearound predictive coding: they deal with more eligible data cachesthan courts, they can use coding technology on their own initiatives,they are less bound by rigid rules of evidence and procedure, and theparties they work with have a high self-interest in promotingcoding.' The interaction with agencies is distinguishable in that the

185. Cf. Losey, supra note 33, at 17-18 ("The use of computerized categorizationtechniques, such as predictive coding, will likely become the norm for large-scale reviewsin the future, given the likelihood of increasing societal acceptance of artificial intelligencetechnologies.... The problem is that considerable sums of money are being spentunnecessarily today while attitudes slowly change over time." (citation and internalquotation marks omitted)).

186. See Lisa C. Wood, Predictive Coding Has Arrived, ANTITRUST, Fall 2013, at 93,95.

187. See supra Parts III and IV (describing the use of predictive coding by privateparties in litigation and government agencies, respectively).

188. See supra Part IV.

[Vol. 93272

PREDICTIVE CODING

agencies themselves must peruse the produced data, whereas judgesonly act as umpires between the parties exchanging the data.8 9

Judges, on the other hand, have strong interests in justice andefficiency, but mere context may render them less experienced orwilling to make a determinative endorsement of particulartechnologies." Additionally, courts are bound by the rules ofevidence and procedure."' Judicial decisions that integrate predictivecoding into a traditional and foreseeable structure have importantramifications both for the technology and how it develops-forexample, what limits on its use will stunt growth in some areas andfoster growth in others? A significant positive, additionally, is thegreat transparency judicial decisions have in this area."

As such, courts and agencies should share their strengths withthe other and complement the other's weaknesses. Both groupsshould also consider how and where to accept sufficient quality

189. Nelson, supra note 63 (illustrating how agencies' interactions are not a zero-sumgame).

190. See Harrison M. Brown, Note, Searching for an Answer: Defensible E-DiscoverySearch Techniques in the Absence of Judicial Voice, 16 CHAP. L. REV. 407, 424 (2013)("Courts have yet to embrace any of the new search technologies, instead only generallyalluding to potential benefits they offer, but not going so far as to expressly endorse aparticular method.").

191. See, e.g., Nathan M. Crystal, Inadvertent Production of Privileged Information inDiscovery in Federal Court: The Need for Well-Drafted Clawback Agreements, 64 S.C. L.REV. 581, 584-85, 591-92 (2013) (discussing privilege and work product); Waxse &Yoakum-Kriz, supra note 56, at 214-19 (addressing the interaction between automatedtechnologies and experts in the federal rules of evidence).

192. See NICHOLAS M. PACE & LAURA ZAKARAS, RAND, WHERE THE MONEYGOES: UNDERSTANDING LITIGANT EXPENDITURES FOR PRODUCING ELECTRONICDISCOVERY 98-99 (2012) (stating that "the legal world has been reluctant to embrace[predictive coding] ... [because] of the absence of widespread judicial approval" andasserting that "the best catalyst for more-widespread use of predictive coding would bewell-publicized instances of successful implementation in cases in which the process hasreceived close judicial scrutiny" (emphasis added)).

2014] 273

NORTH CAROLINA LAW REVIEW

control levels-at what point is the program sufficiently trained andreliable? As previously discussed, this may vary significantlydepending on the legal context. This means that while agencies andcourts can learn from each other's experiments, they also should becareful to acknowledge the differences between them and how thesedifferences may make certain rules either inapplicable or materiallyharmful to a particular context.

CONCLUSIONPredictive coding is still in its trial stages-in all likelihood, it will

not work in all contexts, and it will need guidelines in order to achieveits highest potential in appropriate cases. Because it is stilldeveloping, many questions remain as to what predictive coding cando and what it should do. The debate over how to answer thesequestions will only strengthen understandings of its possibilities andhighlight areas of concern. Raising questions is a positive thing: byidentifying early on the areas where tension and issues may arise,courts and agencies can create rules in light of these issues and play apreventative role. They can also highlight the issue areas that theymay be most concerned about-such as transparency, companies' andattorneys' desire to protect confidential, non-responsive documents,and cooperation. When determining both what areas to focus theirenergies on and how to achieve these goals, courts and agenciesshould also strongly consider research by other groups, such as theSedona Conference's Sedona Principles Addressing ElectronicDocument Production,19 3 as the informational exchange will be best

193. WORKING GRP. ON ELEC. DOCUMENT RETENTION & PROD., THE SEDONACONFERENCE, THE SEDONA PRINCIPLES: BEST PRACTICES RECOMMENDATIONS &PRINCIPLES FOR ADDRESSING ELECTRONIC DOCUMENT PRODUCTION ii (2d ed. 2007),available at https://thesedonaconference.org/download-pub/81.

[Vol. 93274

PREDICTIVE CODING

served through this sort of transparent exchange.194 By showing eachother what predictive coding has achieved and in what contexts it hasdone so, both courts and regulatory bodies can use this knowledge topromote rules that identify and create the best guidelines that allowpredictive coding to grow without stifling justice, impartiality, orefficiency. In order to maintain existing legal principles, or to make aconscious choice to overhaul existing norms, all bodies should behyper-vigilant when addressing new technology to be sure that allregulatory and guiding choices prospectively promote desired futurereforms, rather than resulting in forced, piecemeal changes. Predictivecoding holds great promise for the future of the legal field, but it is upto the community as a whole to ensure a healthy growth environmentso it can achieve its full potential while not alienating existing moraland procedural standards.

CHRISTINA T. NASUTI*

194. See Murphy, supra note 138, at 657. For example, even early on in Da SilvaMoore, Judge Peck cited the Sedona Cooperation Model (original cited in his academicwritings) when addressing the proper ways to handle predictive coding and ESI in thatcase. Da Silva Moore v. Publicis Groupe, 287 F.R.D. 182, 184 (S.D.N.Y. 2012) (citingAndrew Peck, supra note 75, at 29).

** Special thanks go to my loving family and fianc6, who have supported me throughevery stage of this project. I would also especially like to thank Professor Dana Remus,who introduced me to predictive coding and was the inspiration for this Comment's topic.Finally, thank you to the North Carolina Law Review Board and Staff, and Jennifer Littlein particular, for your thorough edits and help along the way. I am sincerely grateful toyou all.

2014] 275