to Speed Document Review and Reduce Costs …...A Practical Guide to Technology Assisted Review 1. Train Review Seed Set 2. Check Results Review Control Sets 3. Apply Computer-assisted

A Practical Guide to Technology Assisted ReviewApplying Transparent, Scalable, Predictive Coding Technology

to Speed Document Review and Reduce Costs

About our Webinars

● Webinars take place monthly and cover a variety of relevant eDiscovery topics

● If you have technical issues or questions, please email [email protected]

● Lexbe webinars are available for viewing (streaming video), or download as a PDF Presentation or an MP3 podcast

● This Webinar and a complete listing of other onDemand webinars is part of the: Lexbe eDiscovery Webinar Series

● For notices of future live and on-Demand webinars as part of this series please email us at [email protected] or: Follow us on LinkedIn

A Practical Guide to Technology Assisted Review

mailto:[email protected]

http://lexbe.com/resources/ediscovery-webinars/?LEX=Webinar-PDF-15010-28-eDiscProcs

https://www.linkedin.com/company/lexbe

About Lexbe

We’re an Austin, TX based provider of highly affordable cloud-based eDiscovery software and services provider, specializing in serving boutique law firms and organizations. We provide:

● The Industry’s Most Affordable and Full-Featured DIY eDiscovery Platform● The Industry’s Fastest eDiscovery Processing & Document Review Software● Experienced eDiscovery Specialists and Expert Consultants

“Secure, easy-to-use and a great review tool”

“Lexbe has long provided an alternative to law firms that want to

avoid the tremendous expense involved with eDiscovery software”

“Cost-effective eDiscovery”


G2 Crowd survey finds that Lexbe delivers best ROI in the industry and leads in 6 key metrics.

Guest Speaker

● Senior eDiscovery Specialist at Lexbe, a provider of cloud-based litigation processing, review and document management software & eDiscovery services

● Certified by the Association of Certified E-Discovery Specialists (ACEDS)

● Erika is also a litigation paralegal with experience in Complex Commercial Litigation

Erika Biller, ACEDS512-677-6916

[email protected]

LinkedIn


Erika Biller Bio


https://www.linkedin.com/in/erika-biller-62652050/

Agenda

Best Practices: Technology Assisted Review● What is Technology Assisted Review (TAR)?

● Why use TAR/Predictive Coding?

● How does TAR/Predictive Coding work?

● Importance of Transparency in TAR Applications

● Parameters for TAR in an ESI Order or Stipulation


What is TAR / Predictive Coding?


● Technology Assisted Review allows a skilled reviewer to train a computer algorithm to identify responsive and non-responsive documents in a document collection.

● As an alternative to manual linear review, predictive coding can drastically reduce the amount of time needed to review increasingly large ESI volumes.

Why Use TAR / Predictive Coding?


● Best opportunities for further cost savings will be reducing review costs.

● Technologies and process improvements, like TAR, reduce costs by increasing attorney review efficiencies

CASE STAGE

Collection 8%

Processing 19%

Review 73%

Total 100%



Increase Review Speed: TAR is designed to complete the review of large ESI collections faster than human reviewers. Applying TAR in a scalable environment maximizes the speed advantage of predictive coding.

Decrease Review Costs: Whether paying per document or per hour, TAR is significantly less expensive than exhaustive manual review.

Increase Review Quality: Many studies conclude that the presumed quality advantage of ‘gold-standard’ manual review is not accurate. TAR can support defensible, high-quality review outcomes.


How Does Assisted Review Work


2. Check Results

Review Control Sets

3. ApplyComputer-assisted Coding of Remaining Case Documents

Seed Set Selection Methodology

● Pre-cull data set using keyword searches prior to generating a Seed Set - OR -

● Generate Seed Set from a random sampling across the entire data set

TAR / Predictive Coding Workflow

1. TrainReview Seed Set

2. Check Results Review Control Sets

3. ApplyComputer-Assisted Coding to Remaining Documents



● A randomized sample of documents, the Seed Set, is generated from the collection.

● A skilled document review professional reviews and codes the seed set.

● The coding decisions made in reviewing the seed set train the predictive coding algorithm to identify responsive content in the remaining documents.









● Iterative samples of 25 computer-reviewed documents, control sets, are inspected to determine algorithm accuracy.

● The responsiveness designation assigned to the document by the computer is either confirmed or overturned.

● An F-Score, derived from precision and recall measures, indicates the stability of the TAR results.


2. Check Results

Review Control Sets







● The TAR algorithm reviews the document collection based on how it was trained during seed set coding and control set review.

● Remaining Documents are tagged as responsive/non-responsive.

● The speed at which the document collection is reviewed by the TAR algorithm is largely based on the computing resources applied to the task.


2. Check Results

Review Control Sets











2. Check Results

Review Control Sets


Precision: A measure of how often the algorithm accurately predicts a document to be responsive; the percentage of responsive documents that are actually responsive within a particular margin of error.

Recall: A measure of what percentage of the responsive documents in a data set have been correctly classified by the algorithm within a particular margin of error.

F-Score: Harmonic mean of precision and recall. **Note: F1 scores should not to be interpreted as a measure of the algorithm’s review quality but rather as an indication of 1) how well the case lends itself to TAR, and 2) the quality of the seed set training.

Understanding TAR/Predictive Coding Results




2. Check Results

Review Control Sets


High Recall, High Precision: All of the responsive documents in the collection were appropriately coded by the algorithm (high recall). All of the documents produced are actually responsive (high precision). Best possible outcome.

Actual

Predicted

Non-Responsive Responsive





2. Check Results

Review Control Sets


High Recall, High Precision. Stabilized Metrics in the Lexbe eDiscovery Platform:





2. Check Results

Review Control Sets


Low Recall, High Precision: Many of the responsive documents in the collection were not appropriately coded by the algorithm (low recall). However, a high percentage of the documents produced are responsive (high precision). Increased risk of under-producing.

Actual

Predicted


X


XX




2. Check Results

Review Control Sets


Low Recall, High Precision. Stabilized Metrics in the Lexbe eDiscovery Platform:





2. Check Results

Review Control Sets


High Recall, Low Precision: All of the responsive documents in the collection have been appropriately tagged by the algorithm (high recall). However, many erroneous documents were incorrectly marked responsive (low precision).

Actual

Predicted


X


X X X X X




2. Check Results

Review Control Sets


High Recall, Low Precision. Stabilized Metrics in the Lexbe eDiscovery Platform:


Importance of Transparency in TAR Applications



2. Check Results

Review Control Sets


●

Defensibility: Without understanding how a particular TAR/predictive coding methodology works, it becomes difficult to explain why the algorithm made certain coding decisions.

Appropriateness: TAR is not meant to be used in any and all review situations. Without understanding how a particular TAR/predictive coding methodology works, it is impossible to determine if it is appropriate for your case.

Importance of Transparency in TAR Applications



2. Check Results

Review Control Sets


● In TAR, Bayesian Probability models the likelihood of something being true about a document, i.e. responsive, based on the millions of data connections created while training the seed set.

● A Naive Bayesian Classifier, used in Assisted Review+, is a probability model with assumptions that allow for pattern recognition among multiple variables that are independent of one another.

WordsResponsive

WordsNon-Responsive

Parameters for TAR in an ESI Order or Stipulation



2. Check Results

Review Control Sets


Sample language regarding TAR for an ESI Agreement or Order

, ,

Summary



2. Check Results

Review Control Sets


● TAR/Predictive Coding allows a skilled reviewer to train a computer algorithm to identify responsive and non-responsive documents.

● You can use TAR/Predictive Coding to increase review speed, decrease review costs, and improve the quality of review results.

● TAR works by teaching a seed set, testing the algorithm against control sets, and applying the improved algorithm to the remainder of the collection.

● Predictive coding performance results are communicated in the form of precision and recall scores.

● It is important to know the underlying logic of the TAR algorithm to interpret, explain, and defend your results.

● Parameters regarding the use of TAR as a review tool should be set forth in the form of a stipulation or an ESI Order.

Lexbe’s Assisted Review+



2. Check Results

Review Control Sets


● Force Multiplier - enabling you to handle cases with millions of documents and tens of millions of pages by only coding a seed set that trains the algorithm on how to code the remaining documents.

● Transparent - enabling you to fully defend the process for coding responsive and non-responsive.

● Lexbe’s Uber Index feeds the Assisted Review+ algorithm with the most complete data set:

1. Image files are automatically run through OCR.2. TAR and search includes native and OCRed versions.3. Lexbe’s assisted review is specific to an email body and/or attachment,

not grouped together.

Thank You for Attending


We’ll be making the following available to webinar attendees:

● A recorded streaming version● MP3 podcast● PDF

Please let us know if you have any questions or comments about this webinar or suggestions for future topics. This webinar is part of the Lexbe eDiscovery Webinar Series. For notices of future live and on-Demand webinars as part of this series please email us at [email protected] or Follow us on LinkedIn.

Presenter:

Moderator:

Erika [email protected]

Jeff [email protected](512) 653-8295

Questions: [email protected]

http://lexbe.com/resources/ediscovery-webinars/?LEX=WebinarBestPracCloudeDisc

https://www.linkedin.com/company/lexbe?LEX=Webinar-PDF-151028-eDiscProc

https://www.linkedin.com/company/lexbe?LEX=Webinar-PDF-151028-eDiscProc


Learn More About Lexbe


‘Cost-effective eDiscovery’

“A powerful litigation document management service”

“Because of the Lexbe software, the entire playing field has been leveled for my firm.”

‘Lexbe cost advantages, SaaS convenience and search capabilities appeal to many small firms

“Lexbe is the easiest eDiscovery software I have ever used’

‘Secure, easy-to-use and a great review tool for consideration’

● The Lexbe eDiscovery Platform, is our cloud-based processing, review and production tool. Designed for Attorneys/legal staff to be DIY and easy to use, with no users fees or case fees. Free standard loading with annual plans.

● Learn about our high-speed/high-capacity eDiscovery services, and expert professional services.

● Request a personalized demo and expert consultation today!

1-800-401-7809 x22 | [email protected]