How You Can Accomplish More with Fewer Reviewers

With the current pandemic and everyone working remote and case teams shrinking, I wanted to take a moment and break down the math behind a lot of problems that people are facing. Because reviewers are working remote, there is a limitation to how large of a review team you can staff. Sure, you can hire remote reviewers, but without an in-person review center, and in-person monitoring of the review team, there is an upper limit to how many people you can really staff. It stops becoming feasible to staff 50 people at once. The overhead of managing, QC’ing, and communicating with the team creates diseconomies of scale that limit the size of your team.

So, with that, how do you accomplish the same amount of work, with a smaller team, remotely, in the same amount of time?

I wanted to step back and look at the math behind that question. Assuming budget is not an issue here, and the deadline can’t be moved, what are the levers you have to hit your deadline?

The Baseline Budget

Let us take the scenario where you have 500k documents that need to be reviewed in 13 weeks and you were planning on using 20 reviewers. With COVID and remote work so prominent now, you don’t think you can successfully staff and manage more than 5 reviewers. If you didn’t change anything about the process, your review would now take 48 weeks.

If you’re trying to get the review done in less time, the obvious response is to tell those 5 reviewers to work harder and smarter until they figure out how to get it done. And there is probably some truth there. Let’s compare how much smarter or harder they would have to work.

Current industry standards are that reviewers generally work 40 hours / week and can review 50 documents per hour. If you wanted to have 5 reviewers accomplish what 20 reviewers did, you would need them to work 160 hours/week and off barely 2 hours of sleep. Or, if you wanted them to accomplish it smarter, they would have to review 200 documents/hour which not many people in the world could do, unless you’re Howard Berg.

Even if you do some combination of better productivity, like say 80 docs/hour, and more hours, like say 60 hours/week. Your team of 5 is still going to going to go 242% over the allocated time (or 17 weeks).

So how do you get your full review done in 13 weeks? Well, you start thinking about using AI to reduce the dataset, streamline the review and narrow the scope of the review to let you hit your deadline. Let us look at each improvement you can make, one-by-one, and see how it helps you hit your deadline with your limited team.

Improvement 1: EaiRLY INSIGHT? Investigation

Save 16.4 weeks by using EaiRLY INSIGHTS? to defensibly eliminate custodians and remove their documents from the data set to be reviewed.

Investing on front end with AI to further investigate and understand data collected will allow for less costs in review as documents can be eliminated from review. A trained ESI researcher can work with the attorneys to understand the fact patterns and goals of the review. Then, in combination with AI-based tools, they can quickly review a single custodian’s dataset at a scale that would be impossible without the AI. There are many great AI tools out in the market, our team personally utilizes NexLP's StoryEngine platform.

With this technique, you can defensibly eliminate custodians and data sources. In a typical matter with 10 custodians, who have had emails, laptops and mobile devices collected, usually two custodians and one data source can be eliminated through this investigation. In our 500k document example, that would amount to 164,000 documents eliminated (100k documents from two custodians, and 64k from largely non-responsive laptop data sources). This saves 16.4 weeks of review -- a pretty good start.

Improvement 2: EDA and Search Term Analysis

Save a further 8 weeks by further narrowing the document set with smarter search terms, informed by EDA and EaiRLY INSIGHT ? Findings.

Search terms and review feel like a chicken-or-the-egg situation. A lot of times, people feel like it’s hard to refine the search terms, without getting into the review to understand where the terms might be overinclusive. However, combining the knowledge from the EaiRLY INSIGHT ? investigation with expert search term consultants who can provide information on things such as, strength of search term (in depth search term analysis) and allow you to increase culling efficiency. This allows you to understand things such as, why particular terms have a lot of false positives. Example of this would be using the word “Beach” when the company’s name is “Beach Front Properties”. You would receive a lot of hits back on the term “Beach” thus raising the amount of documents your team would need to look through increasing reviews hours.

With this approach, we typically see improvements in culling from 25% to 35% per custodian. In our sample case, that ultimately saves 10k documents per custodian over the 8 custodians, meaning that 80,000 documents don’t need to be reviewed and 8 weeks of time is saved.

Improvement 3: All-Star Reviewers

Save 15 weeks by increasing reviewer throughput by leveraging advanced analytics and increase their weekly hours from 40 to 60.

If your managed review team is the same as the EaiRLY INSIGHT ? team, they understand the case and story of the data from the outset of the review and it creates increased accuracy and efficiency.

Often times, it’s the simplest of things that can make the world of difference. Being more strategic about the way you batch documents and prepare them for the review team can make a significant difference in review speeds. For example, batching by concept allows each reviewer to be more familiar with the pertinent issue, so they can move through documents more quickly. Or batching by near duplicates allows the reviewer to quickly transfer knowledge from the first document reviewed to the remainder of the batch, increasing review throughput. On the more technical side, it may be worthwhile to begin separating all PDF documents so that all reviewers are not waiting for large PDF’s to load, wasting unnecessary time.

Through this, your reviewers can do the equivalent to 80 Docs/Hour compared to the baseline of 50 Docs/Hour, saving an additional 9.6 weeks of review. When you combine that with increasing each reviewers’ weekly hours to 60, you save 15 weeks in total.

Closing thoughts

Obviously, this is more of a thought experiment than an actual real-world scenario. In reality, you’d probably push to staff more reviewers, or push to move the deadline. But, I do think it’s helpful to understand the mathematical drivers behind scenarios like this. Ultimately, you can get more done by working reviewers longer and harder. But, you get the most bang for your buck, by figuring out how to reduce the dataset at the outset.

In our scenario, although we would probably spend a bit more time upfront with the EaiRLY INSIGHT ? investigation, it more than pays off in the long run. I think because of the difficult times we are all going through with the current pandemic, until teams can ramp back up and bring employees back, there are ways you can continue to operate and do more with less. So, if you’re feeling overwhelmed because you’re shorthanded and not quite sure how to progress forward, my hope is that these tips will at least point you in the right direction and perhaps innovate and drive positive change.

Be sure to Follow Acorn for future content and Subscribe to Our Newsletter for the Latest Insights!

About Acorn

Acorn is a legal data consulting firm that specializes in AI and Advanced Analytics for litigation applications, while providing rigorous project management to the eDiscovery industry. Although capable of serving AmLaw 100 Law Firms and Corporate Legal departments, Acorn primarily works with large regional, midsize national and boutique litigation firms. Acorn provides a high-touch, customized litigation support services with a heavy emphasis on seamless communications. For more information, please visit www.acornls.com.

?