On this episode of ACEDS #CaseoftheWeek, Kelly Twigger discusses the use of TAR in Livingston v. City of Chicago 2020 WL 6781801 (N.D. Ill. 2020).
Greetings, eDiscovery enthusiasts, and welcome to the ACEDS #CaseoftheWeek. I am Kelly Twigger, the CEO of eDiscovery Assistant and the principal at ESI Attorneys. ACEDS and eDiscovery assistant have teamed up to launch this weekly live stream so that we can help educate lawyers and legal professionals on eDiscovery case law and also to discuss the practical implications of decisions as they come out. And frankly, as you all know, I just like to talk about eDiscovery.
As many of you know, eDiscovery Assistant maintains a curated database of discovery decisions tagged by eDiscovery issues, including today’s issue about technology assisted review. Special welcome to all of our case editing team, who’s joining us on the call today.
Let’s get into our case of the week. This week’s case is a September decision from 2020 in the matter of Livingston vs. City of Chicago. This is a decision from Magistrate Judge Kim out of the Northern District of Illinois. For those of you in the Northern District of Illinois, you’ll know that Judge Kim is very active in discovery matters, very well educated in issues, and writes thoughtful decisions on eDiscovery. There are currently 13 discovery decisions from Judge Kim in our database. He joined the bench in 2010 and has been very prolific.
One of the things that I would encourage you to do, and one of the reasons that we pick the cases of the week, is that oftentimes they provide a really great write up from a judge explaining a decision. A lot of times in case law, we don’t get facts or great analysis on particular issues. And they don’t provide a whole lot of fodder for us to grow and develop in ediscovery; this case does. And so make sure you take a look at it.
The link to the case on eDiscovery Assistant that’s available to everyone is in the details page and in the comments on our LinkedIn site. If you’re on a different platform, you’ll need to check out the LinkedIn site to find the link to this week’s case. There’s also a link to a blog post that we wrote about the case on our blog at eDiscovery Assistant, about the case back in September, and also a link to an article on law.com written by my friend Philip Favro at Driven that discusses the case from a little bit different angle. Great information to check out; great analysis on this case.
Let’s dive in.
The case before us, which is, as I mentioned from September, follows two previous discovery decisions in this case. The most immediate decision prior to the September ruling was from November of 2019. And at that time, the plaintiffs were having a difficult time agreeing on a protocol for identifying and collecting information from the city of Chicago’s email system. In that November decision, the court, the plaintiffs essentially wanted to provide more input on how information was being collected, identified and collected during that ruling. The City wanted to use its Microsoft tool to search for and export items for review, ostensibly, I would think that that was probably in Office365 tool, but it might not have been. There is search functionality available in Office 365 that many parties use to export from. In that case, in that decision, the plaintiffs asked that a neutral party be provided, a neutral third party vendor go in and pull out data from the City’s email system then culled using search terms. The court agreed to that in November. Where we stand now is that the court had required the City to retain an outside vendor to export emails for a specific date range and then apply initial keyword searches using the plaintiff search term. The court also gave the parties at that time the potential to further refine the results if those search terms were still over broad.
But in the November case, it’s important to note that the court rejected the plaintiff’s request that those emails that were collected would then be produced without any further review for responsiveness or privilege, subject to FRE 502d order. That’s very important because it’s kind of unheard of, I think, for us to have a party ask essentially for a document dump. Document dumps are most of the time what we try to fight against because it’s so difficult to manage and it costs so much more money. In this particular instance, we were talking about 9 million documents that were pulled from the search terms. Following the November ruling, documents were pulled and then loaded into Relativity, and at that point we get into the dispute that is the subject of the September 20th, sorry, September 3rd, 2020 ruling from Judge Kim. Let’s talk about this one a little bit.
First what’s interesting about this one is that the motion that’s filed is a motion for compliance. That’s not a motion title that I’ve ever seen before, but it just tells you if you are a young litigator, that you can basically move a court for just about anything and then only certain motions are set out in the rules. In this case, the motion for compliance essentially asked the city to use the search terms and then perform manual review for privilege. And so the fight here is about how the documents that were pulled from the City’s email system are going to be reviewed. The plaintiffs want manual review of them to be done and the City of Chicago wants to be able to use technology assisted review or TAR within Relativity to be able to cull the the 9 million messages down to a reviewable set.
I believe I might be getting numbers wrong actually. I think the 9 million messages is what was pulled. From that 9 million messages, approximately 15 percent or 190,000 documents were actually identified using the plaintiff search terms. So it’s that pool of 190,000 documents that we’re talking about how we’re going to review.
The question before the court here is a couple of things. One, did the November ruling articulate how the city was required to review the documents and the court answered that question in the negative. It said, no, we never decided what the review methodology was that was going to be applied here.
The plaintiffs also wanted the court to adopt their protocol for TAR, which required the City to use TAR on the entire ESI collection so the full 9 million documents, instead of using TAR on 190,000 documents that were subject to the search terms. So really the difference in the party’s position on this case is that the City of Chicago wants to use TAR applied to the 190,000 documents that were responsive to the search terms, and the plaintiffs want to have TAR applied to the full population of 9 million.
Plaintiffs’ reasoning was that, it’s not incredibly articulated, but plaintiff’s reasoning was essentially that they believe if targets applied to the subset of 190,000, that large swaths of documents could be limited eliminated based on coding decisions by the team that sets that teaches the TAR. And the court looked at that analysis and said, if we think of it that way, then any coding decision that any reviewer makes is going to be subject to scrutiny because reviewers make different decisions in manually reviewing documents all the time and a team that’s coding documents for purposes of TAR is not going to be any different. As we get into the court’s decision, there’s an excellent discussion of the type of TAR that’s at issue here, which, as we’ve mentioned, is Relativity and the key features of TAR. And I encourage you to read this. If you’re if you haven’t used TAR before and you feel like it’s something that’s too expensive and out of the realm of possibility for you, I can tell you that it’s not and we’ll get into some of that of the practical implications later. But you want to kind of really start understanding how TAR is used. And I’m going to read to you a quote from the case because it articulates it really clearly. I think it’ll be really helpful for those of you who aren’t as familiar with the TAR process.
“In AI review like a manual review, search parameters are used to cull down a collected data set to a review set. That’s what we have here. That review is then put into the AI application [here Relativity] where the algorithms use data points collected through attorney review of documents in order to organize the documents in the review queue in a more efficient order. The difference between technologies is going to be the way that that algorithm is applied, and you’ll want to understand how the algorithm is being applied in your technology versus how it’s applied in different technologies. With each coding decision the attorneys make, the technology continues to learn and prioritize which documents contain contextually similar content as documents which are coded as responsive. Ai reprioritizes the documents in the review queue every 20 minutes.”
Now that is a huge improvement in technology over when we first started talking about TAR way back in 2015. If you’re talking about Judge Peck’s decision in Rio Tinto in 2005. I’m sorry, not 2015.
“They AI tool does not make any coding decisions about the documents, responsiveness, privilege, confidentiality or issue. Instead it merely shuffles the order of the documents being reviewed based on coding decisions. All documents marked responsive and ultimately produced are done so by human reviewers.”
TAR is taking human review and enhancing it based on technology. It’s taking the decisions that your human reviewers are making and enhancing it based on technology. And the cost savings for that kind of technology [is] are astronomical. For example, let’s talk about the practical implications of reviewing 190,000 emails. On average, an experienced reviewer who’s coding for multiple things, not just responsiveness, but also for privilege and other issue codes, might average between 35 to 40 documents an hour. We’ll be generous and say 40 documents an hour. At 40 documents an hour review of 190,000 emails would take 4,750 hours. Inexperienced coders who aren’t reviewing documents all the time will take longer. At 40 hours a week of work that is 119, weeks of review time. In case you can’t do the math, 52 weeks in a year, we’re talking about a long time to review. Even with 10 attorneys reviewing, it would take almost12 weeks just to finish a first pass review. There’s usually a second pass that also requires that you have review managers, QC time, constantly tweaking the review based on issues that come up and have to be addressed, coding panel changes.
When you think about who the parties are here and you’re talking about the City of Chicago, chances are good that the budget for the amount of time and costs associated with reviewing those documents is very limited. In terms of cost for review, we typically say that first past review is around a dollar a document. So if they reviewed these documents manually, you’re probably looking at in excess cost of about $300,000. That assumes that resources are available and that budget is available to contract that out.
OK, so thinking about this practically. Practically it makes a lot of sense to use any kind of technology that can limit the amount of information that’s available. Looking back at the November order–I missed one thing. The plaintiffs, what was really key fact here for the court and it noted, is that the parties agreed that generally TAR is a more accurate means of producing response ESI than a manual review or keyword searches. So essentially what we’re looking at here is the City of Chicago looking to combine keyword searches to find in the initial data set and TAR to pull that data set down to responsive documents.
The court then looked at the November order, as we mentioned before, and found that no methodology in review had been established, and they specifically rejected that the production would not be made without any further review. This was kind of an interesting quote.
“While the City may up 1.3 million pages of documents on plaintiffs with an entry of a 502d order, it also has the right to perform a review to produce only those documents that are responsive and relevant.”
I found that interesting because if somebody dumped 1.3 million documents on one of my clients, I would take a lot of issue with that.
On the review methodology, what was the court’s analysis? This is really important because what the court said is the plaintiffs were not able to provide any binding legal authority to support their request to use keyword searches to identify or to provide a review responsive ESI. The plaintiffs argued that because that TAR was more effective at identifying responsive documents and traditional manual review, that the pre-TAR culling would eliminate large amounts amounts of ESI. We talked about that earlier and the fact that the court rejected it. And so as we get to essentially what the court held following all of this analysis, they declined to adopt the plaintiffs’ alternative TAR protocol, which would have required to be applied to the entire 9 million document collection versus the 190,000. They found that the City was best positioned as the responding party to decide on what to search for, review, and produce responsive emails. And as a basis for that, they cited to Sedona Principle 6, which essentially says exactly the same thing.
The court cited to the fact that the City had disclosed that they were using Relativity TAR and the methodology and validation procedure that was to be employed. And what was key to the court’s ruling is that the plaintiffs’ insistence that the City must collaborate with them to establish a review, protocol and validation process has no foothold in the federal rules of discovery.
And that is a very important take away from this case. Also, it’s important to understand the context in that here in this case, as the court lays out, the party spent more than a year on a protocol and a method for searching and collecting information which further delayed a four year old case. And we were still looking at weeks to be able to produce information, even using TAR. Remember rule one, speed and efficiency in the judicial process; that’s always going to be a factor in courts decisions as to using technology and moving matters forward.
We talked a little bit about some practical implications of this case, some others that we’re starting to see. We’re starting to see a lot more of these cases where both sides are actively involved in the identification of ESI and the methodology for how ESI is identified. There can be a lot of confusion, misunderstanding, mistrust as to how parties are identifying information, whether they’re specifically coding documents in a certain way to exclude them from a TAR protocol. A lot of that requires you to validate when you receive a production, you need to get in there, dig in, look at the data, see what you’re seeing, make sure you’re getting search results.
If you are using search term strings as responsive, a lot of times we’ll ask for the search term strings as a metadata field so that we can see how many results we get on a search term string. If you’re applying to our protocols to that, you can still include that search term string as a metadata field. That’s one way to be able to use this layered production, but also be able to verify on the back end.
It’s really important, as you see from this court’s ruling, on the fact that there is nothing in the federal rules that allows the plaintiffs here to have input on how documents are reviewed. So it’s important for you to understand that rule. And also, what are your obligations with regard to ESI.
For me, looking at this, at the decisions in this case, it’s a surprising level of involvement for the court, but because the parties couldn’t agree, the court just had to get involved. What’s going to be really important for you practically, is to understand the level of sophistication on both part, on both sides, being able to discuss things in a cooperative manner. Sometimes that just doesn’t work. And you do need to go to the court. Do it earlier than later, so you can continue to move the case forward and make sure that you’ve got facts at hand to be able to demonstrate, and usually those facts include volumes of data. [So 9 million to ]reducing 9 million documents to 190,000 here is really a big deal.
It’s always important to note who the parties are. Here we talked about this a little bit, but the City of Chicago, well, it’s a great big government entity, it is unlikely that it has a staff of attorneys who have weeks upon weeks of time that they can devote to reviewing a document collection.
And we’ve really got to evaluate all of those things. It doesn’t lessen the City’s obligations as a party to respond under the federal rules, but it is something that has to be taken into account. You know there are proportionality considerations. They didn’t come up in this case, but it seemed to me as I was reading the decision, that there were proportionality considerations almost in not using TAR. If TAR could reduce the data set of 190,000 by half or a third or even more, then (I guess that was reversed a third or a half or even more than) cost considerations for the value of the information have to be at issue. There’s a proportionality argument to be made there.
You know, as an overview, the reality is that our judicial system and parties within the system can’t continue to support the cost of the volume of ESI and the costs of review. We have to continue to develop this technology that allows us to get to these answers faster.
Part of what we do for our clients at ESI attorneys and on the eDiscovery Assistant side is to constantly evaluate technologies that are out there. More and more solution providers are incorporating AI, TAR, or call it computer assisted learning, into their platforms. The one lump sum costs that you’re paying for that platform includes that technology. You should be using it. Use it if only to validate your results on the review side, but the reality is, is that we can’t continue to pay as much money as we’re paying in discovery. The solution providers are bringing the technology to the table, and now it’s up to the lawyers to use it effectively and to make arguments about it.
Costs have gone down a lot. You need to be looking into what solutions are available. Do that before you have a case that needs TAR or have your lit support folks look into that for you so that you can discuss and review alternatives so when you have a case, you have three different softwares that you can go to and determine what’s going to be the best value for that case.
We are going to be putting together a session on TAR in webinar format to talk about some of the practical implications of TAR.
What are the things you need to think about? If you’re interested in getting information about that when we put it together, you can sign up at ediscoveryassistant.com to receive our updates, and we’ll send you that information as it becomes available.
That does it for our ACEDS #CaseoftheWeek for December 1, 2020. Thank you very much for joining me. Have a great rest of your day. Stay safe and keep doing better discovery.
Watch Next Episode Live
#CaseoftheWeek will be available on the ACEDS YouTube, Facebook, Twitter, and LinkedIn company pages. You’ll also be able to read episode summaries and watch the recorded show right here on the eDiscovery Assistant blog every Thursday.
