TAR Agreements and Protocols: The Devil’s in the Details
Technology-Assisted Review (TAR) is not a new concept, but can still lead to grounds for contention between parties. According to Federal Rule 26, parties have an obligation to cooperate and form a discovery plan in good faith. But sometimes—even when opposing parties have agreed to the use of TAR—things can go wrong.
Know (and Agree Upon) the Details of the TAR Approach
In Entrata, Inc. v. Yardi Systems, Inc., 2018 WL 5470454 (D. Utah Oct. 29, 2018) the parties met and conferred on multiple occasions regarding the use of TAR. According to the filings, Entrata’s attorneys responded to questions raised by Yardi, including how seed documents would be identified, the use of filters, and how documents not processed through TAR would be treated.
On September 20, 2017, the court entered a standard discovery order indicating the parties should agree on the ESI methodology within 30 days or file competing proposals, which neither side did. Entrata made a document production to Yardi on September 29-30, and on October 2, 2017, Yardi raised concerns regarding TAR and demanded Entrata’s TAR metrics. Yardi claimed that Entrata’s production was a “massive document dump,” and demanded responsive documents withheld via TAR be produced. Entrata indicated Yardi had waited too long to ask about metrics and argued no deficiencies existed. The TAR dispute continued through May 2018, and Yardi filed a motion to compel on the last day of fact discovery, May 21, 2018.
The Magistrate Court denied Yardi’s motion because it failed to provide “specific examples of deficiencies in Entrata’s document production or any specific reason why it question[ed] the adequacy of Entrata’s document collection and review.” On appeal, the district court upheld the decision, explaining that though the parties had never agreed on a discovery plan, Yardi was aware that Entrata would utilize TAR. It further analyzed that Rule 26 does not require parties to “disclose the manner in which documents are collected, reviewed and produced in response to a discovery request,” but that parties must meet and confer, educate themselves regarding ESI to resolve discovery disputes, and endeavor to agree on discovery search and collection methods.
Go by the (Right) Numbers
In In Re Domestic Airline Travel Antitrust Litigation, MDL Docket No. 2656, Misc. No. 15-1404 (Dist. DC, Sept. 13, 2018), the plaintiffs filed a motion to extend fact discovery six months because it found an issue with United Airline’s document production, which was identified using TAR. Specifically, the plaintiffs stated the 3.5 million documents United had produced were only approximately 17% responsive, and they required additional time to review the documents.
In the course of this litigation, the plaintiffs and United had agreed on a discovery schedule and the use of TAR. They had also agreed to a Validation Protocol to ensure that a recall rate of 75% was reached, and that United would “endeavor to achieve a higher estimated recall rate if that rate may be obtained with a reasonable level of precision through reasonable additional training effort.” Pursuant to the Protocol, United was to provide plaintiffs with the results of validation testing, including data such as false negatives, true positives, and false positives.
The TAR validation metrics United provided indicated recall was 85% and precision was 58%. Plaintiffs also analyzed the metrics, and discovered that recall was 97.4% and precision was only 16.7%. After some back and forth between the parties, United admitted the control metrics it had provided were incorrect. The court decided the plaintiffs had good cause for the discovery extension and approved their motion. It reasoned that they had diligently pursued discovery and notified the court as soon as they became aware of the issue with the document production in relation to the schedule. It further reasoned that the plaintiffs would be prejudiced by not having time to review United’s core production.
Practice Pointers for a (Mutually) Agreeable Outcome
These cases highlight several important practice pointers when developing a discovery agreement with opposing counsel that includes the use of TAR:
Cooperation: while litigation by nature involves conflict, parties are required to work together to form a discovery plan, including the use of TAR and validation metrics.
- Parties should discuss seed sets, filters, validation testing, tolerable results for those metrics, and answer related questions from other parties.
- The parties do not necessarily need to agree on target thresholds, but should agree on metrics to gauge effectiveness and the methods to estimate those metrics.
- The metrics should be recall (percent of responsive documents) and precision (the likelihood an algorithm has predicted responsiveness).
Validation: recall and precision should be reliably derived and tested, and shared with opposing parties to ensure comfort with the process.
- The validation protocol should be simple. Just six numbers are required: documents in 1) the delivered set; 2) in the validation sample drawn from that set; 3) responsive documents observed in the sample just noted; 4) documents in the not-delivered set; 5) documents in the validation sample drawn from that set; and 6) responsive documents observed in the validation sample.
- The validation sample should be entirely independent of any seed set or sets of documents used in the development process.
- The validation should have a qualitative component – the significance of any false negatives or false positives should be assessed.
Disclosure: any issues related to cooperation and validation should immediately be brought to the court’s attention. If it is believed that a document production is deficient, the court should be informed using specific examples of inadequacies to make its determination.
In the Entrata matter, the parties failed to agree to a TAR protocol that specified precision and recall metrics, which would have prevented court battles stretching up to the appeals level. In the case of Domestic Airline Travel Antitrust Litigation, United could have prevented conflict before the court by employing reliable control metrics and testing.
Both of these cases are examples of how attention to detail for the creation of the discovery plan, ESI and TAR protocols can save time, energy and frustration. TAR can streamline and enhance the discovery process, but it has to be done right.
To ensure success, it is important to work with a well-versed TAR expert with a proven track record, including experience analyzing and validating results. Experts can even be brought into meet and confer sessions to answer opposing counsel’s concerns and ensure all parties are on the same page. Contacting an expert early in the process can alleviate stress and disagreements that can arise when using TAR to collect client data – situations that don’t benefit any of the parties.
To learn more about TAR agreements and protocols, download “Questions for Opposing Counsel Regarding the Use of TAR” here.