A respectful reminder that TAR includes more than predictive coding, and many approaches “done right” will succeed.
“The so-called gold standard of eyes on review of every document is not the gold standard by any means. Predictive coding, TAR, when done right, has a much better and much cheaper result than annual [sic] review, keywords or anything else.” -Judge Andrew Peck
There is no question Judge Peck’s intrepid thought leadership with regard to the use of technology for document review has been a driving force in the evolution of eDiscovery methodologies, and he is most certainly owed a debt of gratitude for helping to challenge the status quo and question the previously accepted “gold standard” of manual document review.
That said, Judge Peck, like many other thought leaders and policy makers in the eDiscovery community, creates confusion by conflating the terminology and concepts surrounding technology-assisted review (TAR) and by categorically endorsing some approaches while unilaterally vilifying others.
The difficult truth is that there is no single right answer for eDiscovery and no single approach to document review that is best for all use cases. Not all keywords are created equal, nor are all TAR protocols or all manual review processes. A successful, high-quality, cost-effective document review is defined not by the specific algorithms or mechanical steps taken to achieve it, but by the thoughtful, principled, well-motivated design that serves as its foundation. This is undoubtedly what Judge Peck was alluding to when he qualified his endorsement of PC with the phrase “when done right.”
Those familiar with TREC, or with the well-known Maura Grossman-Gordon Cormack JOLT paper Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, XVII RICH. J.L. & TECH. 11 (2011) may recall that it was H5’s results, along with those of Waterloo, that were highlighted as outstanding. What may not be as well known, however, is that H5 used its own TAR methodology, not predictive coding or machine learning, to achieve those results.Thus, H5 knows better than most that keywords, when thoughtfully developed and thoroughly tested, can be incredibly effective since keywords are an integral component of H5’s TAR process.
Dismissing the potential value of keywords “done right” is no more justifiable than embracing the adoption of inexpertly applied and/or inadequately validated machine learning techniques for document review or volume reduction. And treating any single class of document classification tactics as equivalent – failing to recognize differences in their applications, implementations, evaluations and context appropriateness – oversimplifies eDiscovery in a way that may ultimately do more to inhibit progress and innovation than to foster and promote it.
Thus, while we should certainly thank Judge Peck for inspiring the industry to consider alternative paths for eDiscovery, we should also recognize that multiple avenues are open to us and the best directions will differ depending on the destination.
Amanda Jones is a linguist and managing consultant at H5 with expertise in applying advanced linguistic strategies to complex information retrieval projects in the field of eDiscovery.
Visit us at www.h5.com