Read carefully: not all technology-assisted review is created equal
In February of 2011, Maura R. Grossman and Gordon V. Cormack published a groundbreaking study of the effectiveness of technology-assisted review as compared with the of manual review. The study was groundbreaking in that it was the first well-designed study of the accuracy of manual review. (The often-cited Blair and Maron study from 1985, which is often taken to be a study of manual review, is actually a study of the effectiveness of manually constructed Boolean queries.) Drawing on data from the TREC Legal Track’s interactive task, the authors compared the effectiveness of two of the technology-assisted methods evaluated in that TREC exercise to the effectiveness of the manual review teams that supported the evaluation.
The authors concluded that the data analyzed “provide strong evidence that the technology-assisted processes studied here yield better overall results, and better precision, in particular, than the TREC manual review process.” The authors were careful to note that their conclusions held for only the two technology-assisted approaches studied and for the manual review process employed for the TREC exercise.
In an article recently published in the National Law Journal, Michael Duffy takes a look at the conclusions enabled by the Grossman and Cormack study. (Duffy also looks at the conclusions enabled by a second study, conducted by the Electronic Discovery Institute; we’ll consider that study in another posting.) Duffy’s article has a number of limitations. Among these is an overly simplistic view of the range of review methods: Duffy reduces the range to two categories, either manual or automated, when in fact almost all technology-assisted methods in real-world application are somewhere in between those two ends of the spectrum (some automation, some human input).
More importantly, in discussing the Grossman and Cormack article, Duffy criticizes a position the authors do not take. Duffy argues that the data analyzed in the Grossman and Cormack study do not enable the conclusion that all technology-assisted approaches are superior in every instance to all forms of manual review. As already noted, however, Grossman and Cormack say nothing of the kind, taking care to limit their conclusions to just the methods and conditions studied.
Duffy’s article does, however, serve as a helpful reminder of two points. First, not all technology-assisted review approaches are created equal: the fact that some technology-assisted approaches have been found to be effective does not mean that all technology-assisted approaches will be equally effective.
Second, read carefully. Studies of the effectiveness of document review processes and technology-assisted review systems, like the Grossman and Cormack study, are generally, of necessity, complex, controlling for a range of different variables, and the conclusions of such studies usually come with an ample supply of caveats and qualifications. Readers would be well advised to read the studies carefully before jumping to overly broad generalizations.