“Lifting the Hood” on Technology-Assisted Review
Counsel at companies and firms who haven’t warmed to the idea of technology-assisted review often express some oblique concern about how to know what’s going on “under the hood.” It’s an interesting reference, because knowing what’s going on under the hood when computers are involved is a bit counterintuitive. After all, part of the aura—even magic—of a computer is that it does extraordinary things non-mechanically. We can’t really see it do what it does. What’s under the hood is a bunch of circuitry that might as well be jeweled eye candy for all the functionality it presents to us visually.
Most of us confront the mysteries of technology every day, trusting it for precision in some things, satisfied with fuzziness for others. We type numbers into Excel cells and expect that if we’ve input our factors correctly, it will perform calculations exactly—even the most highly-sophisticated formulas—and we often make very important decisions based on the results. On the other hand, we type a bunch of words into Google, and voila!—it brings back the kind of “messy” results we expect, generally some direct hits and some outliers. We know it’s not perfect, we don’t expect perfection, but we don’t really care; most of the time, it’s going to point us in the right direction and that’s good enough.
Interestingly, we don’t feel a pressing need to look “under the hood” when we perform either of these activities. We make assumptions about success based on a general understanding of how things must work, how much confidence and trust we have in those who provide, use and share information about such applications and, most importantly, by the results we get, tempered somewhat by our expectations.
When it comes to technology-assisted review, perhaps we should consider doing the same.
When computers are used in review, no matter to what extent, they can only assist humans. That’s why it’s called technology-assisted review. Predictive coding is technology-assisted. Knowledge-engineering (a.k.a. “rule-based”) is technology-assisted. And although the hood can probably be lifted to expose the minutiae of mathematical algorithms used in predictive coding (for mathematicians and technologists fortunate enough to understand them), or the thousands of sophisticated queries formulated by human-machine iterations for rules-based methods (for those interested in taking the time to pore through them), the proof can really only be in the measureable success of the outcome. How well did the stuff under the hood do to get us what we needed?
More than a description of cylinders and pistons, what we really care about are results—the performance and safety statistics, to beat to a pulp the automobile metaphor—and how they measure up to our expectations. Fortunately for the consumer, such performance statistics for technology-assisted review (known as “precision and recall”) are available via TREC Legal Track (www.trec-legal.umiacs.umd.edu) so the consumer can see the tested methods that perform the best.
It’s true that we should all strive to be educated consumers. In this case, though, it means knowing how technology is used in the document review process. That’s what’s under the hood. We should concern ourselves with the relationship between the technology and the people who are applying it and seek a process with the highest level of transparency. We should seek to understand how the results are measured and validated to ensure the process finds what we need. As with our use of Excel and Google, we should consider the reputation of the provider and understand how they would meet our expectations. That’s what will do the most in avoiding bumps in the road.
Image courtesy of Milestoned