Mark my keywords: experts in search can be a solution for judges
In Assured Guar. Mun. Corp. v. UBS Real Estate Sec. Inc., 2012 U.S. Dist. LEXIS 167981, (S.D.N.Y. Nov. 21, 2012) Magistrate Judge James C Francis IV made two important, and I believe interestingly related statements:
1. That expert testimony was “necessary for [him] to offer an opinion as to the most efficient search protocol.”
2. That one of the options the parties had was to “request the appointment of a neutral consultant who will design a search strategy.”
The first statement brings up the question of who is competent to opine on a search protocol. It is often assumed that ESI or e-discovery expertise is equivalent to search expertise. This is a mistake. The nature of the challenge–finding relevant information in an ocean of electronic data—is not an ESI challenge. It is a search challenge or, as it is referred to in academia, an information retrieval (IR) challenge. An ESI expert designing a search protocol is a matter of the tail wagging the dog.
So what makes a person an expert in information retrieval? The academic community of information retrieval includes scientists with credentials in a range of fields such a computer science and linguistics, as is evident in this list of most-cited IR academics. None would likely claim to be a member of the legal or e-discovery communities and conversely, an attorney or ESI expert is an unlikely member of the IR community. This is not to say that attorneys or ESI experts cannot be competent in IR: a few are. But, as a rule, it is preferable to look for experts in the right domain.
It is therefore important for courts to ask: is a proposed search expert truly a search expert? A court would be unlikely to deem someone a legal expert unless that person had academic credentials in the field of law. Similarly, unless a person has followed advanced academic studies in relevant fields—linguistics when it comes to developing complex queries, or statistics when it comes to measuring the accuracy of a search—they are highly unlikely to be IR experts.
Yet today, courts and practitioners routinely confuse e-discovery expertise with search expertise. It is telling that in Da Silva Moore, when the challenge to be addressed—measuring accuracy—was a statistical challenge, not a single expert in statistics applied to IR was present in the courtroom. (This is why the protocol devised to measure accuracy does no such thing.) By contrast, testimony by an IR expert in Kleen made evident the scientific basis of the field.
The second statement is one of particular interest to me, as I have been amongst those who believe that “Honest Brokers,” with requisite academic and professional credential in the field of IR, would be much more effective and efficient in designing, executing and measuring searches (including in responsiveness review) than attorneys and ESI experts engaged in battle can ever be. The case for such Honest Brokers is simple:
- First, the interest of society is for courts to resolve disputes based on a full set of facts, not on the limited set of facts permitted to emerge through adversarial gamesmanship. The Honest Broker can serve that interest.
- Second, because the Honest Broker would be an expert in the field of IR, it is certain that the resulting search protocol would be more robust, its accuracy more reliably measured, and its execution far cheaper than the current alternative. (The concern that adding such an expert could increase the overall cost of discovery is unfounded: professionals in any domain produce substantially more effective and efficient processes than non-professionals. This holds true in every domain, and it holds true for information retrieval.)
My takeaway: Judge Francis’s two statements, taken together, point in an interesting direction:
- Judge Francis says that search (IR) is an expert, scientific domain. Indeed, it is well-established in academia and is distinct from law, e-discovery, or ESI expertise. It follows that experts should, therefore, generally reside in that domain.
- If search is recognized as a science, it cannot be addressed like some commercial dispute: a solution does not result from acrimonious negotiations. If search is a science, then the argument for an adversarial process on search protocols in responsiveness review or the measurement of accuracy falls away: the adversarial process is entirely unlikely to produce a scientifically valid search protocol.
- An Honest Broker can better serve society’s interests by following reliable, replicable protocols based on established science (including the TREC research), ensuring that a production achieves high recall (finding all the facts) in discovery. The Honest Broker would also better serve the parties’ financial interests by ensuring high precision (a more on-target result set) with greater automation and reliability than any non-IR expert could ever hope to achieve.