Is legal research too important to be left to lawyers?

The question may seem needlessly provocative. Shouldn’t legal academics be the best qualified to conduct and evaluate legal research? How could it be otherwise? The current context with the accelerating trend towards interdisciplinary approaches, however, makes the question legitimate. Don’t get me wrong. I have nothing against interdisciplinary legal research. Quite the opposite. Most of my research qualifies as interdisciplinary. (Some may argue that I’m not really a lawyer, but that’s another debate.) The problem I see has more to do with the research attracting funding and appearing (or not appearing) in top law journals.

A Primarily European Problem

The problem is primarily a European one. Legal scholarship in the United States and in places like Israel has long embraced interdisciplinarity. It is common for law professors at top US law schools to have a PhD in another discipline as well as a JD in law and the Israeli legal academy has very few researchers left doing only doctrinal analysis. These places already have a solid ecosystem of practices, institutions and, crucially, knowledgeable scholars to supervise, produce and assess interdisciplinary work.

Not so in Europe. In most corners of the European legal academy (there are, of course, some exceptions), doctrinal approaches continue to reign supreme. Compared to other academic disciplines, there is little in the way of a methodological discussion. In the Dutch-speaking world articles and PhD dissertation have been citing the one same methodology handbook for years. My French colleagues still talk about the “plan binaire” as if it added up to a scientific methodology. Given the emphasis of the doctrinal approach on textual sources and interpretation you would expect legal scholarship to be to some degree conversant with advances in linguistics and neighbouring disciplines. But you might count the number of European law professors familiar with relevance theory, post-Gricean pragmatics and semantic prototypes on the fingers of your hand. (Some lawyers would retort that they are perfectly able to produce correct/valid interpretations of legal texts without all this theoretical baggage. But that’s the difference between being right and being able to provide a rigorous, scientific explanation for such a claim. It is also, by the way, the difference between the idiot savant, who is always right but doesn’t quite know why, and the scientist, who may not always be right, but is able to supply a systematic explanation for her assertions.)

Interdisciplinary Legal Research Is Evaluated by Scholars Without Interdisciplinary Background

The upshot is that legal research claiming to make innovative use of methods – whether randomized controlled experiments, statistical modelling, Natural Language Processing (NLP) or machine learning – developed outside the legal field is largely reviewed and evaluated by people who have little, if any, knowledge of their assumptions, strengths and limitations.

Methodological ignorance is often betrayed by silly comments. Recently, I submitted an application to the legal panel of the Flemish research agency proposing to investigate differences in legal thinking across legal traditions using randomized controlled vignette experiments. One reviewer objected that my suggestion to conduct statistical robustness checks – a standard practice in quantitative research to ensure that results are not driven by problematic assumptions – was “discriminatory” and would go against “diversity” because it would exclude some observations! Another reviewer objected that the assumption of the controlled experimental design was that “lawyers don’t differ”. A comment that flies directly in the face of the main reason randomized controlled experiments have become the gold standard for causal inference across all empirical disciplines. Randomization and control groups are precisely there to account for unobserved or even unknown differences (as well as well-known observable ones). Similarly, commenting on a manuscript applying various NLP technique, including one known as probabilistic topic modelling, the scholar who was then serving as the editor in chief of the prominent law journal where I had submitted it described text-mining as “status quo oriented” (unlike the doctrinal method, which is seemingly presumed to be reform-oriented)!

Methodological ignorance can induce enthusiastic reactions to interdisciplinary approaches as well as hostile ones. As a reviewer for various research funding bodies I occasionally see proposals from legal academics that seek to impress panelists by proposing to apply some NLP procedures or other quantitative techniques. I’ve once looked at a proposal that basically proposed to use feedback from US in-house lawyers on the regulations that worked for their own corporation to train a supervised machine learning to predict what legislation would be good for business and innovation in the European Union. To anyone remotely familiar with NLP and machine learning, it was not only manifestly infeasible under the current state of the state, but it can be confidently said that machine learning and NLP will never develop techniques to extract from legislative texts information that is simply not there. Still, I heard some colleagues were ecstatic about the “awesome methodology” of the proposal. So I’m afraid to say that the such a “fake it” strategy is often rewarded.

What Can Be Done?

If reviewers and panels can’t tell the wheat from the chaff, it’s neither good for science nor for society and the taxpayers who foot the bill. Yet there’s something akin to a chicken and egg problem here. We need more interdisciplinary legal scholars to evaluate interdisciplinary legal research, but we won’t get the critical mass of interdisciplinary scholars unless we train and fund more of them. So what can be done? One solution is for research funding bodies to make panel interdisciplinary (it’s already the case for ERC panels). Law journals, as some already do, should also consider inviting more reviewers from other disciplines. Finally, European law schools should be encouraged to recruit academics with training in other disciplines.

These solutions are not easy to implement. To get computer scientists with little background in law and lawyers with little background in computer science to engage in a fruitful dialogue over the merits of a piece of interdisciplinary legal research isn’t easy. But, at this point and until the emergence of a more established community of interdisciplinary researchers, legal research – as Clemenceau once said about war and the military – is too important to be left to lawyers alone.

#legalresearch #NLP