Lies, Damn Lies and Statistics

 * My standard disclaimer: While this opinion piece is written specifically for paralegals for the 2019 LSO Bencher Election for which I am a candidate, I also write for other licensees, other professional, laymen and anyone interested. Please have patience if it seems a bit pedantic or discusses what you might already understand. Reducing complex issues to clear and exacting ideas requires it.

Lies, Damn Lies and Statistics

The Study and the Onus of Proof: In order to overcome the ‘reasonable limit in a free and democratic society’, the LSO puts up this study as evidence of the widespread systemic racism in the legal profession. Be clear here, the onus of proof is on the LSO to make that case. They have to show a NEED to infringe on the right of free speech. 

Ok then ... let’s look at this study. There is a bit of terminology to understand and a few broad paradigms to consider when evaluating any study. While all this isn’t rocket science and nothing a layman can’t understand, a person still has to think through it carefully to get at the relevant truth. Here are some things to consider ...

Gestalt: I co-opt the term Gestalt for this discussion. This is the background, premises and assumptions that are inherit in any survey or study. For example, who is motivated to be involved in the study, whose interests is the study meant to serve, does the terminology embedded in the study let alone how the questions are asked to imply a favoured position? Look at the way I titled this post for example. I’ve always liked the quote but, in this case, it sets you up to be predisposed to believing that there is something wrong with the survey(s) the LSO used to make their claim of widespread systemic racism.

Example Study of Suicide: For an academic example of Gestalt, Emile Durkheim, I think one of the first sociologists did a study of suicides in Europe and found that statistically Catholics had fewer suicides per capita than Protestants. But on analysis it was postulated that the numbers were skewed because the data was taken from coroner’s reports and since Catholic neighborhoods were more likely to have a Catholic coroner, who by virtue of his being Catholic would understand the purgatory implications of declaring the person committed suicide. As a result, possibly facing pressure from family members not to declare the son or daughter as having committed suicide, the coroner notes the death as something other than a suicide. The point here is to show that cultural or other factors not easily noticed (or wanted to be noticed) can affect the result.

Methodology: At the opposite pole, there is a cliché that says if you go looking for something ... you will find it. The methodological propriety is crucial in any study to make sure the study gives you accurate data. Confirmation bias, for example, is a well understood phenomena and, in this study the question should be asked, does starting with the premise that there is widespread systemic racial discrimination in the legal profession put the whole exercise at risk?

Data pool, sample size & extrapolations are all methodological considerations – not everyone is in a target pool so surveys are done by taking answers from a percentage of the population and then extrapolating out to 100%. Obviously, the smaller the sample the less reliable the survey. That’s why they say ‘A sample of one does not a survey make’.

Terminology (jargon): Used not just in the study ‘explanatory notes’ (and there is a lot) but also in the questions can couch or frame the discussion in certain terms so that it can pre-load an expectation in the respondent ... even to the point of making it so that only those that know and understand the jargon even respond to the survey. 

Validity: Does the tool you are using to measure the thing you want to measure actually does measure the thing you want to measure? For example, a meter stick is a valid tool to measure distance but not valid to measure weight.

Reliability: Does the tool you are using measure the thing you are trying to measure accurately? For example, is your meter stick actually one meter long?

You have all heard the disclaimer at the end of a survey that says something like ... ‘This survey is accurate within plus or minus X% ... 19 out of 20 times. What they are saying there is that most of the time (19 of 20 times in fact) that measurement they are using is valid. 1 in 20 times the measurement is invalid ... ie ... for some reason the tool used didn’t measure the thing that is was intended to measure. Maybe the questions were meant to measure a group’s sympathy but instead (for 1 in 20 groups) the responses given actually were a reflection of the group’s special knowledge or some other causal factor. 

Correlation: Two more events happen in apparent connection but note the presence of a correlation is not sufficient to infer the presence of a causal relationship (i.e., correlation does not imply causation).

Spurious Relationship: When some things happen at the same time but one does not cause the other. For example, if you looked at the stats you would find that sales for ice cream go way in ‘correlation’ to the number of weddings. You might conclude that a lot of people have ice cream at weddings but this is a spurious relationship. The real reason is that both occur mostly during the summer and not that one causes the other.

Saliency: there are two dimensions to consider when asking peoples’ opinions. The first relates to their belief, the second relates to how strongly they hold that belief. So for example, if you asked me ... yes or no ... if I thought that we should keep the Queen as head of state in Canada, I might say yes. But if you then asked if I would go to war to keep the Queen ... ummm, I might then say no. So, the saliency of the question to the person is a factor in determining the weight of a given question or answer on a survey.

Conclusions: Certainly, a review of the conclusions drawn from the data that is gathered needs to be considered. Are the conclusions sequitur from the data (a reasonable inference from the information obtained), are there more than one conclusion that can be drawn, have spurious relationships been filtered out and again how is saliency of the response considered.

Humorous anecdote here ... two paralegals own property across the street from each other and every morning the neighborhood gets woken by the two of them yelling across the street at each other from their respective balconies. A new resident in the neighborhood hears this and asks the postman ... ‘What’s with those guys ...?’ ‘Oh that’s Mr. Jones & Mr. Smith ... they are always arguing but they’ll never agree with each other’ comes the reply. â€˜Why not ... surely they can agree on something.’ â€˜No ... they always argue from different premises.’  

Perspective: More accurately difference in perspective (part of the Gestalt) factors in at every stage of this analysis. Perspective can make a difference in the questions, the answers and the conclusions drawn. The best known cliché might be the one about ‘half empty – half full’. Both are correct in that they are both factually accurate but the conclusions drawn from each can be different are different based on perspective. 

Open ended questions: These are problematic for drawing definitive conclusions because they are open to subjective interpretation. There is a difference between ‘hard’ evidence (say a document with a date and signature) and ‘soft’ evidence (he said – she said type). This means that many of the conclusions are non-sequitor or at least not sufficiently proven. The inference they take home is that because a person says they felt like a particular act by someone else ...’people whisper when I walk into the courtroom’ or ‘I’m often mistaken for a clerk or client and not a lawyer’ it was related to racism. This does not prove it to be true. 

Hearsay rule as a metaphor: Putting on my legal practitioner hat, I’m reminded about the hearsay statement rules. The restriction relates to the statement being evidence of ‘truth of the contents’ but not to the fact the statement was made. So, these type of ‘feeling’ statements are not evidence that there is in fact racism but only that the person felt there was racism. I am not saying that there is no racism but just that it doesn’t follow from the evidence this survey provides. The conclusion drawn by the LSO, and that they want enshrined as an unassailable truth however, is racism not only exists but is rampant enough to warrant overriding everyone’s fundamental right to free speech. 

Perspective Affects the Conclusion: Even if we accept at face value that the survey found... “of ‘racialized’ respondents 45% said they believe they were subject to some form of racism’, doesn’t that mean that 55% felt they were not so subject. Your ‘take away’ then is clearly dependent on the perspective you have.

Different Paradigms: In fact perspective affects the whole paradigm of the issue ... For example, is the ‘approrpriate’ number 45% or if not what is the approriate number? I mean is it that we have to get that number down to 20%? ... 15% or what? The question is unanswerable in that there is actually no answer that you can say is the RIGHT answer in terms of public policy. Let me explain.

We all love mom and maple syrup and we all want Access to Justice right? Part of that is the cost of suing someone. But how easy do we really want to make it? On one hand we don’t want to make it so inexpensive that people can sue each other at the drop of a hat. On the other hand, we don’t want to make it too expensive to make it a barrier. But how much is too much and how little is too little. The answer depends on your perspective and there is no objectively right answer.

The point is that when proponents of the forced statement of principles say ‘racism is rampant in the legal system’ it is clearly a perspective based on a perceived ‘right’ number that they happen to have in mind. However, this is put forward as a kind of unchallengeable fact.

Type of Research: This affects the ‘weight’ of the evidence put forward assuming the study passes all the other critical considerations listed above.  

Primary research in social sciences is done by survey and/or questionnaire where they formulate the questions, conduct the survey and then draw conclusions. The SoP study was based on primary research. Certainly this has some value but it is only one study.

Secondary research relates to how a question or series of questions is formulated and then the researcher searches and reviews all the relevant primary research that is available. Conclusions are then derived from the different sources. Although not infallible is a type of ‘independent verification’ that gives considerably more weight to the conclusions. The LSO study does not have that weight.

But does the LSO say ... you know ... this is only one study but here’s what we found? No, they make seemingly definitive statements about systemic racism in the legal profession with seeming disregard for giving a balanced account of the considerations involved.

In contrast, I haven’t talked much about the actual numbers nor drawn conclusions from those numbers at all. All I said is that use of studies and statistical analysis is complex fraught with fallacy, misconception and subjective interpretation.  I put this forward so that all these factors can be considered when deciding if the LSO has met the burden needed to impede on the fundamental right of free speech.