The towering inference inferno

Lately, coincidental and tangential prompts seem to have folded my thoughts back over over onto themselves in my sense-making about forms of inference in research.

It is without doubt that when we use inference (that is, when we reach conclusions), we do not really use one distinctly separate form over another. We need a combination of (at least) three types of inference in our sense-making as humans. It may well be that inference is how as a species we made the break from intelligence to self-consciousness in our evolution. Curiosity feeds our abilities to infer beyond what is under our noses, or what is apparent to the eye. In daily life how we reach conclusions might seem simple and straightforward, even a taken-for-granted, but as with everything else the closer we look into it the more detail we find (until, if we look closely enough, owe eventually discover that all we are capable of seeing this way is our labels, but that's another story).

Masters students must at some point in their degree get to grips with their getting to grips with inference. That is, at some point they will have to show their workings, not just their conclusions. This usually hits only at the end of the course, and in a panic, when designing (preferably) or finishing off (less preferably) the dissertation or Management Research Challenge (in Henley's vernacular). Defending their research "methodology" is an odd experience for a lot of managers because it's not really how they work at work. Nevertheless, an awareness of forms of inference is important as it provides rigour to the investigation of the issue or question. While for many MBA students a research design starts as a bit of a black box, anyone reading interview transcripts for long enough will find that they simply must adopt a way of making sense of all that data, not least because they have the practical goal of reaching a set of conclusions in order to hand the damned thing in and graduate.

Conclusions should flow from pattern inferred from, or in, the data and also by the thread established from reading the relevant management literature. At a fundamental level (i.e. "knowledge") sense-making is always about pattern. The question in a dissertation is whether you can isolate "a" particular pattern in an orderly, clear and defensible way. Would another person be able to follow you down the same path to replicate, test or further develop/explore what you found? More or less, that is what you're being asked to do in a dissertation. Along the way, you meet certain forks in the road, and these have been given signposts, not all of which are genuinely useful (or, at least, usefulness is over-stated). For example, "qualitative" versus "quantitative" has become a misleading signpost in social science because what most people call quantitative turns out to be a survey measuring a qualitative (analogue) relationship rather than a (digital) count. A more useful signpost might have "testing a prediction/explaining" versus "expanding understanding/exploring".

Knowing which way to turn leads to the use of inference. There are, I'd say, four ways to approach how you might look at your data, though in fact they're really all just different angles on the same thing:

1. You may start with an already known rule, which you then seek to find evidence of in your data. This uses deductive reasoning, where a single case is tested under a known rule. If it meets what the rule predicts, then the rule stands, for now. If it does not, then there is something wrong with the rule. This way of testing, however, does not reveal the new rule, it only shows that there is a limitation in the present one. The fluid and highly complex way that the social world operates makes deduction a bit of a minefield, as most "rules" (existing theories) are built on the most slender of axiomatic assumptions. However, as a way of exposing how flimsy most social theory is, deduction can be very useful. Deduction is designed to test the validity of a premise. It cannot deliver new knowledge.
2. You may generalise outward from a sample (a single case or set of cases) to reach a conclusion and form a rule, using the regular features present in your data. This is quantitative inductive reasoning, in which you are led to a generalisation. This is where the sheer weight of number/count of some quality in the sample implies the presence of those qualities in the population beyond the sample. The risk is that turning a single case into a rule carries a level of probability that your conclusion is not representative of a rule at all but may be built on chance, or bias, rather than reality. With quantitative induction, your thinking extends beyond what you see in the sample only in terms of qualities you see in your sample. It's a bit like an archaeologist digging and finding the top of a buried stone structure. Based on what they have found, they may confidently extend the trench in the expectation that the wall will continue.
3. You may find and arrange qualitative interpretations from your data in such a way as to conclude that other features (e.g. things that are already known from general theory) may also be at present in the general population, even though you do not see them directly in the sample. If you say you are a white wine drinker, I could with reasonable probability say that you also drink red, even if you didn't tell me this. This would be a qualitative induction, where I generalise outward from a set of clues (some in my sample, some not) to find a new version or formulation (a new angle, perhaps). It's likely that most MBA dissertations rely on qualitative induction to get to their conclusions. Going back to the archaeologist above, if she finds a piece of a buried wall, and combines the find with knowledge about the thickness and size of walls and buildings from other digs, she has the basis of an informed opinion as to what kind of site it is. This may lead her to test out the consequences of this conclusion by walking five meters along and digging where there might be a corner (which is a deductive action). Induction and deduction work hand in hand, but only to explore new versions of existing knowledge.
4. Finally, there is abductive inference, where you explore by deliberately trying to find a new way of explaining what in the data can't be explained by the first three above. It's for those parts of our understanding where we've sort of reached the limits or ceiling of what we know, or we suspect the 'orthodox' explanation to be wrong. Einstein was a great example of the abductive thinker. When he needed to break through the barriers to the laws of physics set down by, among others, Newton, he used his imagination to connect pattern for which there was no data (this is what I try to get at in the PD sessions on the MBA). When the world surprises us, or when we need to adopt an attitude of being prepared to be surprised by the world (which is more often than we accept), we start with abduction. Embrace the not knowing, abandon your old taken-for-granteds, and then relax. This can sometimes lead to a flash of insight, and a new rule or hypothesis jumps into our minds. I believe this is the creative process, and its what great artists, inventors, pioneers and thinkers have used. But it needs time, space, freedom and these are often what modern business cultures deny their leaders. 

In summary, abductive inference starts with a known result and then leaps over several walls to find a known rule from somewhere completely different and finds how they this gives a fresh or deeper insight (which is how metaphor works) and then new explanation of the case you see before you.