A Diversion from E. T. Jaynes

Yes. I said it was time to move onto other applications of Bayes' theorem. I will get to using Bayes' theorem to solving problems; however, permit me a digression to explain my passion for a Bayesian approach.

Early in my exploration of applications of Bayes' theorem, I came across the book, Probability Theory - The Logic of Science by E. T. Jaynes. Perhaps his approach spoke to me as a fellow physicist. As I read, I recognized the same logic employed by my physics professors as they taught me the details of the domain I had been pursuing since I was in third grade.

Jaynes begins by describing our messy world; that is, a world not merely of "Yes" and "No," but of many, many, many more "Maybe's". In this world, one can, in certain cases, perform Boolean reasoning; however, much of our day-to-day life cannot be described by "Yes" and "No," but by "Almost never", "Unlikely," "Probably," and "Almost certainly".

This day-to-day language is the language of plausibility. Further, Jaynes argues that this day-to-day language is not merely vague and ambiguous words. No, me can actually map this language of everyday situations onto the mathematical concept called probability. He introduces the process of mapping our informal, plausible reasoning onto a mathematical foundation. He provides axioms, postulates, and theorems and then investigates the mathematical consequence of these ideas.

Along the way, he introduces a robot, programmed by us, but able to reason probabilistically; that is, the robot can determine the mathematical consequences of prior information about the world and additional data it is fed.

Why does this matter?

When I first began investigating the use of Bayes' theorem, the debate about the consequence of the dependence of Bayesian inference on prior data was raging. Some, "the frequentists," argued, "How can you depend on the implicit and differing beliefs of people to perform reasoning?" Others, "the Bayesians," argued, "How can you not rely on all information available to people to perform reasoning?" Although the passion of this argument has lessened, Jaynes began to put the argument to rest by putting the reasoning from prior beliefs on a reasonable and mathematical footing. Further, he was able to demonstrate that many of the conclusions reached by "the frequentists" were simply special cases (like the Special Theory of Relativity) of the Bayesian approach.

I'll continue to refer to Jaynes' work in the future as we approach different problems. I encourage you to read the book especially if you come from a "hard" scientific background. It will help you to both understand the broad application of Bayesian techniques but to see that conclusions based on applications of Bayes' theorem are reasonable, rational, and perhaps, even "rigorous."