Artificial Intelligence #64: Statistical inference: A good way to understand the mathematical foundations of machine learning

This week, I spent time in the ancient and beautiful Jesus College at the University of Oxford - discussing among other things, the maths of AI for engineering.?

I loved the ancient library!

We are soon launching the next cohort of Artificial Intelligence: Cloud and Edge implementations at the #universityofoxford?

Over the years, I have taken a maths based approach to AI. This is not easy but I like it since its such a sound way to teach AI fundamentals.?

The key is to teach the maths of AI in specific concepts and link them to underlying maths principles?

One such concept is statistical inference?

I have discussed about this before in

?Machine learning vs statistics?

Statistical inference vs predictive inference?

How do we combine statistical thinking and machine learning?

In this post, I explore the idea of statistical inference further and how it helps to unite many ideas

You should first read the three posts above, especially statistical inference vs predictive inference.

Specifically, we are talking here of statistical inference (distinct from predictive inference) which is what we normally mean in machine learning parlance.

Firstly, while we think of probability theory and statistics together, they represent different aspects of the same problem i.e. the understanding of stochasticity or randomness. In probability theory, we capture the randomness underlying a process and then try and model what can happen. In statistics, we observe what is happening and? try to explain the underlying process that explains the phenomenon.?

We start with the idea of a statistical proposition.? Broadly, a statistical proposition is a statement that can be either true or false? - a good reference documentary HERE?

Hence, statistical inference infers the properties of an underlying proposition?

What type of propositions ?

Some common forms of statistical proposition are the following: (source wikipedia)

• a point estimate, i.e. a particular value that best approximates some parameter of interest;
• an interval estimate, e.g. a confidence interval (or set estimate), i.e. an interval constructed using a dataset drawn from a population so that, under repeated sampling of such datasets, such intervals would contain the true parameter value with the probability at the stated confidence level;
• a credible interval, i.e. a set of values containing, for example, 95% of posterior belief;
• rejection of a hypothesis
• clustering or classification of data points into groups.

Also, there are two main schools or paradigms of statistical inference: Frequentist paradigm and Bayesian paradigm

Frequentist inference calibrates the plausibility of propositions by considering (notional) repeated sampling of a population distribution to produce datasets similar to the one at hand. By considering the dataset's characteristics under repeated sampling, the frequentist properties of a statistical proposition can be quantified—although in practice this quantification may be challenging.

Examples of frequentist inference are

• p-value
• Confidence interval
• Null hypothesis significance testing

Bayesian inference describes degrees of belief using the 'language' of probability; beliefs are positive, integrate into one, and obey probability axioms. Bayesian inference uses the available posterior beliefs as the basis for making statistical propositions.

Examples of Bayesian inference are

• Credible interval for interval estimation
• Bayes factors for model comparison

Thus, the idea of inference unites statistics and machine learning if we consider that statistical inference and predictive inference refer to related but distinct things.

In subsequent posts, we can explore this idea further by considering black box models, highly parameterised models and parameter estimation techniques.?

References: wikipedia - statistical inference