What distinguishes a directed graphical model from an undirected model?

1 Directed models

Directed graphical models, also known as Bayesian networks or belief networks, use directed acyclic graphs (DAGs) to encode conditional dependencies among random variables. A DAG is a graph where each node represents a random variable and each edge represents a direct influence from one variable to another. The direction of the edge indicates the causal relationship between the variables. For example, in a DAG that models the weather, the season node might have an outgoing edge to the temperature node, indicating that the season affects the temperature.

The main advantage of directed models is that they can capture causal relationships and allow for intuitive reasoning and interpretation. They can also exploit conditional independencies to simplify the joint distribution of the variables. The main challenge of directed models is that they require specifying the causal direction for every edge, which might not be always clear or available. They also have to deal with issues such as cycles, latent variables, and confounding factors.

2 Undirected models

Undirected graphical models, also known as Markov networks or Markov random fields, use undirected graphs to encode marginal dependencies among random variables. An undirected graph is a graph where each node represents a random variable and each edge represents a direct or indirect association between two variables. The absence of direction means that the edge does not imply causality or influence, but only correlation or co-occurrence. For example, in an undirected graph that models the weather, the season node and the temperature node might be connected by an edge, indicating that they are correlated, but not necessarily causally related.

The main advantage of undirected models is that they can represent symmetric and complex dependencies without imposing a causal structure. They can also handle cycles, latent variables, and confounding factors more naturally. The main challenge of undirected models is that they require specifying the potential functions that define the strength of the associations between the variables, which might be hard to estimate or interpret. They also have to deal with issues such as normalization, inference, and learning.

3 Equivalence and conversion

Directed and undirected models are not mutually exclusive, but rather complementary ways of expressing probabilistic relationships. In fact, under certain conditions, they can be equivalent or converted to each other. Two graphical models are equivalent if they encode the same set of conditional independencies among the random variables. For example, a directed model with no cycles, no latent variables, and no confounding factors is equivalent to an undirected model with the same structure and potential functions proportional to the conditional probabilities.

Converting a directed model to an undirected model is called moralization. It involves adding edges between all the parents of each node and making the graph undirected. Converting an undirected model to a directed model is called triangulation. It involves adding edges between all the non-adjacent nodes in each cycle of length greater than three and making the graph directed. However, these conversions are not always unique or meaningful, and they might introduce additional dependencies or lose causal information.

4 Applications and examples

Both directed and undirected models have a wide range of applications in various domains, such as computer vision, natural language processing, bioinformatics, and social network analysis. Examples of directed models include Naive Bayes, Hidden Markov Model, and Bayesian Network. Naive Bayes is a simple and efficient model that assumes conditional independence among the features given the class label. Hidden Markov Model is a sequential model that captures the temporal dependencies between hidden states and observed outputs. Bayesian Network is a general and flexible model that can represent any DAG structure and perform probabilistic inference and learning. Examples of undirected models are Ising Model, Boltzmann Machine, and Markov Random Field. Ising Model is a simple and classical model that describes the interactions between binary variables in a grid. Boltzmann Machine is a neural network model that learns the joint distribution of binary variables using stochastic gradient descent. Lastly, Markov Random Field is a general and flexible model that can represent any undirected graph structure and perform probabilistic inference and learning.

5 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?