Deep Belief Networks

What are deep belief networks ?

Deep belief networks (DBNs) are a type of deep learning model that consists of multiple layers of hidden units or neurons. They are a class of generative models that can learn to extract hierarchical representations of data. DBNs are typically composed of two main types of layers: the visible layer and the hidden layers.

The visible layer represents the input data, which could be, for example, an image, a sequence of words, or any other form of structured data. The hidden layers, on the other hand, capture increasingly abstract and complex features of the input data. Each layer is connected to the layer above and below it, but there are no connections between units within the same layer.

DBNs are typically trained using a technique called unsupervised learning, specifically a type of unsupervised learning known as restricted Boltzmann machine (RBM) training. RBMs are used to pretrain the layers of the DBN in a layer-by-layer fashion. Once the RBMs have been pretrained, the DBN can be fine-tuned using supervised learning methods, such as backpropagation, to perform tasks like classification or regression.

One key advantage of DBNs is their ability to learn hierarchical representations of data. The lower layers of the network capture low-level features, such as edges or corners in an image, while the higher layers learn more abstract concepts. This hierarchical representation allows the network to effectively model complex patterns and generate new samples that resemble the training data.

DBNs have been successfully applied to various tasks, including image and speech recognition, natural language processing, and recommendation systems. However, their training process can be computationally expensive and requires a large amount of labeled data. Advances in deep learning, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs), have gained more popularity in recent years, but DBNs still serve as an important foundation in the field of deep learning.

What is Restricted Boltzmann machine Training ?

Restricted Boltzmann Machine (RBM) training is an unsupervised learning technique used to pretrain the layers of a deep belief network (DBN) or to train RBMs as standalone models. RBMs are energy-based probabilistic models that learn to represent the joint distribution of visible and hidden variables.

The training process for an RBM involves iteratively adjusting the model's parameters to minimize the difference between the data distribution and the RBM's learned distribution. RBM training is typically performed using a technique called contrastive divergence (CD). Here's a high-level overview of the training process:

1. Initialize the RBM: Randomly initialize the weights and biases of the RBM.
2. Positive phase: Present an input data sample to the RBM's visible layer and compute the probabilities of the hidden units being activated given the visible units.
3. Negative phase (Gibbs sampling): From the activations of the hidden units in the positive phase, generate a "reconstruction" of the visible units by sampling from the conditional probabilities of the visible units given the hidden units. Then, use this reconstruction as input to the RBM and compute the probabilities of the hidden units being activated given the reconstructed visible units.
4. Update the RBM's parameters: Compute the differences between the positive and negative phases' expectations and adjust the weights and biases of the RBM using a learning rule such as gradient descent. The goal is to minimize the difference between the actual data distribution and the distribution represented by the RBM.
5. Repeat steps 2-4 for multiple iterations: Perform the positive phase, negative phase, and parameter update steps for a fixed number of iterations or until convergence is reached.

The CD algorithm is an approximation to the maximum likelihood learning for RBMs, which makes the training process computationally efficient. It helps RBMs learn to capture the statistical patterns and dependencies in the training data. After RBM training, the weights and biases of the RBM can be used to initialize the corresponding layers in a DBN or for other downstream tasks.

It's important to note that RBM training is an unsupervised learning technique, meaning it does not require labeled data. It focuses on learning the underlying structure and patterns in the data without explicit target information.

What is contrastive divergence traning ?

CD training approximates the full Markov chain Monte Carlo (MCMC) method called Gibbs sampling, where the Markov chain is allowed to reach equilibrium. Instead of waiting for full convergence, CD training performs just a few steps of Gibbs sampling. This approximation makes the training process computationally efficient while still enabling the RBM to learn meaningful representations of the data.

Overall, contrastive divergence training provides a practical and effective way to train RBMs by approximating the maximum likelihood learning procedure through the use of Gibbs sampling.

What is Gibbs Sampling ?

Gibbs sampling is a technique used to generate samples from a joint probability distribution when it is difficult to directly sample from the distribution. It is particularly useful for models with complex dependencies and high-dimensional spaces.

Here's a brief explanation of Gibbs sampling:

1. Initialize the variables: Start with an initial configuration of the variables in the joint distribution.
2. Iterate through the variables: Select a variable and update its value based on the values of the other variables in the distribution.
3. Sampling: Given the current values of all other variables, sample a new value for the selected variable according to its conditional probability distribution given the other variables.
4. Repeat steps 2-3: Continue iterating through the variables, updating and sampling, until convergence or a desired number of iterations.

Gibbs sampling exploits the fact that, at equilibrium, the joint distribution can be decomposed into conditional distributions of individual variables given the values of the other variables. By iteratively sampling each variable conditioned on the others, the Markov chain generated by Gibbs sampling gradually explores the entire joint distribution and converges to a stationary distribution.

Through this iterative process, Gibbs sampling can be used to generate samples that approximate the true distribution. These samples can then be used for various purposes such as estimating statistics or performing inference in probabilistic models.

Gibbs sampling is a fundamental technique in the field of Markov chain Monte Carlo (MCMC) methods and is widely applied in areas such as Bayesian statistics, machine learning, and statistical physics.

What are Markov chain Monte Carlo methods ?

Markov Chain Monte Carlo (MCMC) methods are a class of algorithms used to sample from probability distributions. They are particularly useful when it is difficult or impractical to directly sample from a distribution, but it is possible to evaluate the probability density function (pdf) up to a constant factor.

MCMC methods use the concept of Markov chains, which are mathematical models that describe a sequence of random variables, where the probability of each variable depends only on the previous value in the sequence. In the context of MCMC, the variables in the Markov chain represent different states or configurations of the system being modeled.

What is joint probability distribution ?

The joint probability distribution refers to the probability distribution of a set of random variables. It provides information about the simultaneous occurrences or combinations of events involving multiple random variables.

Let's consider an example of rolling two fair six-sided dice. We can define two random variables, X and Y, representing the outcomes of the first and second dice, respectively.

The joint probability distribution for this example can be represented using a joint probability table. The table will have rows corresponding to the possible outcomes of the first die (X) and columns corresponding to the possible outcomes of the second die (Y). Each entry in the table represents the probability of the specific combination of outcomes.

Here's an example of a joint probability table for the two dice:

? ?| 1? ? 2? ? 3? ? 4? ? 5? ? 6?
-------------------------------
1? | 1/36 1/36 1/36 1/36 1/36 1/36
2? | 1/36 1/36 1/36 1/36 1/36 1/36
3? | 1/36 1/36 1/36 1/36 1/36 1/36
4? | 1/36 1/36 1/36 1/36 1/36 1/36
5? | 1/36 1/36 1/36 1/36 1/36 1/36
6? | 1/36 1/36 1/36 1/36 1/36 1/36
?        

Major Applications of Deep Belief Networks ?

1. Image and Speech Recognition: DBNs have been used for tasks such as image classification, object recognition, and speech recognition. They can automatically learn hierarchical representations from raw data, allowing them to capture complex patterns and features in images and audio.
2. Natural Language Processing: DBNs have been employed in natural language processing tasks such as text classification, sentiment analysis, and language modeling. They can learn representations of text data that capture semantic relationships and contextual information.
3. Recommender Systems: DBNs have been used in recommender systems to provide personalized recommendations to users. By learning patterns and preferences from user behavior data, DBNs can effectively model user-item interactions and make accurate recommendations.
4. Drug Discovery: DBNs have been applied in the field of drug discovery to assist in the identification of potential drug compounds. They can learn complex relationships between molecular structures and biological activities, helping in the prediction of drug efficacy and toxicity.
5. Anomaly Detection: DBNs can be used for detecting anomalies or outliers in various types of data, such as network traffic, financial transactions, or system logs. By learning the normal patterns from the training data, DBNs can identify deviations that indicate unusual or suspicious behavior.
6. Generative Modeling: DBNs can be employed for generative modeling tasks, such as generating realistic images, music, or text. By training on a large dataset, DBNs can learn the underlying distribution of the data and generate new samples that resemble the training examples.
7. Reinforcement Learning: DBNs have been used in combination with reinforcement learning algorithms to improve the learning and decision-making capabilities of agents in complex environments. DBNs can help in discovering meaningful representations of states and actions, leading to more efficient and effective learning.

Python code to build a one layer deep belief network from scratch !

import numpy as n


class DeepBeliefNetwork:
? ? def __init__(self, input_size, hidden_size, output_size):
? ? ? ? self.input_size = input_size
? ? ? ? self.hidden_size = hidden_size
? ? ? ? self.output_size = output_size


? ? ? ? # Initialize weights and biases
? ? ? ? self.weights1 = np.random.randn(input_size, hidden_size)
? ? ? ? self.biases1 = np.zeros(hidden_size)
? ? ? ? self.weights2 = np.random.randn(hidden_size, output_size)
? ? ? ? self.biases2 = np.zeros(output_size)


? ? def sigmoid(self, x):
? ? ? ? return 1 / (1 + np.exp(-x))


? ? def forward_pass(self, X):
? ? ? ? # Hidden layer activation
? ? ? ? hidden_activation = np.dot(X, self.weights1) + self.biases1
? ? ? ? hidden_output = self.sigmoid(hidden_activation)


? ? ? ? # Output layer activation
? ? ? ? output_activation = np.dot(hidden_output, self.weights2) + self.biases2
? ? ? ? output = self.sigmoid(output_activation)


? ? ? ? return hidden_output, output


? ? def train(self, X, y, learning_rate, epochs):
? ? ? ? for epoch in range(epochs):
? ? ? ? ? ? # Forward pass
? ? ? ? ? ? hidden_output, output = self.forward_pass(X)


? ? ? ? ? ? # Backpropagation
? ? ? ? ? ? error = y - output
? ? ? ? ? ? output_delta = error * output * (1 - output)
? ? ? ? ? ? hidden_error = np.dot(output_delta, self.weights2.T)
? ? ? ? ? ? hidden_delta = hidden_error * hidden_output * (1 - hidden_output)


? ? ? ? ? ? # Update weights and biases
? ? ? ? ? ? self.weights2 += learning_rate * np.dot(hidden_output.T, output_delta)
? ? ? ? ? ? self.biases2 += learning_rate * np.sum(output_delta, axis=0)
? ? ? ? ? ? self.weights1 += learning_rate * np.dot(X.T, hidden_delta)
? ? ? ? ? ? self.biases1 += learning_rate * np.sum(hidden_delta, axis=0)


? ? def predict(self, X):
? ? ? ? _, output = self.forward_pass(X)
? ? ? ? return output