BxD Primer Series: Restricted Boltzmann Machine Neural Networks

Hey there ??

Welcome to BxD Primer Series where we are covering topics such as Machine learning models, Neural Nets, GPT, Ensemble models, Hyper-automation in ‘one-post-one-topic’ format. Today’s post is on?Restricted?Boltzmann Machine Neural Networks. Let’s get started:

The What:

Restricted Boltzmann Machine (RBM) is a type of energy-based unsupervised learning model which addresses some of the limitations of?Boltzmann Machines. It consists of two layers of nodes: visible layer and hidden layer. The visible layer represents input data, while hidden layer learns to represent underlying patterns in data. After network training, the visible layer acts as output layer.

In an RBM, there are no connections between?nodes within same layer. Connections between visible and hidden layers are undirected, which means that information can flow in both directions. Weights between the nodes in visible and hidden layers are learned during training, using a technique called contrastive divergence.

It is a type of generative model used to generate new data that is similar to training data. They are also used for anomaly detection, dimensionality reduction, and collaborative filtering.

Main difference between Boltzmann Machines and RBMs are:

• BMs have fully connected layers, which means that there are?connections between all units?in the network. This makes BMs computationally expensive to train as the number of units in network increases.
• RBMs have a simpler structure, with?connections only between visible and hidden layers. This makes RBMs more computationally efficient and easier to train.
• RBMs are primarily used for unsupervised learning tasks, whereas BMs can be used for both supervised and unsupervised learning tasks.

Note: Unlike some other neural network architectures, such as feedforward neural networks, RBMs do not use back-propagation to update weights. Instead, RBMs are typically trained using a technique called contrastive divergence, which involves updating weights in the direction that maximizes difference between probabilities of input data and reconstructed data.

Two layers in a RBM:

The visible layer is composed of a set of input nodes, which represent the features or attributes of input data. Each input node is connected to every node in hidden layer. The connections between input and hidden layers are weighted and the strength of connection determines influence of input on hidden layer.

The hidden layer is composed of a set of hidden nodes, which are not directly connected to each other. There are no connections between nodes within hidden layer. Instead, each hidden node is connected to every input node in visible layer.

During training, RBM learns to adjust the weights of connections between visible and hidden layers in order to maximize the likelihood of training data. By doing so, RBM learns to capture underlying patterns and relationships in input data.

The How:

Training process of a RBM is same as that of BM, which we already covered in a previous edition, please check?here.

Here are the general steps:

1. Initialize the RBM?by setting weights and biases of visible and hidden units to random values.
2. Present the RBM with?input data?which consists of binary or continuous values representing visible units.
3. Calculate activation probabilities?of hidden units?based on input data and current weights and biases of visible and hidden units. Sigmoid activation function is typically used for this.
4. Once the activation probabilities of hidden units are calculated, RBM?samples?from these probabilities to obtain the states of hidden units.
5. Calculate?activation probabilities of visible units?based on the states of hidden units and weights and biases of visible and hidden units.
6. Similar to step 4, RBM?samples?from the activation probabilities of visible units to obtain the states of visible units.
7. Steps 3 to 6 are?repeated?for a fixed number of iterations or until convergence is reached.

Binary RBM v/s Gaussian RBM:

RBMs can be divided into two main types: Binary RBMs and Gaussian RBMs. Main difference between the two is the type of units used in visible layer of the network.

In a Binary RBM, visible units are binary and can only take on values of 0 or 1. They are useful for collaborative filtering (used to recommend items to users based on their preferences) type tasks, where input data is composed of presence/absence records.

In a Gaussian RBM, visible units are continuous and can take on any real value. It allows the network to model continuous data, such as audio signals or sensor data. They are useful in speech recognition and natural language processing type applications.

The hidden units in both types of RBMs are typically binary, although it is possible to use continuous hidden units as well.

Difference between the two is also reflected in the way probabilities are calculated.

• In binary RBM, the probability of a visible unit taking on a value of 1 is calculated using the logistic function.
• In Gaussian RBM, the probability is calculated using a Gaussian distribution with a mean and standard deviation.

The Why:

Reasons for using RBMs:

1. Capable of modeling high-dimensional inputs with non-linear relationships between variables.
2. Can be trained in a distributed manner using?parallel tempering?technique.
3. Can be used for generative modeling, in applications such as image and audio generation.
4. Can learn useful feature representations of input data, which can improve the performance of downstream tasks such as classification or regression.
5. Can be combined with other types of neural networks, such as convolutional neural networks (CNNs) or recurrent neural networks (RNNs), to form hybrid architectures.

The Why Not:

Reasons for not using RBMs:

1. Not well-suited for modeling sequential data because they lack a recurrent connection that allows them to model dependencies over time.
2. RBMs are designed to work with binary data and are not as effective for continuous data. Gaussian RBMs are one solution to this problem, but they are still not as effective as other deep learning models.
3. It is often difficult to understand the underlying factors that contribute to model's output.
4. RBMs may not be as effective on small datasets as they require large amounts of training data to learn meaningful representations.

Time for you to support:

1. Reply to this email with your question
2. Forward/Share to a friend who can benefit from this
3. Chat on Substack with BxD (here)
4. Engage with BxD on LinkedIN (here)

In next edition, we will cover Deep Belief Neural Networks.

Let us know your feedback!

Until then,

Have a great time! ??

#businessxdata?#bxd?#Restricted #Boltzmann #Machine #neuralnetworks #primer