Importance of connecting nodes in NN

When we talk about fully connected neural network, we think about some input going through few linear transformation phase and transporting to be the form of output. Throughout the training process the linear transformation coefficients get updated.

Designing?multi?layer?perceptron?(a?form?of?neural?network), we carefully craft two aspects:

1. How many nodes will be there in each layer, and how many layers should be there in the network

2. How should we initialize the node weights and bias, and what should be the optimizer and loss function

From the deployment perspective, it is crucial to consider the complexity of the network, and the aim is to make the networks computationally lightweight. In that case, it is important to understand how we are calculating neural network time complexity and space complexity.

For a fully connected neural network, the matrix multiplication each linear layer primarily contributes to complexity. If there is N nodes, the complexity is O(N).

For MLP type of networks, if we want to make the network competitionally lightweight, we have to decrease the of nodes. More specifically, if number of layers can be mitigated, the computation is lightweight.

Throughout the literature review, we observe many attempts for lightweight neural network, such as Fuzzy neural network (2), Probabilistic Neural Network (3), U-Net?, Densenet, Resnet, RNN.

Now, we observe some common patterns while designing the connection of nodes.

1. Connecting nodes of two different layers (UNet, Resnet)
2. Expanding(Fuzzy Neural Network) or Concising (Probabilistic Neural Network) a certain layer
3. Connecting nodes within the same layer (RNN)

We are optimizing between two aspects:

1. Making the network computationally lightweight
2. Making the training efficient i.e. keeping in mind the loss?, optimization function

For Network like U-Net, let’s first observe the structure:

As we can see, In Fig: UNet Structure, first it is downsampled, then it is upsampled with the previous samples and the first samples. For example if convolutional layer is being used, in the initial layers the image is less filtered, so we have an idea about the temporal location, which in later filters get lost i.e. the later filters have more focus on the certain target. If our target is a function of relative location i.e context conventional fully connected network will not be much helpful. U-net works well providing context of the target in the network(5). Thus no additional layers are required for computing the location.

What of the target is the function of time as well. We can take ECG signal for example. It is crucial to keep track of what is happening in a few of the previous events and future events. In the paper (7), there is a detailed consideration of how to keep track of R-R peaks for heart rate estimation

Here CNN helps to get rid of the other

signal artifacts. But use of LSTM (RNN) helps to keep the relative distance information, which helps to measure the heart rate.

Let’s understand Fuzzy neural network and It’s application. Instead of node connection, it breaks down a certain layer to different steps. But how does this help, and what are the applications

For example let’s take application of various heart disease detection (8). From Fig: Fuzzy probabilistic network, we can see a combination of fuzzy and probabilistic neural network approach. When we observe the connections of the next layer, we can see skip connections and how the clustering varies depending on how the threshold is set in fuzzy layer. Here, the fuzzy layer thresholding helps the network reduce tje dimensionality of the matrix multiplication for the next layers which essentially reduce the complexity of that layer. Whereas thresholding is O(1) time complexity since it can be done while computing the result of the previous fully connected layer and O(N) space complexity

This article tries to provide variations of node connections in neural network seen in research. To conclude, it is evident how the artificial neural network is designed depends on the application and the relationship between the input and target.

Reference

1. Answer to What is the time complexity of a trained neural network? by Dr. S. Pradeep https://www.quora.com/What-is-the-time-complexity-of-a-trained-neural-network/answer/Dr-S-Pradeep?ch=15&oid=1477743672059475&share=24034786&srid=5NTef&target_type=answer
2. https://images.app.goo.gl/niQ3DqQ63y9hwE1F7
3. https://www.researchgate.net/publication/342250251_An_Inspired_Machine-Learning_Algorithm_with_a_Hybrid_Whale_Optimization_for_Power_Transformer_PHM
4. https://arxiv.org/pdf/1802.06955#:~:text=The%20U%2DNet%20model%20provides,for%20segmentation%20tasks%20%5B12%5D
5. https://towardsdatascience.com/unet-line-by-line-explanation-9b191c76baf5
6. https://www.researchgate.net/publication/374807775_Privacy_Preserving_Heart_Rate_Estimation_from_ECG_and_PPG_Signals_for_Application_in_Remote_Healthcare
7. https://www.researchgate.net/publication/374807775_Privacy_Preserving_Heart_Rate_Estimation_from_ECG_and_PPG_Signals_for_Application_in_Remote_Healthcare
8. https://www.researchgate.net/publication/353878492_Handling_Vagueness_in_Heart_Disease_Detection_Using_Fuzzy_Particle_Swarm_based_Probabilistic_Neural_Network