The relationship between chip computing power and deep learning

Fancy Wang 0111 2022

Deep learning algorithms become the current mainstream

The huge improvement of chip computing power and the emergence of big data have made "deep learning" with higher complexity in neural networks once again attract widespread attention.

The essence of deep learning is to build a neural network model with many layers and use a large amount of training data to let the machine learn important features, and finally achieve high accuracy in classification or prediction.

Deep learning is a perceptron-based model that mimics human mechanisms and the signal processing patterns of neurons, allowing computers to analyze data on their own to find feature values.

It is a natural idea to develop more hardware in order to speed up deep learning computations. Semiconductor manufacturers have developed some AI chips to handle deep learning. There are various models for deep learning. For example, the following three DNNs are commonly used in Google data centers.

1) Multilayer Perceptron (MLP) Each subsequent layer of an MLP is a set of non-linear functions, which are the sum of weights from all outputs of previous layers.

2) Convolutional Neural Networks (CNN). In CNN, each subsequent layer is a set of nonlinear functions, which are derived from the sum of weights of spatially nearby subsets of outputs from previous layers, and the weights are spatially multiplexed.

3) Recurrent Neural Network (RNN). Each subsequent layer of an RNN is a collection of weights, outputs, and nonlinear functions of the previous state. The most popular RNN is long short-term memory. (LSTM), the beauty of which is the ability to autonomously decide which states and which states to pass to the next layer. Weights are multiplexed in time.

One advantage of deep learning architectures compared to traditional artificial neural networks is that deep learning techniques can learn hidden features from raw data. Each layer trains a set of features based on the output of the previous layer to form a feature hierarchy. The innermost layer, which recombines the features of the previous layer, can identify more complex features.

For example, in the context of a face recognition model, raw image data of a human head as a vector of pixels is fed to the model in its input layer. Each hidden layer can then learn more abstract features from the output of the previous layer. For example: the first (hidden layer) identifies lines and edges; the second layer identifies faces, such as noses, glasses, etc.; the third layer combines all the previous features to generate a face image.

One of the reasons for the success of DNN is its ability to learn higher-level feature representations over successive non-linear layers. In recent years, advances in hardware and learning techniques for building deeper networks have further improved classification performance.

The ImageNet Challenge exemplifies the trend towards deeper networks. The state-of-the-art methods that have emerged in different years have grown from 8 layers (AlexNet) to 19 layers (VGGN) to 152 layers (Res Net) and 101 layers (ResNet). ). However, the development to deeper networks greatly increases the latency and power consumption of feed-forward inference.

However, most of the proposed improvements to deep learning are based on empirical evaluations, and there is still no concrete theoretical analysis basis to answer why deep techniques are superior to traditional neural networks.

Also, there is no clear boundary between DNNs and traditional neural networks regarding the number of hidden layers. In general, a neural network with two or more hidden layers and state-of-the-art training algorithms can be considered a deep learning model. However, RNNs with only one hidden layer are also relegated to deep learning because they have recurrent functions over the units of the hidden layer, which can be equivalent to DNNs.

We are a?Chinese?supplier?of network cards and modules (10/40/100G), direct?factory?resources, and provide a chain of?OEM?and?ODM?services?for well-known brands. Welcome to consult