Neural Network in C Part 1: Idea

Welcome, normally I enjoy programming in Python and using the TensorFlow library to create machine learning applications. I was able to create a website that loads the popular MNIST dataset and use user input to determine what digit is being drawn on a canvas.

I had a vision about really upping the game when it came to machine learning. How do we learn something? We build it ourselves! Why not create our own machine learning algorithm from scratch? Of course it has to be blazing fast and accurate. Lets use the C programming language for this.

The idea is this:

1. download.py will download the MNIST data and save it in a binary format. This will consist of training images and training labels.
2. neural_network.c will be our implementation of a simple neural network. We should execute this with the gcc compiler.

Now we know that each image in the MNIST dataset is 28x28 pixels. This should mean that there are 784 input neurons. The input layer will take the raw pixel values of input images.

Take a look at this sample digit from the MNIST dataset, I was able to label each pixel. This is our input layer here.

The input layer will receive the raw pixel values of the image.

The hidden layer will consist of 128 neurons. This has been found to work well through experimentation. 128 neurons is a good balance between computation time and risk of over fitting.

The output layer is 10 neurons representing digits 0-9.

So lets imagine a 28x28 grid of 784 pixels connecting to 128 hidden layers and then outputting to 10 output layers. Looks like this:

We can go ahead and prepare our data today, and later I will update the C code.

Lets get to work on download.py. Go ahead and import numpy, tensorflow(not cheating just need the MNIST dataset), os, and struct libraries. Create a method to save the MNIST binary and a main method.

import numpy as np
import tensorflow as tf
import os
import struct
def save_mnist_binary(images, labels, image_path, label_path):
    with open(image_path, 'wb') as img_file, open(label_path, 'wb') as lbl_file:
        img_file.write(struct.pack('>IIII', 2051, len(images), 28, 28))
        lbl_file.write(struct.pack('>II', 2049, len(labels)))
        for image, label in zip(images, labels):
            img_file.write(image.tobytes())
            lbl_file.write(struct.pack('B', label))
def main():
    (train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.mnist.load_data()
    images = np.concatenate((train_images, test_images), axis=0)
    labels = np.concatenate((train_labels, test_labels), axis=0)
    images = (images / 255.0 * 255).astype(np.uint8)
    save_path = './mnist_data'
    if not os.path.exists(save_path):
        os.makedirs(save_path)
    save_mnist_binary(images, labels, os.path.join(save_path, 'train-images-idx3-ubyte'),
                      os.path.join(save_path, 'train-labels-idx1-ubyte'))
if __name__ == "__main__":
    main()

        

This Python code is designed to load the MNIST dataset, process it, and then save it in a specific binary format. The method save_mnist_binary() will write binary numbers and sizes. The main() method loads the dataset, combines the training and testing datasets, normalizes the images, and saves them in a binary format.

Executing this code gives a lot of information.

/home/bot/PycharmProjects/neural/.venv/bin/python /home/bot/PycharmProjects/neural/download.py 
2024-06-19 18:24:27.714157: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-06-19 18:24:27.794061: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-06-19 18:24:28.162367: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-06-19 18:24:29.136788: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT

Process finished with exit code 0        

Regardless of whatever is happening here, I see a file saved with our binary image and label data.

So now we have normalized the images to the range [0, 255] and converted them to uint8 (unsigned 8-bit integers). This is a good starting point for our project. I am still working out some bugs in the C code I have so far. -Henry