The Millennial Founder‘s Guide to Neural Networks

So this is inspired by a comment I got on yesterday’s post “The Millennial Founder’s guide to AI” asking for more clarification on Neural Networks (NNs). I’m not gonna lie, this is not a simple topic, this article in no way professes to teach you how to make NNs nor does it claim to teach you how to fully understand one.

The goal of this article is to give you the tools necessary to be able to communicate effectively what one is and roughly what it’s doing. In short what it’s doing is making a decision on something. It tries to emulate how the human brain works to do so.

The Human Brain

When you see handwriting (or computer text) there are a sequence of events that have to happen starting with the light hitting your optic nerve, on to the stages your brain goes through to make sense of it and if it is a question your brain will go on to answer the question.

Right now your brain is asking a series of questions and running pattern recognition algorithms to determine what these symbols mean. You’ve seen enough styles of 2 to know that the first one is 2, you seen enough addition symbols done in many ways to recognize that it’s a plus sign then 3 and finally equals. Without even being asked your brain is likely to compute it and come up with the answer 5 of course.

But how does it know?

Neurons

A neuron stores information. In most cases it’s a floating point number between 0 and 1 (0.1-0.9 in-between). The input layer will have whatever you give it. Meaning it will try to find some way of encoding what you input into it into some form of system relating to 0-1. Each one of the circles in the diagram above represents a neuron and the lines represent some kind of logic that changes the next layer. Imagine it like a giant game of Kerplunk. When you change the input numbers it affect the spokes which changed what state the balls are.

We’re going to use the example of handwriting recognition from above. In order for this process to begin we need to encode the data.

Let’s say for example the area we will read is 28px by 28px. That gives us 784 pixels to work with. So imagine the first layer of neurons being 784 neurons long. With each one holding a light to dark value of 0-1.

The network

When you give the system an input it populates all of the first layer of neurons it affects the spokes that also affect the next layer of neurons.

The middle layers are referred to as “the hidden layers” mostly because we don’t know what those neurons will be until there is an input. But those inner layers of the network are essentially the process it needs to go through to determine the output.

They recognize patterns by asking questions. During the training it will have “learned” that if there is a small curve on pixels 12x12 through 18x23 for example and it has 9x17 through 14x18 then then assign a 0-1 on one of the neurons on the next layer.

This essentially is a set of weights and biases that suggest at each stage indicating which of the 0-9 or A-Z it might be. The more complex the possibilities the more layers it will need to come up with the right answer.

It does this going through all of the layers (biases, weights and conditions) and it uses the process of elimination or the process of verification at each stage to get to the answer.

It assigns these weights and biases from its training period where the NN is given a set of examples and tries to match them, the answer is then provided and the accuracy is calculated. Each time it gets an answer wrong or right the optimization algorithms course correct making sure the next answer is closer and closer to the correct answer, as close to 100% of the time as possible. The new information it gets every time it course corrects it adjusts the weights and biases making it more likely the outcome will be correct.

Kinda like a complex game of Guess Who! “Are they male of female” “Does he have a beard or not?” “Hat or no hat” eventually finding it’s way to the correct person. The layers of the NN are essentially the questions you would ask in Guess Who! To determine who the character is.

The NN is estuary doing the same thing based on its training. It’s learning the questions to ask. It’s learning the rules of the game.

By the time it gets to the final layer in the network it should have enough information to correctly guess what the symbol is to the desired threshold (for example 90% of the time).

Essentially asking, if it has this, if it doesn’t have this, if it doesn’t have this but has this all the way, then bingo! It must be a nine because it has a round thing on the top and something like a straight line in the bottom. It will do this for all the possible variations of a 9. Statistically when it has this, this, and this and doesn’t have this, this or this it will be a 9.

It then repeats the process for all the other characters. In our case above recognizing 2+3 and answering the question 5.

This is essentially what Apple Math Notes is doing now!

(All be it a lot more complex)

Conclusion

Making a neural network is hard, it requires incredibly complex logic and math but it doesn’t mean that you can’t understand what it’s doing. You don’t need to have a working understanding of thermo dynamics and kinematic to own operate a car or to understand what’s going on under the hood.

Fuel + Air + compression = propulsion.

It’s a shame that large AI companies have made this seem as complex as possible so that the average person can’t get their head around it but the reality is that it’s just a mathematical version of the game Guess Who!

With each layer it gets closer and closer to the answer. This is why I’m not worried about AI. Because I know that no matter how good it gets at answering questions its processes still revolve around logic and deduction.

It’s not really capable of making the kinds of analogies I’ve made in this article. The creativity element isn’t there. I think before we can get to computer creativity and AGI/ASI (next article) we’ll have to explore different options.

Come back for tomorrow’s article for more on this.