Understanding the Foundations of Neural Networks: Building a Perceptron from Scratch in Python

TL;DR

I implemented the historical perceptron and ADALINE algorithms that laid the groundwork for today’s neural networks. This hands-on guide walks through coding these foundational algorithms in Python to classify real-world data, revealing the inner mechanics that high-level libraries often hide. Learn how neural networks actually work at their core by building one yourself and applying it to practical problems.

The Origin Story of Neural Networks

Have you ever wondered what’s happening inside the “black box” of modern neural networks? Before the era of deep learning frameworks like TensorFlow and PyTorch, researchers had to implement neural networks from scratch. Surprisingly, the fundamental building blocks of today’s sophisticated AI systems were conceptualized over 60 years ago.

In this article, we’ll strip away the layers of abstraction and journey back to the roots of neural networks. We’ll implement two pioneering algorithms — the perceptron and the Adaptive Linear Neuron (ADALINE) — in pure Python. By applying these algorithms to real-world data, you’ll gain insights that are often obscured when using high-level libraries.

Whether you’re a machine learning practitioner seeking deeper understanding or a student exploring the foundations of AI, this hands-on approach will illuminate the elegant simplicity behind neural networks.

The Classification Problem: Why It Matters

What is Classification?

Classification is one of the fundamental tasks in machine learning — assigning items to predefined categories. It’s used in countless applications:

• Determining whether an email is spam or legitimate
• Diagnosing diseases based on medical data
• Recognizing handwritten digits or faces in images
• Predicting customer behavior

At its core, classification algorithms learn decision boundaries that separate different classes in a feature space. The simplest case involves binary classification (yes/no decisions), but multi-class problems are common in practice.

Our Dataset: https://www.kaggle.com/datasets/uciml/breast-cancer-wisconsin-data/

For our exploration, we’ll use the Breast Cancer Wisconsin Dataset, a widely used dataset for binary classification tasks. This dataset contains features computed from digitized images of fine needle aspirates (FNA) of breast masses, describing characteristics of cell nuclei present in the images.

Each sample in the dataset is labeled as either malignant (M) or benign (B), making this a perfect binary classification problem. The dataset includes 30 features, such as:

• Radius (mean of distances from center to points on the perimeter)
• Texture (standard deviation of gray-scale values)
• Perimeter
• Area
• Smoothness
• Compactness
• Concavity
• Concave points
• Symmetry
• Fractal dimension

By working with this dataset, we’re tackling a meaningful real-world problem while exploring the foundations of neural networks.

The Pioneers: Perceptron and ADALINE

The Perceptron: The First Trainable Neural Network

In 1957, Frank Rosenblatt introduced the perceptron — a groundbreaking algorithm that could learn from data. The perceptron is essentially a single artificial neuron that takes multiple inputs, applies weights, and produces a binary output.

Here’s how it works:

1. Each input feature is multiplied by a corresponding weight
2. These weighted inputs are summed together with a bias term
3. The sum passes through a step function that outputs 1 if the sum is positive, -1 otherwise

Mathematically, for inputs x?, x?, …, x? with weights w?, w?, …, w?

and bias b:

output = 1 if (w?x? + w?x? + … + w?x? + b) > 0, otherwise -1

The learning process involves adjusting these weights based on classification errors. When the perceptron misclassifies a sample, it updates the weights proportionally to correct the error.

ADALINE: Refining the Approach

Just a few years after the perceptron, Bernard Widrow and Ted Hoff developed ADALINE (Adaptive Linear Neuron) in 1960. While structurally similar to the perceptron, ADALINE introduced crucial refinements:

1. It uses a linear activation function during training rather than a step function.
2. It employs gradient descent to minimize a continuous cost function (the sum of squared errors).
3. It makes predictions using a threshold function, similar to the perceptron.

These changes make ADALINE more mathematically sound and often yield better convergence properties than the perceptron.

Hands-on Implementation: From Theory to Code

Let’s implement both algorithms from scratch in Python and apply them to the Breast Cancer Wisconsin dataset.

?? Project Structure

ml-from-scratch/
│── 2025-03-03-perceptron/   # Today's hands-on session
│   ├── data/                # Dataset & model artifacts
│   │   ├── breast_cancer.csv           # Original dataset
│   │   ├── X_train_std.csv             # Preprocessed training data
│   │   ├── X_test_std.csv              # Preprocessed test data
│   │   ├── y_train.csv                 # Training labels
│   │   ├── y_test.csv                  # Test labels
│   │   ├── perceptron_model_2feat.npz  # Trained Perceptron model
│   │   ├── adaline_model_2feat.npz     # Trained ADALINE model
│   │   ├── perceptron_experiment_results.csv  # Perceptron tuning results
│   │   ├── adaline_experiment_results.csv     # ADALINE tuning results
│   ├── notebooks/           # Jupyter Notebooks for exploration
│   │   ├── Perceptron_Visualization.ipynb
│   ├── src/                 # Python scripts
│   │   ├── data_preprocessing.py       # Data preprocessing script
│   │   ├── perceptron.py                # Perceptron implementation
│   │   ├── train_perceptron.py          # Perceptron training script
│   │   ├── plot_decision_boundary.py    # Perceptron visualization
│   │   ├── adaline.py                   # ADALINE implementation
│   │   ├── train_adaline.py             # ADALINE training script
│   │   ├── plot_adaline_decision_boundary.py  # ADALINE visualization
│   │   ├── plot_adaline_loss.py         # ADALINE learning curve visualization
│   ├── README.md            # Project documentation        

GitHub Repository:https://github.com/shanojpillai/ml-from-scratch/tree/9a898f6d1fed4e0c99a1a18824984a41ebff0cae/2025-03-03-perceptron

?? How to Run the Project

# Run data preprocessing
python src/data_preprocessing.py

# Train Perceptron
python src/train_perceptron.py
# Train ADALINE
python src/train_adaline.py
# Visualize Perceptron decision boundary
python src/plot_decision_boundary.py
# Visualize ADALINE decision boundary
python src/plot_adaline_decision_boundary.py        

?? Experiment Results

By following these steps, you’ll gain a deeper understanding of neural network foundations while applying them to real-world classification tasks.

Project Repository

For complete source code and implementation details, visit my GitHub repository: GitHub Repository: ml-from-scratch — Perceptron & ADALINE

Understanding these foundational algorithms provides valuable insights into modern machine learning. Implementing them from scratch is an excellent exercise for mastering core concepts before diving into deep learning frameworks like TensorFlow and PyTorch.

This project serves as a stepping stone toward building more complex neural networks. Next, we will explore Multilayer Perceptrons (MLPs) and how they overcome the limitations of the Perceptron and ADALINE by introducing hidden layers and non-linearity!