Neural Networks from Scratch

A Neuron Is Logistic Regression

One neuron computes activation(w · x + b) — exactly the logistic regression you already know. The power comes from stacking: a layer is many neurons running in parallel; a network is layers feeding into layers. In between sit non-linear activation functions:

def relu(z):              # the modern default
    return np.maximum(0, z)

Without the non-linearity, stacked linear layers collapse into one linear layer. The activations are what let networks learn curves, interactions, and arbitrary shapes — a wide-enough network can approximate any continuous function.

Forward Pass

A two-layer network for binary classification:

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def forward(X, W1, b1, W2, b2):
    Z1 = X @ W1 + b1          # (n, hidden)
    A1 = relu(Z1)
    Z2 = A1 @ W2 + b2         # (n, 1)
    return sigmoid(Z2), (Z1, A1)   # prediction + cache for backprop

Backpropagation: The Chain Rule, Organised

Training needs the gradient of the loss with respect to every weight. Backprop computes it layer by layer, flowing backwards:

def backward(X, y, W2, cache, y_hat):
    Z1, A1 = cache
    n = len(X)
    dZ2 = y_hat - y                      # loss gradient at output
    dW2 = A1.T @ dZ2 / n
    db2 = dZ2.mean(axis=0)
    dZ1 = (dZ2 @ W2.T) * (Z1 > 0)        # chain rule through ReLU
    dW1 = X.T @ dZ1 / n
    db1 = dZ1.mean(axis=0)
    return dW1, db1, dW2, db2

Nothing mystical: each line is the chain rule applied to one layer. PyTorch automates exactly this bookkeeping — once you’ve written it by hand once, autograd stops being magic.

The Training Loop

rng = np.random.default_rng(42)
W1 = rng.normal(0, 0.1, (X.shape[1], 16)); b1 = np.zeros(16)
W2 = rng.normal(0, 0.1, (16, 1));          b2 = np.zeros(1)

for epoch in range(500):
    y_hat, cache = forward(X, W1, b1, W2, b2)
    dW1, db1, dW2, db2 = backward(X, y, W2, cache, y_hat)
    for p, g in [(W1,dW1),(b1,db1),(W2,dW2),(b2,db2)]:
        p -= 0.1 * g          # gradient descent step

Forward, backward, step — the same three beats as linear regression, and the same three beats that train GPT-class models on thousands of GPUs.

What You Now Understand

• Why depth matters: layers compose features into higher-level features.
• Why initialisation is random: symmetric weights would never differentiate.
• Why training is expensive: every step is a full forward+backward over the data.

Next topic: the same network in PyTorch, where autograd, GPUs, and optimisers come free.