Linear Regression from First Principles

The Simplest Model That Works

Linear regression predicts a number as a weighted sum of features:

price = w1 * area + w2 * bedrooms + b

Training means finding the weights w and bias b that make predictions closest to the true answers. “Closest” is defined by a loss function — for regression, mean squared error (MSE):

MSE = mean((y_true - y_pred) ** 2)

Squaring punishes big misses heavily and makes the maths differentiable.

Gradient Descent in Five Lines

How do you find the best weights? Start anywhere, compute the slope of the loss with respect to each weight, and step downhill:

w = 0.0
lr = 0.01                       # learning rate
for _ in range(1000):
    grad = -2 * np.mean(x * (y - w * x))   # dMSE/dw
    w -= lr * grad

That loop — compute gradient, step against it — is the engine inside almost everything in ML, including the largest neural networks. Linear regression also has an exact closed-form solution, but gradient descent is the idea that scales.

The scikit-learn Version

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

model = LinearRegression()
model.fit(X_train, y_train)            # training happens here
preds = model.predict(X_test)

mean_absolute_error(y_test, preds)     # average miss, in rupees

Three lines of substance: fit, predict, evaluate. Every scikit-learn model follows this exact API, which is why learning it once pays off across dozens of algorithms.

Reading the Model

model.coef_        # one weight per feature
model.intercept_   # the bias term

A coefficient of 4200 on area means: holding other features constant, each extra square foot adds ₹4,200 to the prediction. This interpretability is why linear models remain the default in finance and medicine.

Why the Test Split Is Sacred

Evaluating on training data is like grading students on questions they memorised. The test set simulates data from the future — touch it only once, at the end. If you tune your model based on test scores repeatedly, you’ve silently turned it into a second training set.