Linear Regression

A fundamental statistical method for modeling the relationship between a dependent variable and one or more independent variables by fitting a linear equation to observed data.

Supervised Learning Parametric Model Regression Analysis Ordinary Least Squares

Linear regression is a foundational technique in statistics and machine learning used to predict a continuous target variable based on one or more predictor variables. It assumes a linear relationship between the inputs and the output, making it both interpretable and computationally efficient. Despite its simplicity, it remains one of the most widely used algorithms across economics, finance, biology, engineering, and social sciences.

The method was first formalized in the early 19th century by mathematicians Adrien-Marie Legendre and Carl Friedrich Gauss, who independently developed the method of least squares to minimize the sum of squared residuals in astronomical data analysis.

Quick Facts

Also Known AsOLS Regression, Linear Least Squares
Model TypeSupervised, Parametric, Linear
Output TypeContinuous Numerical Value
Computational CostLow to Moderate

Mathematical Formulation

At its core, linear regression models the expected value of the target variable \(Y\) as a linear combination of the input features \(X_1, X_2, \dots, X_p\) and an error term \(\epsilon\):

Y = β0 + β1X1 + β2X2 + ⋯ + βpXp + ε

In matrix notation, this simplifies to:

Y = Xβ + ε

Where \(Y\) is an \(n \times 1\) response vector, \(X\) is an \(n \times (p+1)\) design matrix (including a column of ones for the intercept), \(β\) is a \((p+1) \times 1\) coefficient vector, and \(ε\) represents the random error term.

Simple vs. Multiple Regression

  • Simple Linear Regression: Uses a single predictor variable (\(p = 1\)). The model fits a straight line through the data points.
  • Multiple Linear Regression: Uses two or more predictors (\(p \geq 2\)). The model fits a hyperplane in \(p+1\)-dimensional space.

Key Assumptions

For ordinary least squares (OLS) estimators to be unbiased and efficient, several classical assumptions must hold:

  1. Linearity: The relationship between predictors and the response is linear in parameters.
  2. Independence: Observations are independent of each other (no autocorrelation).
  3. Homoscedasticity: The variance of the error terms is constant across all levels of the independent variables.
  4. Normality of Errors: Residuals are normally distributed, especially important for hypothesis testing with small samples.
  5. No Multicollinearity: Predictors are not highly correlated with each other, ensuring stable coefficient estimates.

Violations of these assumptions can be diagnosed using residual plots, the Durbin-Watson test, VIF (Variance Inflation Factor), and Q-Q plots. Remedies include data transformation, regularization, or switching to generalized linear models (GLMs).

Estimation Methods

The coefficients \(β\) are typically estimated using Ordinary Least Squares (OLS), which minimizes the sum of squared residuals:

β^ = argminβ ∑i=1ⁿ (yi − xi\(T\)β)2

The closed-form solution is given by the normal equation:

β^ = (X\(T\)X)−1 X\(T\)Y

When \(X^TX\) is singular or near-singular (e.g., high multicollinearity or \(p > n\)), Gradient Descent or regularized variants like Ridge and Lasso regression are preferred.

Implementation Example

Below is a practical implementation using Python's scikit-learn library, demonstrating data preparation, model training, and evaluation:

Python / scikit-learn
import numpy as np from sklearn.linear_model import LinearRegression from sklearn.model_selection import train_test_split from sklearn.metrics import mean_squared_error, r2_score # Sample dataset: 100 observations, 3 features np.random.seed(42) X = np.random.rand(100, 3) y = 2.5 + 1.8*X[:, 0] - 0.7*X[:, 1] + np.random.randn(100)*0.5 # Split data X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) # Train model model = LinearRegression() model.fit(X_train, y_train) # Predict & Evaluate y_pred = model.predict(X_test) print(f"Coefficients: {model.coef_}") print(f"Intercept: {model.intercept_}") print(f"R² Score: {r2_score(y_test, y_pred):.4f}")

The output provides the learned coefficients, which represent the estimated change in \(Y\) for a one-unit increase in each predictor, holding all other variables constant. The \(R^2\) score indicates the proportion of variance in the target variable explained by the model.

Limitations & Extensions

While powerful and interpretable, linear regression has notable limitations:

  • Linearity Constraint: Fails to capture non-linear relationships without feature engineering or polynomial expansion.
  • Sensitivity to Outliers: Squared error loss heavily penalizes extreme values, skewing coefficient estimates.
  • Multicollinearity: Highly correlated predictors inflate variance and make coefficient interpretation unstable.
  • Heteroscedasticity: Non-constant error variance violates OLS assumptions, leading to inefficient estimates.

Common Extensions:

  • Polynomial Regression: Adds higher-order terms to model curvature.
  • Ridge/Lasso/Elastic Net: Introduces \(L_2\)/\(L_1\) regularization to prevent overfitting and handle multicollinearity.
  • Generalized Linear Models (GLM): Extends linearity to non-Gaussian distributions (e.g., Logistic, Poisson regression).
  • Stepwise/Best Subset Selection: Automated feature selection techniques for high-dimensional data.

References & Further Reading

  1. James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An Introduction to Statistical Learning (2nd ed.). Springer.
  2. Hastie, T., Tibshirani, R., & Friedman, J. (2023). The Elements of Statistical Learning (3rd ed.). Springer.
  3. Gujarati, D. N., & Porter, D. C. (2009). Basic Econometrics (5th ed.). McGraw-Hill Education.
  4. Berry, P. M., & Kromrey, J. D. (2015). "A Guide to Linear Regression Assumptions." Journal of Statistics Education, 23(1).
  5. Aevum Encyclopedia Editorial Board. (2025). "Verification Standards for Mathematical & Statistical Entries." Internal Methodology Documentation.