🎲Data, Inference, and Decisions Unit 7 – Linear Regression & Correlation

What's This All About?

• Linear regression models the relationship between a dependent variable and one or more independent variables by fitting a linear equation to observed data
• Correlation measures the strength and direction of the linear relationship between two variables
• Aims to find the line of best fit that minimizes the sum of squared residuals (differences between observed and predicted values)
• Assumes a linear relationship exists between the variables, residuals are normally distributed, and observations are independent
• Can be used for prediction, forecasting, and understanding the impact of variables on an outcome
• Helps answer questions like "How does changing X affect Y?" or "What is the expected value of Y given a certain value of X?"
• Provides a simple, interpretable model for continuous outcomes

Key Concepts to Know

• Dependent variable (Y): The variable being predicted or explained by the independent variable(s)
• Independent variable (X): The variable(s) used to predict or explain the dependent variable
• Slope ($\beta_1$): Represents the change in Y for a one-unit increase in X, holding other variables constant
  • Interpreted as the effect of X on Y
• Intercept ($\beta_0$): The predicted value of Y when all independent variables are zero
• Residuals: The differences between the observed and predicted values of Y
• Coefficient of determination (R-squared): Measures the proportion of variance in Y explained by the model
  • Ranges from 0 to 1, with higher values indicating a better fit
• Pearson correlation coefficient (r): Measures the strength and direction of the linear relationship between two variables
  • Ranges from -1 to 1, with 0 indicating no correlation, and -1 or 1 indicating a perfect negative or positive correlation, respectively

The Math Behind It

• The linear regression equation is $Y = \beta_0 + \beta_1X + \epsilon$, where $\epsilon$ represents the error term
• Ordinary least squares (OLS) is the most common method for estimating the coefficients ($\beta_0$ and $\beta_1$)
  • OLS minimizes the sum of squared residuals: $\sum_{i=1}^{n} (y_i - \hat{y}_i)^2$
• The slope ($\beta_1$) is calculated as $\frac{\sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^{n} (x_i - \bar{x})^2}$
• The intercept ($\beta_0$) is calculated as $\bar{y} - \beta_1\bar{x}$
• The coefficient of determination (R-squared) is calculated as $1 - \frac{\sum_{i=1}^{n} (y_i - \hat{y}_i)^2}{\sum_{i=1}^{n} (y_i - \bar{y})^2}$
• The Pearson correlation coefficient (r) is calculated as $\frac{\sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum_{i=1}^{n} (x_i - \bar{x})^2}\sqrt{\sum_{i=1}^{n} (y_i - \bar{y})^2}}$
• Hypothesis tests and confidence intervals can be constructed for the coefficients and correlation using t-distributions and Fisher's z-transformation, respectively

Real-World Applications

• Economics: Modeling the relationship between price and demand, or income and consumption
• Finance: Predicting stock prices based on market indicators or company performance
• Healthcare: Examining the effect of a drug dosage on patient outcomes, or the impact of risk factors on disease prevalence
• Social sciences: Investigating the relationship between education level and income, or age and political preferences
• Environmental studies: Analyzing the influence of temperature on crop yields, or the effect of pollution on air quality
• Marketing: Predicting sales based on advertising expenditure or customer demographics
• Sports: Modeling the relationship between player statistics and team performance, or the impact of training on athlete performance

Common Pitfalls and How to Avoid Them

• Outliers: Unusual observations that can heavily influence the regression line
  • Identify and investigate outliers using residual plots and Cook's distance
  • Consider removing or transforming outliers if they are due to measurement error or are not representative of the population
• Multicollinearity: High correlation among independent variables, which can lead to unstable coefficient estimates
  • Check for multicollinearity using correlation matrices and variance inflation factors (VIF)
  • Address multicollinearity by removing redundant variables, combining related variables, or using regularization techniques (ridge or lasso regression)
• Non-linearity: When the relationship between variables is not linear, leading to poor model fit
  • Examine scatterplots and residual plots to detect non-linear patterns
  • Consider transforming variables (log, square root, etc.) or using polynomial regression to capture non-linear relationships
• Heteroscedasticity: When the variance of the residuals is not constant across the range of the independent variable(s)
  • Detect heteroscedasticity using residual plots or statistical tests (Breusch-Pagan, White's test)
  • Use robust standard errors or weighted least squares to account for heteroscedasticity
• Autocorrelation: When residuals are correlated with each other, violating the independence assumption
  • Check for autocorrelation using residual plots or statistical tests (Durbin-Watson)
  • Address autocorrelation by including lagged variables, using time series models (ARIMA), or employing generalized least squares (GLS)

Tools and Software

• Microsoft Excel: Offers basic linear regression functionality through the "Data Analysis" add-in
• R: A popular open-source programming language for statistical computing and graphics
  • Packages like

    lm()

    ,

    ggplot2

    , and

    car

    provide extensive regression capabilities
• Python: A versatile programming language with numerous libraries for data analysis and machine learning
  • Libraries like

    statsmodels

    ,

    scikit-learn

    , and

    seaborn

    offer regression modeling and visualization tools
• SPSS: A widely-used commercial statistical software package with a user-friendly interface for regression analysis
• SAS: Another commercial software suite that provides comprehensive regression tools and advanced statistical techniques
• Tableau: A data visualization platform that allows users to explore relationships between variables and create interactive regression models
• MATLAB: A numerical computing environment with built-in functions for linear regression and data visualization

Practice Problems

1. Given a dataset with advertising expenditure and sales, build a linear regression model to predict sales based on advertising spend. Interpret the coefficients and assess the model's goodness-of-fit.
2. Investigate the relationship between a car's fuel efficiency (mpg) and its weight (lbs) using linear regression. Determine the strength and direction of the correlation, and predict the fuel efficiency for a car with a given weight.
3. Analyze the effect of study hours on exam scores for a group of students. Construct a 95% confidence interval for the slope coefficient and test the hypothesis that studying has no impact on exam performance.
4. Explore the relationship between a city's population and its air pollution levels. Identify and address any violations of the linear regression assumptions, and discuss the implications of your findings.
5. Compare the performance of simple linear regression and multiple linear regression in predicting house prices based on features like square footage, number of bedrooms, and location. Evaluate the models using appropriate metrics and cross-validation techniques.

Going Beyond the Basics

• Regularization techniques (ridge, lasso, and elastic net) for handling high-dimensional data and multicollinearity
• Generalized linear models (GLMs) for modeling non-normal response variables (logistic regression for binary outcomes, Poisson regression for count data)
• Non-parametric regression methods (splines, local regression, and kernel regression) for capturing complex, non-linear relationships
• Bayesian linear regression for incorporating prior knowledge and quantifying uncertainty in parameter estimates
• Quantile regression for modeling the relationship between variables at different quantiles of the response distribution
• Regression discontinuity designs for estimating causal effects in observational studies with a threshold-based treatment assignment
• Multilevel (hierarchical) models for analyzing data with a nested structure (students within schools, employees within companies)
• Structural equation modeling (SEM) for testing and estimating causal relationships among latent and observed variables