Causality and Simple Linear Regression in Statistics

CHAPTER 9 SIMPLE LINEAR REGRESSION

Causality (also referred to 'cause and effect') is the relation between one process (the cause) and another (the effect), where the first is understood to be partly responsible for the second. In general, a process has many causes, which are said to be causal factors for it, and all lie in its past. An effect can in turn be a cause of many other effects, which all lie in its future.

Cause Effect Reason Result Indepent Dependent

Some causality questions Some causality questions 1.Does the advertising (C) will increase sales (E)? 2.Is there a causal relationship between the number of people who pass in front of a store and the sale of the store? 3.Does personal income determine food consumption?

Interaction Variables Effect Effect Cause Cause Exchange Rate Exchange Rate Inflation Rate Inflation Rate Cause Cause Effect Effect

In statistics, simple linear regression is a linear regression model with a single explanatory (independent) variable. That is, it concerns two-dimensional sample points with one independent variable and (conventionally, the x and y coordinates in a Cartesian coordinate system) and finds a linear function (a non-vertical straight line) that, as accurately as possible, predicts the dependent variable values as a function of the independent variables. one dependent variable The adjective simple refers to the fact that the outcome variable is related to a single predictor.

Simple linear regression Simple linear regression One y and one x. Use x to predict y. Use a linear model/equation and fit it by least squares

Regression analysis Regression analysis Relating two data matrices/tables to each other Purpose: prediction and interpretation Y-data X-data

In Simple linear regression is the least squares estimator of a linear regression model with a single (independent) variable. In other words, simple linear regression fits a straight line through the set of n points in such a way that makes the sum of squared residuals of the model (that is, vertical distances between the points of the data set and the fitted line) as small as possible. The adjective simple refers to the fact that the outcome variable is related to a single predictor. The slope of the fitted line is equal to the correlation between y and x corrected by the ratio of standard deviations of these variables. The intercept of the fitted line is such that it passes through the center of mass (x, y) of the data points.

Y = a + bX

Linear regression model y = a + bx Y = 2 +.5X 5 4 3 y .5 2 1 1 -1 0 1 2 3 4 5 x

Terminology Simple linear regression is a statistical method that allows us to summarize and study relationships between two continuous (quantitative) variables: One variable, denoted x, is regarded as the predictor, explanatory, or independent variable. The other variable, denoted y, is regarded as the response, outcome, or dependent variable. Dependent variable (DV) = response variable = left-hand side (LHS) variable Independent variables (IV) = explanatory variables = right-hand side (RHS) variables = regressor

These questions can be asked by regression 1.Is there an association between independent variables and the dependent variable? 2.Does the independent variable(s) explain the variation in dependent variable? 3.Is the causal relation between the independent variable(s) and dependent variable linear or non- linear? 4.In dynamic regression can we make any estimation for the future?

Simple Linear Regression Model The equation that describes how y is related to x and an error term is called the regression model. The simple linear regression model is: y = 0 + 1x + where: 0 and 1 are called parameters of the model, is a random variable called the error term.

Simple Linear Regression Equation Positive Linear Relationship y Regression line Intercept Slope 1 is positive 0 x

Simple Linear Regression Equation Negative Linear Relationship y Regression line Intercept 0 Slope 1 is negative x

Simple Linear Regression Equation No Relationship y Regression line Intercept 0 Slope 1 is 0 x

Interpretation of 0 and 1 0 (intercept parameter): is the value of y when x = 0. * 1 (slope parameter): is the change in y given x changes by 1 unit.

Estimated Simple Linear Regression Equation The estimated simple linear regression equation y = + b b x 0 1 The graph is called the estimated regression line. b0 is the y intercept of the line. b1 is the slope of the line. is the estimated value of y for a given value of x. y

Estimation Process Sample Data: x y x1 . . . . xnyn Regression Model y = 0 + 1x + Regression Equation E(y|x) = 0 + 1x Unknown Parameters 0, 1 y1 b0 and b1 Estimated provide point estimates of 0 and 1 Regression Equation b y 0 = + b x 1

It is common to make the additional hypothesis that the ordinary least squares method should be used to minimize the residuals. Under this hypothesis, the correctness of a line through the sample points is measured by the sum of squared residuals (vertical distances between the points of the data set and the fitted line), and the goal is to make this sum as small as possible.

Model = + + ~ , 0 ( N ) Y x 0 1 1 > 0 Positive Association 1 < 0 Negative Association 1 = 0 No Association

Minimize the sum of squared errors 0 Mean response when x=0 (y- intercept) ^ ^ ^ = + y x 0 1 1 Change in mean response when x increases by 1 unit (slope) 0, 1 are unknown parameters (like ) 0+ 1x Mean response when explanatory variable takes on the value x Task: Minimize the sum of squared errors: 2 2 ^ y ^ ^ ^ y ^ ^ n n = + = = + x SSE y y x 0 1 0 1 i i i i = = 1 1 i i

Estimated Standard Error Parameter: Slope in the population model ( 1) ^ ^ ^ ^ = + Estimator: Least squares estimate: y x 1 0 1 Estimated standard error: ^ = s/ S ^ xx 1 2 ^ y = y n SSE = 2 s 2 ( 2 n ) 2 = Sxx x x

Least Squares Method Least Squares Method 4 x3 e3 3 E(Y)=a + bX 2 y x1 e2 e1 1 x2 0 0 1 2 3 4 5 x

Least Squares Method Least squares method minimizes the sum of squares of errors (deviations of individual data points form the regression line) Such a and b are called least squares estimators (estimators of parameters and ). The process of getting parameter estimators (e.g., a and b) is called estimation Regress Y on X Lest squares method is the estimation method of ordinary least squares (OLS)

Formulation Formulation of Simple of Simple linear Model Model Y = f(X) Y = f(X) f(X) = f(X) = 0 0+ linear Regression Regression + 1 1X Xi i f(X) = f(X) = 0 0+ + 1 1X Xi i+ + i i

Least Squares Method 1 + + = X bX a Y Y + = = ) ( a Y bX a Y Y Y = + = = ) ( ) ( bX a Y Y Y = = Y E bX 2 2 2 ( ) = + + + 2 2 2 2 2 ( ) 2 bXY 2 2 Y a bX Y a b X aY abX ) Y = = 2 2 2 ( ( ) Y Y a bX = 2 2 ( ) Min Min Y a bX How to get a and b that can minimize the sum of squares of errors? INTT 103

Least Squares Method 2 Linear algebraic solution Compute a and b so that partial derivatives with respect to a and b are equal to zero ( ) ( a a 0 = + X b Y na ) ( ) 2 2 Y a bX = = + = 2 2 2 0 na Y b X Y X = = a b Y b X n n

Least Squares Method 3 Take a partial derivative with respect to b and plug in a you got, a=Ybar b*Xbar ( 2 = b b 0 = + X a XY X b n n ( 2 + n n ( n n ) ( ) 2 2 ( ) Y a bX = + = 2 2 2 0 b X XY a X ( ) + = 2 2 0 b X XY Y b X X Y X + = 2 0 b X XY b X ) 2 X Y X = 0 b X XY b ) 2 2 n X X XY X Y = b

Least Squares Method 4 Least squares method is an algebraic solution that minimizes the sum of squares of errors (variance component of error) ( ) ( )( ) SS n XY X Y X X Y Y xy = = = b ( ) 2 2 X X SS 2 n X X x Y X = = a b Y b X n n n X 2 Y X X XY = a ( ) 2 2 X

Least Squares Method Slope for the Estimated Regression Equation SS xy = = b b 1 SS y = ( )( ) Where SS y y x x xy 2 = ( ) and SS x x x

Least Squares Method y-Intercept for the Estimated Regression Equation = b y b x 0 1 a=bo _ x = mean value for independent variable _ y = mean value for dependent variable

OLS: Example (First way) No x y x-xbar -14.5 -9.5 -1.5 y-ybar -8.5 -16.5 -1.5 (x-xb)(y-yb) 123.25 156.75 (x-xbar)^2 210.25 90.25 1 2 3 4 5 6 43 48 56 61 67 70 128 120 135 143 141 152 2.25 22.75 42.75 193.75 2.25 12.25 90.25 156.25 3.5 9.5 12.5 6.5 4.5 15.5 Mean Sum 57.5 345 136.5 819 541.5 561.5 ( )( ) SS X X Y Y 541 5 . xy = = = = 9644 . b 2 ( ) 561 5 . X X SS x = = = 136 5 . 9644 . 57 5 . 0481 . 81 a Y b X

OLS: Example (Second way) No x y xy x^2 1849 2304 3136 3721 4489 4900 1 2 3 4 5 6 43 48 56 61 67 70 128 120 135 143 141 152 5504 5760 7560 8723 9447 10640 Mean Sum 57.5 345 136.5 819 47634 20399 n X n XY X Y + 6 47634 345 819 2 = = = 964 . b ( ) 2 6 20399 345 2 X n X 2 Y X X XY + 819 20399 345 47634 2 = = = 81 048 . a ( ) 2 6 20399 345 2 X

OLS Y hat = 81.048 + .964X 150 140 130 120 40 50 60 70 x Fitted values y

How to interpret b? For unit increase in x, the expected change in y is b, holding other things (variables) constant. For unit increase in x, we expect that y increases by b, holding other things (variables) constant. For unit increase in x, we expect that y increases by .964, holding other variables constant.

Estimated Regression Equation Example: Reed Auto Sales Slope for the Estimated Regression Equation SS 20 xy = = = 5 b 1 4 SS x y-Intercept for the Estimated Regression Equation = = = 20 5 ) 2 ( * 10 b y b x 0 1 Estimated Regression Equation = + 10 5 * y x

Scatter Diagram and Regression Line 26 24 = + y 10 5 x 22 20 18 number of cars sold 16 14 12 .5 1.0 1.5 2.0 2.5 3.0 3.5 number of ads

Example: Estimate of Residuals y = y e y x 1 3 2 1 3 y 14 24 18 17 27 15 25 20 15 25 -1.0 -1.0 -2.0 2.0 2.0

Course Decomposition of total sum of squares Relationship Among SST, SSR, SSE Summer Mining Data SST = SSR + SSE ( y = + 2 2 2 ( iy y ) iy y ) ( y ) i i where: SST = total sum of squares SSR = sum of squares due to regression SSE = sum of squares due to error

Decomposition of total sum of squares y = y e y y = = 2) y 2) ( ( SSE y SSR y y y -5 5 0 -5 5 25 25 0 25 25 -1 -1 -2 2 2 1 15 1 25 4 20 4 15 4 25 SSE=14 SSR=100 Check if SST= SSR + SSE

Coefficient of Determination The coefficient of determination is: r2 = SSR/SST r2 = SSR/SST = 100/114 = 0.8772 The regression relationship is very strong; about 88% of the variability in the number of cars sold can be explained by the number of TV ads. The coefficient of determination (r2) is also the square of the correlation coefficient (r).

Sample Correlation Coefficient r = (sign of ) Coefficien t of Determinat ion 1b r = 2 (sign of 1 ) b r = + = . 0 8772 . 0 936 r

Sampling Distribution of b1

Estimate of 2 The mean square error (MSE) provides the estimate of 2. s2 = MSE = SSE/(n 2) where: = y 2) y SSE ( i i

R2 and Goodness-of-fit Goodness-of-fit measures evaluates how well a regression model fits the data The smaller SSE, the better fit the model Large R square means the model fits the data

Correlation Coefficient, r : The quantity r, called the linear correlation coefficient, measures the strength and the direction of a linear relationship between two variables. The linear correlation coefficient is sometimes referred to as the Pearson product moment correlation coefficient in honor of its developer Karl Pearson. The mathematical formula for computing r is:

The value of r is such that -1 < r < +1. The + and signs are used for positive linear correlations and negative linear correlations, respectively. Positive correlation: If x and y have a strong positive linear correlation, r is close to +1. An rvalue of exactly +1 indicates a perfect positive fit. Positive values indicate a relationship between x and y variables such that as values for x increases, values for y also increase. Negative correlation: If x and y have a strong negative linear correlation, r is close to -1. An rvalue of exactly -1 indicates a perfect negative fit. Negative values indicate a relationship between x and y such that as values for x increase, values for y decrease. No correlation: If there is no linear correlation or a weak linear correlation, r is close to 0. A value near zero means that there is a random, nonlinear relationship between the two variables.

A perfect correlation of 1 occurs only when the data points all lie exactly on a straight line. If r = +1, the slope of this line is positive. If r = -1, the slope of this line is negative. A correlation greater than 0.8 is generally described as strong, whereas a correlation less than 0.5 is generally described as weak. These values can vary based upon the "type" of data being examined. A study utilizing scientific data may require a stronger correlation than a study using social science data.