Lecture Notes – Math 130 – Regression

Lecture Notes – Math 130 – Regression (Chapters 7 – 10) Exploring Relationships Between Variables Chapter 7 – Scatterplots, Association, and Correlation We’ll now look at relationships between two quantitative variables. Relationships between two qualitative variables will be covered in Chapter 26 (chi-squared test of association) The best way to visualize such data is with a scatterplot. You can see patterns, trends, relationship, and even outliers. Look for : • Direction o Positive, negative • Form o Linear, curved o • Strength o Strong, weak o • Unusual features o Outliers o Clusters or subgroups Scatterplot Details y-axis is the vertical, x-axis is the horizontal Label axes, indicate scale Usually, the response variable goes on the vertical and the explanatory variable goes on the horizontal. Also called “independent” and “dependent” variables. Correlation Measure of the association between two variables. 1 − ̅ − = −1 • • • • • Note: • • • • Always between –1 and +1 Correlation between x and y is the same as between y and x Unitless (scale doesn’t matter) Sign of r indicates direction of the association Magnitude indicates the strength (above or below .80 strong, below .30 weak, approx.) ONLY describes a linear relationship Outliers are a problem Correlation does not imply causation (Show stork example on p177) Many people say “correlation” when they mean “association” o Statisticians reserve “correlation” for r. Straightening Scatterplots We can transform our data to make data look more linear. (Look at example on Page 179), Popular transformations • Square of y • Square root of y • 1/y • ln(y) Pro – More linear Con – Interpretation is more difficult Chapter 8 – Linear Regression For a scatterplot, we may be able to model the relationship between two variables with a straight line. Example: 8.61, p 225 Data collected on annual mortality rate (deaths per 100,000) for males in 61 large towns in England and Wales. Also recorded water hardness (ppm of calcium). Let’s try to model mortality as a function of calcium. Draw a line through the data. This is a linear model. What is the best line? Different people might draw different lines. We’ll call the estimate from the line , as opposed to the true value y. For a given x, we can then have the true value of y and the predicted value of . The residual at a specific x is residual = = − Some residuals are positive and some are negative. We could think of minimizing the sum of these residuals, but that would cancel some values out. We’ll define the best fitting line as the one that has the smallest sum of squared residuals. Linear model: Algebra: = +" Statistics: = "# + " Often, we replace y and x by the variable names $) %&'(% = "# + " *&'*+, We have to find the y-intercept and slope coefficients (b0 and b1). " = "# = −" ̅ Usually derived using calculus to minimize ∑ ( − )0 = ∑ ( − "# − " )0 See posting on CMS for an algebraic derivation. To get regression and residuals plot in Rcmdr. • Go to Statistics Fit Models Linear Regression • Select mortality as the response variable • Select calcium as the explanatory variable • Click OK For our data: > numSummary(HardWater[,c("calcium", "mortality")], statistics=c("mean", + "sd"), quantiles=c(0,.25,.5,.75,1)) mean sd n calcium 47.18033 38.09397 61 mortality 1524.14754 187.66875 61 Call: lm(formula = mortality ~ calcium, data = HardWater) Residuals: Min 1Q -348.61 -114.52 Median -7.09 3Q 111.52 b0 b1 Max 336.45 R2 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 1676.3556 29.2981 57.217 < 2e-16 *** calcium -3.2261 0.4847 -6.656 1.03e-08 *** --Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 7 =8 Residual standard error: 143 on 59 degrees of freedom Multiple R-squared: 0.4288, Adjusted R-squared: 0.4191 F-statistic: 44.3 on 1 and 59 DF, p-value: 1.033e-08 ∑ 0 −2 Model Assessment – More on Residuals We can think of each data value as Data = Model + Residual We can see how well our model fits by looking at what the model missed (Residuals). After doing a regression, we can plot the residuals versus the observed x values, and we hope to find nothing. Then the model has explained everything but random error. We want to see a plot that looks like random scatter. • No outliers • No curved shape • No “wedge” shape o Indicates that variance is not constant Residual standard deviation = 7 ∑ 0 =8 −2 For our mortality example, we can get a residual plot as follows: • Do the regression as described above • Go to Models Add observation statistics to data • Uncheck everything but residuals • Go to Graphs Scatterplots • Select calcium as the x-variable • Select residuals.RegModel.1 as the y-variable • Uncheck the optional checkboxes that are checked • Click OK The residual plot is given below. I think it looks like mostly random scatter. I’d conclude that my linear regression is working reasonably well. R2 – the Variation Accounted For Another model assessment measure is : 0 = 0 . This is the percentage of the variation in y that is explained by the linear regression on x. An R2 close to 0 is bad, and a value close to 1 is perfect. There is no set cutoff for what determines a “good” R2 value. Different fields tend to use different criteria. Regression Assumptions and Conditions We make several assumptions when doing linear regression: • Linearity assumption o Checked with a scatterplot • Equal variances assumption o Check that the residuals plot doesn’t have a “wedge” shape As always, outliers are a problem. Chapter 9 – Regression Wisdom We’ll only cover the sections on: • Extrapolation • Outliers, Leverage, and Influence • Lurking variables and causation Extrapolation Making a prediction beyond the range of your observed x values is called extrapolation. It can be dangerous, as you don’t know if the linear relationship between your variables still exists. For example, baby growth curves can behave quite linearly over small periods of time. However, making a prediction of a child’s height at 3 years based on these curves may predict them to be 9 feet tall! The regression would ignore the fact that growth is not linear, but levels out as time goes on. Outliers, Leverage, and Influence I’ve mentioned that outliers can be a problem. With regression, that may or may not be the case. We can have several types of outliers: • A value far from your data in the y direction is often called an outlier • A value far from your data in the x direction is often said to have high leverage. o A point with high leverage can potentially change the regression line, but it doesn’t always use it. • Any point that influences the regression line is called influential. Not all outliers/high leverage points are influential. The best way to check is to run a regression with and without any suspected points to see what happens. For example, the observation at x = 40 below on the left, would probably not change the regression line much if it were omitted, as it already lies close to the line of points we have. The observation at x = 40 below on the right would probably change the regression line dramatically if it were omitted, as it is far from the line of points. The regression lines with and without the observation are on the graph. The observation at x = 40 is influential in this case. High leverage – Influential High leverage – not influential 100 60 80 50 60 40 40 30 20 20 10 0 0 10 20 30 40 50 0 0 10 20 30 40 50 Warning: Influential points can hide in plots of residuals. High leverage points often pull the lines towards them, and so have small residuals. They are easier to see in a scatterplot of the original data or by running regressions with and without such points. Lurking Variables and Causation We can’t argue based on a regression that one variable causes the other. • Even for really high R2. • Even for really straight line scatterplots Observational data can only show an association. There may be some unseen lurking variable that is really explaining what is going on. For example, your book shows a regression for life expectancy across multiple nations plotted against the number of televisions. As number of TV’s rises, so does life expectancy!!! However, there is the lurking variable that is probably something like “standard of living” that is explaining both variables. Chapter 10 – Re-Expressing Data, Getting is Straight Here, I’m only going to focus on two sections • Straight to the Point • Goals of Re-Expression Straight to the Point After doing a regression, we might check the residuals plot and find that there is some curvature in it. When that happens, our regression didn’t work very well. However, we can sometimes transform our data so that we can apply our regression. Popular transformations: • • ; • 0 • ln ( ) The main downside to doing a “re-expression” of the data is that interpretation may become less straightforward. Goals of Re-Expression There are several goals when we transform our data: 1. Make the distribution more symmetric 2. Make the spreads of several distributions more alike 3. Make the form of a scatterplot more nearly linear 4. Make the scatter in a scatterplot spread out evenly rather than having a “wedge shape”