Regression Analysis

Regression Analysis March 19, 2014 Outline • Regression with two variables, x and y • Confidence limits for regression results Regression Analysis – Student’s t distribution • MATLAB and Excel functions for regression • Data transformations • Multilinear regression – Use of LINEST for multilinear regression • Fifth programming assignment 2 Regression vs. Interpolation Regression (fitting data) • Interpolation seeks to pass an approximation function through every data point • Regression is a statistical approach that seeks an approximate relationship among variables • Looks for measure of error in result • Used to determine trends in experimental data that have some uncertainty • Not necessary (as in interpolation) for relationship to pass through data points • Typically have many more data points than constants in the regression – Useful when we trust the data absolutely – Usually used with accurate tabular data to determine intermediate points not in the table – Used in numerical analysis applications such as numerical integration • Less important with computer calculations of complex engineering functions 3 4 Finding a and b Linear Regression y i y 𝑦i xi Fitted Line x indicates data points • Seeks approximate linear relationship among data set (xi, yi) • Fit equation:𝑦𝑖 = 𝑎 + 𝑏𝑥𝑖 • Notation 𝑦𝑖 indicates approximate value, which may be different from data yi • Equations for a and b based on minimizing sum of squares of differences between actual and approximate data N ^ N S   ( yi  yˆ i )  ( yi  a  bxi )  min i 1 2 2 i 1 5 ME 309 – Numerical Analysis of Engineering Systems • To get the minimum take partial derivatives with respect to a and b and set them to zero N N N S S N  2( yi  a  bxi )(1)  2 yi  2aN  2b xi  0  ( yi  a  bxi ) 2   a a i 1 i 1 i 1 i 1 N N N N S S N  2( yi  a  bxi )( xi )  2 xi yi  2a  xi  2b xi2  0  ( yi  a  bxi ) 2   b b i 1 i 1 i 1 i 1 i 1 x • Solve the first equation for a and substitute it into the second equation 1 1 x and y are  y  b x x y  y N N i 1 N N i N i 1 i a i 1 N i i 1 N i  x  by mean values 6 1 Regression Analysis March 19, 2014 Equations for a and b Measures of Error • Substitute equation for a into equation for b (both copied below) and solve for b 2 x y  2a  x  2b x  0  y  b x N N a i 1 N i i 1 i 1 i b i 1 i 2 i i 1 s y| x     N  xi yi    xi   yi   xi yi  N x  y i 1  i 1  i 1   i 1 2 N N 2  N  xi2  N x   N  xi2    xi  i 1 i 1  i 1  N N i N N N i • Standard Error, sy|x, and R2 N N N (y i 1 i  yˆ i ) 2 R2  1 N 2 ( N  2) s y2| x N y i 1 2 i  N y 2 Numerator in R2 called residual sum of squares; denominator called total sum of squares • R2 varies between 0 and 1 and is a measure of how well the regression equation explains the data variation – R2 = 1 means the regression is perfect and R2 = 0 means the regression is useless • First solve for b then solve for a – Can set a = 0 to force line through origin • The R2 desired in a particular problem depends on experience with typical data for that problem • Can use equations with all sums or means 7 8 Example Calculation xi yi xiyi ŷi  yi  yˆ i  1 10 1 100 10 7.4 6.76 2 19 4 361 38 19.5 0.25 3 27 9 729 81 31.6 21.16 4 44 16 1936 176 43.7 0.09 5 58 25 3364 290 55.8 4.84 15 158 55 6490 595 sums 33.10 xi2 N  N  N  N  xi yi    xi   yi   i 1  i 1  b  i 1 2 N  N  N  xi2    xi  i 1  i 1  5595  15158   12.100 2 555  15 Confidence Limits N 2 yi2 a • What is the statistical uncertainty in calculated values of a or b? • Based on used of Student’s t distribution N  y  b x i i 1 i 1 i 5 158  12.115  4.710 5 N (y s y| x  i 1 i – Important statistical distribution for determining uncertainty in many applications – Distribution depends on random variable, t, and degrees of freedom, n – Define ta,n as the point in the distribution where the probability that t  ta,n is a – Find ta,n from tables or computer functions  yˆ i ) 2 N 2 33.10   3.3216 52 R2  1 ( N  2) s y2| x N y i 1  1 2 i  N y 2 (5  2)3.3216 2  0.9779 2 9 6490  5158 5 10 Student's t Distribution Confidence Limits II 0.40 • The confidence limits for the regression parameters a and b are given by the following equations nnu==55 30 nnu==30 0.30 0.25 f(t) – For using n data points and probability (1 – a) that the limits are correct 0.35 0.20 • Typically choose a = .05 for 95% confidence 0.15 b  ta 2,n2 s y| x a  ta 2,n2 s y|x N  (x  x ) i 1 i i 2 1  n x 2 0.10  ( xi  x )2 0.05 N i 1 Standard errors for a and b 11 0.00 -4 -3 -2 -1 0 1 2 3 12 4 t ME 309 – Numerical Analysis of Engineering Systems 2 Regression Analysis March 19, 2014 Student’s t Distribution – One-sided Test Area under curve to right of ta,n = 1.96 is a, the probability that t  1.96, and 1 – a is the probability that –  t  1.96 Student’s t Distribution – Two-sided Test Can use tables or computer functions to find ta,n value for given probability Often we are interested in the probability for an absolute value of |t|. This is called a two-tailed distribution test Because the distribution is symmetric we can find ta/2,n as the value for which the probability that |t| ≥ ta/2,n is a 1 – a, typically 95%, is the probability that |t|  ta/2,n a is called the significance level 13 Student’s t-Distribution Table t-Distribution Functions • Excel: t.inv.2t(a,n) returns the value ta/2,n such that the probability that the absolute value of a random t-distributed variable, |t|, is greater than ta/2,n is a • MATLAB: The function tinv(1 – a/2,n) returns the value ta/2,n such that the probability that a random t-distributed variable, t, is greater than ta,n is 1 - a/2 Excel: TINV(0.05, 3) = 3.1824 >> tinv(.975,3) ans = 3.1824 14 Critical t Distribution Values and Equivalents 1-sided 2-sided MATLAB Excel MathTable n=1 n=5 n = 10 n = 15 n = 30 n = 50 n = 100 n = 1000 80% 60% 0.800 0.40 0.2 1.376 0.920 0.879 0.866 0.854 0.849 0.845 0.842 90% 80% 0.900 0.20 0.1 3.078 1.476 1.372 1.341 1.310 1.299 1.290 1.282 95% 90% 0.950 0.10 0.05 6.314 2.015 1.812 1.753 1.697 1.676 1.660 1.646 97.5% 95% 0.975 0.05 0.025 12.71 2.571 2.228 2.131 2.042 2.009 1.984 1.962 99% 98% 0.990 0.02 0.01 31.82 3.365 2.764 2.602 2.457 2.403 2.364 2.330 99.5% 99% 0.995 0.01 0.005 63.66 4.032 3.169 2.947 2.750 2.678 2.626 2.581 95% probability that |t|  2.131 and 97.5% probability that t  2.131 15 Excel compatibility function is TINV 99.9% 99.8% 0.999 0.002 0.001 318.3 5.893 4.144 3.733 3.385 3.261 3.174 3.098 16 http://www.itl.nist.gov/div898/handbook/eda/section3/eda3672.htm Back to Example Regression Plot • Get confidence limits on a and b b  ta 2,n2 s y| x  (x  x ) i i 1 a  ta 2,n2 s y|x 1  n s y| x N x 2  (x  x) i 1 i  (x  x ) 2 i 2 i i 1 s y| x N  N 1  n 2 i x 2  3.321646 N  (x  x) i 1 3.321646  1.050397 10 2 1 15 5   3.6538 5 10 2 i • Pick a = 0.05, t0.05/2,5-2 = 3.1824 – b = 10.82  (3.1824)(1.05) = 10.823.34 – a = –4.7  (3.1824)(3.65) = –4.711.6 • Note that a = 0 is within the confidence limits 17 ME 309 – Numerical Analysis of Engineering Systems 18 3 Regression Analysis March 19, 2014 Prediction Confidence Limits Excel LINEST Function • What does 𝑦𝑖 = 𝑎 + 𝑏𝑥𝑖 predict? – Average value of y* for several observations of the same x* Average a  bx *  ta 2,n2 s y|x 1 ( x *  x )2  n N  ( xi  x )2 i 1 1 x *  x  One a  bx *  t a 2 , n  2 s y| x 1   N observation n 2  (x  x) i 1 Uncertainty increases as we get further from the mean x • Array function for linear regression • Returns slope, intercept, standard errors for slope and intercept, R2, sy|x, and various regression statistics • Select area of two column and five rows • Enter formula =LINEST( y_array, x_array, zero_choice, TRUE) – Set zero_choice to FALSE to force regression through origin (default for omitted zero_choice is TRUE) 2 i 19 LINEST Worksheet Results 1 2 3 4 5 MATLAB Regression A B Slope Intercept Standard Error in Standard Error in Slope(SEb) Intercept (SEa) R2 sy|x F statistic Degrees of Freedom(df) Regression sum of Residual sum of squares* squares* Rule of thumb: The slope and intercept should be at least twice their standard errors to be significantly different from zero ConfLimits = Coefficient ± ta/2,df(Standard Error) *Definitions on next slide 21 MATLAB 2nd-order polyfit >> x=1:5; >> y= {10 19 27 44 58]; >> p=polyfit(x,y,2) p = 1.3571 3.9571 4.8000 >> xx = 1:.1:5; >> yfit = polyval(p,xx); >> plot (x,y,'o',xx,yfit) 20 x = 1 2 3 4 5 y = 10 19 27 44 58 >> p = polyfit(x,y,1) p = 12.1000 -4.7000 % slope intercept >> yfit=polyval(p,x) yfit = 7.400 19.500 31.600 43.700 55.800 >> yresid = y – yfit; ssresid is >> ssresid= sum(yresid.^2); residual sum of >> sstotal=(length(y)-1)*var(y) squares from >> Rsqd=1-ssresid/sstotal linest Rsqd = 0.9779 >> [r m b] = regression(x,y) r = 0.9889 m = 12.1000 b = -4.700 linest regression sum of squares = 22 sstotal - ssresid Data Transformations Polyfit returns coefficients p(k) such that p(x) = p1xn + p2xn-1+ … + pn+1; here p(x) = 1.3571x2 + 3.9751x + 4.8000 Polyval evaluates a polynomial with coefficients p1 to pn+1 for input x, which is a row matrix here 23 ME 309 – Numerical Analysis of Engineering Systems • Linear Regression can be used for nonlinear problems if they can be transformed into linear ones • From data on k and T, fit A and E in the following equation: k = Ae-E/RT(R known) • Solution: ln(k) = ln(A) – E/RT • For y = a + bx with y = ln(K) and x = 1/T, a regression gives a = ln(k) and b = -E/R • Several different transformations possible 24 4 Regression Analysis March 19, 2014 Multilinear Regression General Regression • Want to examine case where more than one variable that affects an outcome • Use notation so that we can write code for any number of predictive variables • Call predictive variables x1, x2, x3, etc. • Call response variable y – Example is emissions from diesel engine that depends on fuel properties • emissions = b0 + b1(cetane) + b2(aromatics) + b3(density) – In previous emissions example, x1 = cetane, x2 = aromatics, x3 = density, and y = emissions – For three variables the equation is y = b0 + b1 x1 + b2 x2 + b3 x3 – Use measured data on emissions, cetane, aromatics to find b0, b1, b2, b3 • Another example: CSUN_GPA = b0 + b1HS_GPA + b2SAT_MATH + b3SAT_VERBAL 25 General Equation and Data 26 Data Set with K = 3 and N = 8 • In general we can have K predictive K variables, x1 to xk bj x j • General model equation: y  b0  j 1 • How do we represent the data?  – Each data set consists of one value of y and one value for each of the xj variables – For data set m, we can call the value of y, ym, and we can call the value of xj for data set m xjm m 0 1 2 3 4 5 6 7 y0 y 2.55 1.95 1.89 2.24 2.31 1.74 1.87 0.83 x1 3.00 3.47 3.14 3.46 3.59 1.75 3.03 3.18 x2 440 350 440 350 450 200 310 290 x3 500 • Each data 400 set, m, has a 540 value for y 370 and each xj 480 x12 320 x35 470 400 • What are y4, x26, x17? 27 Summary of Data 28 How do we find bj? • We use K different variables (x1 to xK) to predict the value of another variable, y • We have N sets of data – numbered from m = 0 to m = N – 1 – each data set has one value of y, called ym, and one value of each xj, called xjm – x and y data usually from file input – all data used to determine b0 to bK • Derive equations for b0 to bK by minimizing sum of squares of y – yregression • The derived equations for the bj are solved in the following manner • Define x0m = 1 for all m = 0, ... , N – 1 – Note that there is no x0 in model – Setting x0m = 1 used to simplify equations • Values of b0, ... , bK found by solving K + 1 simultaneous linear equations Ai0 b0 + Ai1 b1 + Ai2 b2 +.....+ AiK bK = ci (i = 0,…, K) – Compute Aij and ci from input data – Use Gaussian elimination solve equations 29 ME 309 – Numerical Analysis of Engineering Systems 30 5 Regression Analysis March 19, 2014 Equations to be Used More Equations to be Used N 1 • Compute the Aij coefficients • Compute the ci coefficients • Use Gaussian elimination routine to solve for the bj Aij   xim x jm • Compute estimated y values m 0 N 1 ci   xim ym  Aijb j  ci j 1 • Compute the R2 value m 0 K K yˆ m  b0   b j x jm i  0,, K j 0 R2  1  Will need these equations for programming assignment five N 1 ( y m 0 N 1 m  yˆ m )2   2   ym2   N  y   m 0  31 32 Multilinear LINEST Excel LINEST Function • Excel LINEST can handle multilinear regressions • To fit K different independent variables – General formula is =LINEST(yRange, 1 xRange, zero?, TRUE) 2 2.55 3.00 440 500 – Select K+1 columns 3 1.95 3.47 350 400 and 5 rows for formula 4 1.89 3.14 440 540 5 2.24 3.46 350 370 – In this example 6 2.31 3.59 450 480 =LINEST(A2:A9, 7 1.74 1.75 200 320 B2:D9,, TRUE) 8 1.87 3.03 310 470 9 0.83 3.18 290 400 – Control+Shift+Enter Bold red letters and numbers are – Results on slide after worksheet rows and columns next A y – Select a range of K+1 columns and five rows and enter the following formula =LINEST(y_range, range_for_all_x, zero_choice, TRUE) – Press control+shift+enter – Data for all x variables must be in adjoining columns B x1 C x2 D x3 33 LINEST Results 1 2 3 4 5 Example Reslts A B C Slope K Slope K – 1 Slope K – 2 Standard Error in Standard Error Standard Error Slope K(SEK) in Slope (SEK-1) (SEK-2) R2 sy|x #N/A F statistic Degrees of #N/A Freedom Regression sum Residual sum #N/A of squares of squares Rule of thumb: The slope and intercept should be at least twice their standard errors to be significantly different from zero 34 35 ME 309 – Numerical Analysis of Engineering Systems x3 slope x2 Slope x1 Slope Intercept Value -0.00509 0.00940 -0.5146 2.3966 Std err 0.00429 0.00448 0.4216 1.3503 R2 | sy|x 0.579 0.444 #N/A #N/A F | df 1.835 4 #N/A #N/A SS 1.087 0.789 #N/A #N/A • F | df indicates F statistic and degrees of freedom in two adjacent columns • SS is sum of squares terms • Bad fit because of large standard errors36 6 Regression Analysis March 19, 2014 Fifth Programming Assignment • Use MATLAB tools to generate cubic spline and plot results • Use MATLAB tools to generate Newton polynomial and plot results • Write a VBA program for Newton polynomials • Use LINEST for multilinear regression in Excel • Do calculations with finite difference derivative expressions 37 ME 309 – Numerical Analysis of Engineering Systems 7