01 - Quantitative Methods

01 Quantitative Methods Quantitative Methods 2024 Level II High Yield Notes LM01 Basics of Multiple Regression and Underlying Assumptions Uses of multiple linear regression Multiple regression allows us to determine the effect of more than one independent variable on a particular dependent variable. Multiple regression can be used: • To explain the relationships between financial variables: e.g., the relationship between inflation, GDP growth rates and interest rates. • To test existing theories – e.g., are equity returns impacted by a stock’s market cap and value/growth factors. • To make forecasts – e.g., using variables such as financial leverage, profitability, revenue growth, and changes in market share to predict whether a company will face financial distress. The basics of multiple linear regression A multiple linear regression model has the general form: Yi = b0 + b1 X1i + b2 X2i + ⋯ + bk Xki + εi , i = 1, 2,….n The slope coefficient bj measures how much the dependent variable Y changes when the independent variable, Xj, changes by one unit holding all other independent variables constant. The intercept coefficient b0 represents the expected value of Y if all independent variables are zero. For example, consider the following regression equation: Y = 0.2 + 0.6X1 + 0.5 X2 + ϵ If X1 changes by 1 unit and X2 remains constant, then Y will change by 0.6 units. Similarly, if X1 remains constant and X2 changes by 1 unit, then Y will change by 0.5 units. If X1 and X2 are each zero, then the expected value of Y is 0.2. Assumptions underlying multiple linear regression The five main assumptions underlying multiple regression models are: 1. Linearity: The relationship between the dependent variable and the independent variables is linear. 2. Homoskedasticity: The variance of the regression residuals is the same for all observations. 3. Independence of errors: The observations are independent of one another. This implies the regression residuals are uncorrelated across observations. © IFT. All rights reserved 1 Quantitative Methods 2024 Level II High Yield Notes 4. Normality: The regression residuals are normally distributed. 5. Independence of independent variables: 5a. Independent variables are not random. 5b. There is no exact linear relation between two or more of the independent variables or combinations of the independent variables. Commonly used diagnostic plots Scatterplots of dependent and independent variables are used to check if the assumptions of ‘linearity’ and ‘independence of independent variables’ have been violated. Scatterplots of residuals is used to check if the assumptions of ‘homoskedasticity’ and ‘independence of errors’ have been violated. A ‘Q-Q’ plot of residuals is used to check if the assumption of ‘normality’ has been violated. © IFT. All rights reserved 2 Quantitative Methods 2024 Level II High Yield Notes LM02 Evaluating Regression Model Fit and Interpreting Model Results Goodness of fit R2 and adjusted R2 R2 measures the percentage of variation in Y that is explained by the independent variables. In multiple regression, R2 increases as we add new independent variables, even if the ̅2 amount of variation explained by them is not statistically significant. Hence, the adjusted R is used because it does not automatically increase as independent variables are added to the model. AIC and BIC When evaluating a collection of models that explain the same dependent variable, we cannot rely on the adjusted R2 alone. Two commonly used statistics for this purpose are Akaike’s information criterion(AIC) and Schwarz’s Bayesian information criterion (BIC). These stats are often provided as part of the regression software output. Lower values of both measures are better. When do we prefer one measure over the other? • • AIC is preferred if the model is used for prediction purposes BIC is preferred when the best goodness of fit is the goal Testing joint hypothesis for coefficients Hypothesis tests of a single coefficient The hypothesis tests of a single coefficient in a multiple regression are identical to those in a simple regression. If we are testing simply whether a variable is significant in explaining the dependent variable’s variation, the hypotheses are: H0: bj = 0 Ha: bj ≠ 0 The statistical software produces the t-statistics and P-values for a test of the slope coefficient against zero for each independent variable in the model. t-stats greater than the critical value indicate that the variable is significant. Joint F-test The joint F-test is used to jointly test a subset of variables in a multiple regression, where the “restricted” model is based on a narrower set of independent variables nested in the broader “unrestricted” model. The null hypothesis is that the slope coefficients of all independent variables outside the restricted model are zero. It is calculated as: © IFT. All rights reserved 1 Quantitative Methods F= 2024 Level II High Yield Notes (Sum of squares error restricted model − Sum of squares error unrestricted)/q Sum of squares error unrestricted model/(n − k − 1) where: q is the number of restrictions, i.e. the number of variables omitted If the calculated F-stat exceeds the critical F-value we can conclude that at least one of the omitted variables is statically significant. General linear F-test The general linear F-test is an extension of the joint F-test, where we test the significance of the whole regression equation. The null hypothesis is that the slope coefficients on all independent variables are 0, against the alternative that at least one coefficient is different from 0. The F-stat is calculated as: F= mean regression sum of squares MSR = mean squared error MSE If the calculated F-stat exceeds the critical F-value we can conclude that at least one of the slope coefficients is different from 0. Forecasting using multiple regression To forecast the value of a dependent variable using a multiple linear regression model, follow these three steps: ̂0 , b ̂1 , b ̂2 , … . , b ̂k of the regression parameters b0, b1, b2,…bk. The ̂ 1. Obtain estimates b symbol indicates that the values are estimated. ̂ ̂ 2. Determine the assumed values of the independent variables X̂ 1i , X 2i , … … , X ki ̂ using the following equation: 3. Compute the predicted value of the dependent variable, 𝑌𝑖 ̂0 + b ̂1 ̂ ̂k ̂ ̂ Yi = b X1i + b̂2 ̂ X2i + ⋯ … + b Xki The level of uncertainty around the forecast of the dependent variable is called the standard error of forecast. This forecast error depends on how well the independent variables (X1, X2, X3…) were forecasted i.e. the sampling error, as well as the model error. The larger the sampling error, the larger is the standard error of the forecast of Y and the wider is the confidence interval. © IFT. All rights reserved 2 Quantitative Methods 2024 Level II High Yield Notes LM03 Model Misspecification Misspecified functional form Whenever we estimate a regression, we must assume that the regression has the correct functional form. This assumption can fail in several ways as shown in Exhibit 2. Failures in Regression Functional Form Explanation Consequence Omitted variables One or more important variables are omitted from the regression May lead to heteroskedasticity or serial correlation Inappropriate form of variables Ignoring a nonlinear relationship between the dependent and independent variable May lead to heteroskedasticity Inappropriate variable scaling One or more regression variables may need to be transformed before estimating the regression. May lead to heteroskedasticity or multicollinearity Inappropriate data pooling Regression model pools data from different samples that should not be pooled May lead to heteroskedasticity or serial correlation Heteroskedasticity There are two types of heteroskedasticity: • Unconditional heteroskedasticity: When heteroskedasticity of the error variance is not correlated with the independent variables. Unconditional heteroskedasticity is not a problem. • Conditional heteroskedasticity: The error variance is correlated with the values of the independent variables. Conditional heteroskedasticity is a problem. It results in underestimation of standard errors, so t-statistics are inflated and Type I errors are more likely. The figure below illustrates conditional heteroskedasticity. © IFT. All rights reserved 1 Quantitative Methods 2024 Level II High Yield Notes Conditional heteroskedasticity can be detected using the Breusch–Pagan (BP) test. It can be corrected by computing ‘robust standard errors’. Serial correlation In serial correlation, regression errors in one period are correlated with errors from previous periods. It is often found in time-series regressions. The following figure demonstrates the presence of serial correlation. When the previous error term is positive the next error term is also most likely positive and vice versa. Consequences: Independent Variable Is Lagged Value of Dependent Variable No Yes © IFT. All rights reserved Invalid Coefficient Estimates No Yes 2 Invalid Standard Error Estimates Yes Yes Quantitative Methods 2024 Level II High Yield Notes Serial correlation can be detected using Breusch–Godfrey (BG) test. It can be corrected by computing ‘robust standard errors’. Multicollinearity Multicollinearity may occur when two or more independent variables are highly correlated or when there is an approximate linear relationship among independent variables. Consequences: The standard errors are inflated so the t-stats of coefficients are artificially small and cannot reject the null hypothesis. Multicollinearity can be detected by using the variance inflation factor (VIF). VIF values above 5 warrant further investigation. Whereas, VIF values above 10 indicate serious multicollinearity problems. It can be corrected by dropping one or more of the regression variables, using a different proxy for one of the variables, or increasing the sample size. Exhibit 14 summarizes the three issues arising from regression assumption violations. Assumption Violation Issue Detection Correction Homoskedastic error terms Heteroskedastic error terms Biased estimates of coefficients’ standard errors Visual inspection of residuals; Breusch– Pagan test Revise model; use robust standard errors Independence of observations Serial correlation Inconsistent Breusch– estimates of Godfrey test coefficients and biased standard errors Independence of independent variables Multicollinearity Inflated standard errors © IFT. All rights reserved 3 Variance inflation factor Revise model; use serialcorrelation consistent standard errors Revise model; increase sample size Quantitative Methods 2024 Level II High Yield Notes LM04 Extensions of Multiple Regression Influence analysis An influential observation is an observation whose inclusion may significantly alter regression results. Two kinds of observations may potentially influence regression results: • • A high-leverage point: A data point having an extreme value of an independent variable (X). An outlier: A data point having an extreme value of the dependent variable (Y). Detecting Influential Points Leverage (hii) A high-leverage point can be identified using a measure called leverage (hii). Leverage measures the distance between the value of the ith observation of an independent variable and the mean value of that variable across all n observations. Leverage ranges from 0 to 1, the higher the leverage the more distant the observation’s value is from the mean, and hence the more influence it can exert on the estimated regression line. Statistical software packages can easily calculate the leverage measure. Interpretation: If Then The observation is potentially influential 𝑘+1 Leverage > 3 ( 𝑛 ) where: k is the number of independent variables Studentized residual (ti*) An outlier can be identified using a measure called studentized residuals. Statistical software packages can calculate and present this measure. Interpretation: If | ti*| > 3 | ti*| > critical value of the t-statistic with n-k-2 degrees of freedom Then Flag observation as being an outlier Flag outlier observation as being potentially influential Cook’s distance Cook’s distance, or Cook’s D (Di), is a metric for identifying influential data points. It measures how much the estimated values of the regression change if observation i is deleted from the sample. Interpretation: If Di > 0.5 © IFT. All rights reserved Then The ith observation may be influential and merits further investigation. 1 Quantitative Methods Di > 1.0 Di > 2√𝑘/𝑛 2024 Level II High Yield Notes The ith observation is highly likely to be an influential data point. The ith observation is highly likely to be an influential data point. Dummy variables in multiple regression Dummy (or indicator) variables represent qualitative independent variables. They take on a value of 1 if a particular condition is true and 0 if that condition is false. To distinguish among n categories, the model must include n – 1 dummy variables. An intercept dummy adds to or reduces the original intercept if a specific condition is met. When the intercept dummy is 1, the regression line shifts up or down parallel to the base regression line. Yi = b0 + d0Di + b1Xi + εi. A slope dummy allows for the slope of the regression line to change if a specific condition is met. Yi = b0 + b1Xi + d1DiXi + εi. It is also possible for a regression model to use both intercept and slope dummy variables. Yi = b0 + d0Di + b1Xi + d1DiXi + εi. Exhibit 11, Panel A, shows the effect of an intercept dummy variable. Exhibit 11, Panel B, shows the effect of a slope dummy variable. © IFT. All rights reserved 2 Quantitative Methods 2024 Level II High Yield Notes Exhibit 11, Panel C, shows the combined effect of a intercept and slope dummy variable. Logistic regression model Qualitative dependent variables are outcome variables describing data that fit into categories (e.g. bankrupt or not bankrupt). A logistic transformation is applied when the model contains qualitative dependent variables. The logistic transformation is: ln[P/(1 − P)] The logit transformation linearizes the relation between the transformed dependent variable and the independent variables. © IFT. All rights reserved 3 Quantitative Methods 𝑙𝑛 ( 2024 Level II High Yield Notes 𝑃 ) = 𝑏0 + 𝑏1 𝑋1 + 𝑏2 𝑋2 + 𝑏3 𝑋3 + € 1−𝑃 The natural logarithm (ln) of the odds of an event happening is the log odds which is also called the logit function. Logistic regression coefficients are typically estimated using the maximum likelihood estimation (MLE) method rather than by least squares. In a logit model, slope coefficients are interpreted as the change in the log odds that the event happens per unit change in the independent variable, holding all other independent variables constant. A likelihood ratio (LR) test is a method to assess the fit of logistic regression models. The test is similar to the joint F-test. It compares the fit of the restricted and unrestricted models. © IFT. All rights reserved 4 Quantitative Methods 2024 Level II High Yield Notes LM05 Time-Series Analysis Time series A time series is a set of observations on a variable measured over different time periods. A time series model allows us to make predictions about the future values of a variable. Linear vs log-linear trend models • • • • When the dependent variable changes at a constant amount with time, a linear trend model is used. The linear trend equation is given by yt = b0 + b1 t + εt , t = 1, 2, … , T When the dependent variable changes at a constant rate (grows exponentially), a loglinear trend model is used. The log-linear trend equation is given by ln yt = b0 + b1t, t = 1, 2, …, T A limitation of trend models is that by nature they tend to exhibit serial correlation in errors, due to which they are not useful. The Durban-Watson statistic is used to test for serial correlation. If this statistic differs significantly from 2, then we can conclude the presence of serial correlation in errors. To overcome this problem, we use autoregressive time series (AR) models. Requirements for a time series to be covariance stationary AR models can only be used for time series that are covariance stationary. A time-series is covariance stationary if it meets the following three conditions: • The expected value of the time series (its mean) must be constant and finite in all periods. • The variance must be constant and finite in all periods. • The covariance of time series, past or future must also be constant and finite in all periods. Autoregressive (AR) models An autoregressive time series model is a linear model that predicts its current value using its most recent past value as the independent variable. An AR model of order p, denoted by AR(p) uses p lags of a time series to predict its current value. xt = b0 + b1 x(t–1) + b2 x(t–2) + … + bp x(t– p) + εt The chain rule of forecasting is used to predict successive forecasts. The one-period ahead forecast of xt from an AR(1) model is x̂t+1 = b̂0 + b̂1 xt xt+1 can be used to forecast the two-period ahead value : x̂t+2 = b̂0 + b̂1 xt+1 Autocorrelations of the residuals • If an AR model has been correctly specified, the residual terms will not exhibit serial © IFT. All rights reserved 1 Quantitative Methods • • • 2024 Level II High Yield Notes correlation. We cannot use the Durban–Watson statistic to test for serial correlation in AR models. Instead, we check the autocorrelations of the residuals. The autocorrelations of residuals are the correlations of the residuals with their own past values. The autocorrelation between one residual and another one at lag k is known as the kth order autocorrelation. If the model is correctly specified, the autocorrelation at all lags must be equal to 0. A t-test is used to test whether the error terms in a time series are serially correlated. residual autocorrelation Test stat = standard error Mean reversion A time series is said to be mean-reverting if it tends to fall when its level is above its mean and rise when its level is below its mean. If a time series is covariance stationary, then it will be mean-reverting. The mean-reverting level is calculated as: b0 Mean– reverting level xt = 1 − b1 In- sample and out- of- sample forecasts, root mean squared error criterion (RMSE) There are two types of forecasting errors based on the period used to predict values: • In-sample forecast errors: these are residuals from a fitted series model used to predict values within the sample period. • Out-of-sample forecast errors: these are regression errors from the estimated model used to predict values outside the sample period. Out-of-sample analysis is a realistic way of testing the forecasting accuracy of a model and aids in the selection of a model. Root mean squared error (RMSE), square root of the average squared forecast error, is used to compare the out-of-sample forecasting performance of the models. If two models are being compared, the one with the lower RMSE for out-of-sample forecasts has better forecast accuracy. Instability of coefficients Estimates of regression coefficients of the time-series model can change substantially across different sample periods used for estimating the model. When selecting a time period: • Determine whether economics or environment have changed. • Look at graphs of the data to see if the time series looks stationary. Most economic and financial time series data are not stationary. © IFT. All rights reserved 2 Quantitative Methods 2024 Level II High Yield Notes Random walk A random walk is a time series in which the value of the series in one period is the value of the series in the previous period plus an unpredictable random error. The equation for a random walk without a drift is: xt = xt−1 + εt The equation for a random walk with a drift is: xt = b0 + xt−1 + εt They do not have a mean-reverting level and are therefore not covariance stationary. For example, currency exchange rates. Unit root • • • • For an AR (1) model to be covariance stationary, the absolute value of the lag coefficient b1 must be less than 1. When the absolute value of b1 is 1, the time series is said to have a unit root. All random walks have unit roots. If the time series has a unit root, then it will not be covariance stationary. A random-walk time series can be transformed into one that is covariance stationary by first differencing the time series. We define a new variable y as follows: yt = xt − xt−1 = εt , where E(εt ) = 0, E(ε2t ) = σ2 , E(εt εs ) = 0 if t ≠ s We can then use and AR model on the first-differenced series. Unit root test We can detect the unit root problem by using the Dickey-Fuller test. It is a unit root test based on a transformed version of the AR (1) model xt = b0 + b1 xt−1 + εt Subtracting xt-1 from both the sides, we get xt − xt−1 = b0 + (b1 − 1) xt−1 + εt or xt − xt−1 = b0 + g1 xt−1 + εt where 𝑔1 = 𝑏 − 1 If b1 = 1, then g1 = 0. This means there is a unit root in the model. Seasonality If the error term of a time-series model shows significant serial correlation at seasonal lags, the time-series has significant seasonality. This means there is significant data in the error terms that is not being captured in the model. Seasonality can be corrected by including a seasonal lag in the model. For instance, to correct seasonality in the quarterly time series, modify the AR (1) model to include a seasonal lag 4: © IFT. All rights reserved 3 Quantitative Methods 2024 Level II High Yield Notes xt = b0 + b1 x(t–1) + b2 x(t−4) + εt If the revised model shows no statistical significance of the lagged error terms, then the model has been corrected for seasonality. Autoregressive conditional heteroskedasticity (ARCH) • • • • If the variance of the error in a time series depends on the variance of the previous errors than this condition is called autoregressive conditional heteroskedasticity (ARCH). If ARCH exists, the standard errors for the regression parameters will not be correct. We will have to use generalized least squares or other methods that correct for heteroskedasticity. To test for first-order ARCH, we regress the squared residual on the squared residual from the previous period. ε̂2t = a0 + a1 ε̂2t−1 + ut If the coefficient in our model is statistically significant, the time- series model has ARCH(1) errors. If a time-series model has significant ARCH, then we can predict the next period error variance using the formula: ̂2t+1 = â0 + â1 ε̂2t σ Working with two time series If a linear regression is used to model the relationship between two time series, a test such as the Dickey-Fuller test should be performed to determine whether either time series has a unit root. • If neither of the time series has a unit root, then we can safely use linear regression. • If one of the two time series has a unit root, then we should not use linear regression. • If both time series have a unit root and they are cointegrated (exposed to the same macroeconomic variables), we may safely use linear regression. • If both time series have a unit root but are not cointegrated, then we cannot not use linear regression. The Engle-Granger/Dicky-Fuller test is used to determine if a time series is cointegrated. Selecting an appropriate time-series model Section 16.1 from the curriculum provides a step-by-step guide on selecting an appropriate time-series model. 1. Understand the investment problem you have, and make an initial choice of model. One alternative is a regression model that predicts the future behavior of a variable based on hypothesized causal relationships with other variables. Another is a time-series © IFT. All rights reserved 4 Quantitative Methods 2024 Level II High Yield Notes model that attempts to predict the future behavior of a variable based on the past behavior of the same variable. 2. If you have decided to use a time-series model, compile the time series and plot it to see whether it looks covariance stationary. The plot might show important deviations from covariance stationarity, including the following: • a linear trend; • an exponential trend; • seasonality; or • a significant shift in the time series during the sample period (for example, a change in mean or variance). 3. If you find no significant seasonality or shift in the time series, then perhaps either a linear trend or an exponential trend will be sufficient to model the time series. In that case, take the following steps: • Determine whether a linear or exponential trend seems most reasonable (usually by plotting the series). • Estimate the trend. • Compute the residuals. • Use the Durbin–Watson statistic to determine whether the residuals have significant serial correlation. If you find no significant serial correlation in the residuals, then the trend model is sufficient to capture the dynamics of the time series and you can use that model for forecasting. 4. If you find significant serial correlation in the residuals from the trend model, use a more complex model, such as an autoregressive model. First, however, reexamine whether the time series is covariance stationary. The following is a list of violations of stationarity, along with potential methods to adjust the time series to make it covariance stationary: • If the time series has a linear trend, first-difference the time series. • If the time series has an exponential trend, take the natural log of the time series and then first-difference it. • If the time series shifts significantly during the sample period, estimate different time-series models before and after the shift. • If the time series has significant seasonality, include seasonal lags (discussed in Step 7). 5. After you have successfully transformed a raw time series into a covariance-stationary time series, you can usually model the transformed series with a short autoregression. To decide which autoregressive model to use, take the following steps: • Estimate an AR(1) model. • Test to see whether the residuals from this model have significant serial correlation. © IFT. All rights reserved 5 Quantitative Methods • 2024 Level II High Yield Notes If you find no significant serial correlation in the residuals, you can use the AR(1) model to forecast. 6. If you find significant serial correlation in the residuals, use an AR(2) model and test for significant serial correlation of the residuals of the AR(2) model. • If you find no significant serial correlation, use the AR(2) model. • If you find significant serial correlation of the residuals, keep increasing the order of the AR model until the residual serial correlation is no longer significant. 7. Your next move is to check for seasonality. You can use one of two approaches: • Graph the data and check for regular seasonal patterns. • Examine the data to see whether the seasonal autocorrelations of the residuals from an AR model are significant (for example, the fourth autocorrelation for quarterly data) and whether the autocorrelations before and after the seasonal autocorrelations are significant. To correct for seasonality, add seasonal lags to your AR model. For example, if you are using quarterly data, you might add the fourth lag of a time series as an additional variable in an AR(1) or an AR(2) model. 8. Next, test whether the residuals have autoregressive conditional heteroskedasticity. To test for ARCH(1), for example, do the following: • Regress the squared residual from your time-series model on a lagged value of the squared residual. • Test whether the coefficient on the squared lagged residual differs significantly from 0. • If the coefficient on the squared lagged residual does not differ significantly from 0, the residuals do not display ARCH and you can rely on the standard errors from your time-series estimates. • If the coefficient on the squared lagged residual does differ significantly from 0, use generalized least squares or other methods to correct for ARCH. 9. Finally, you may also want to perform tests of the model’s out-of-sample forecasting performance to see how the model’s out-of-sample performance compares to its insample performance. Using these steps in sequence, you can be reasonably sure that your model is correctly specified. © IFT. All rights reserved 6 Quantitative Methods 2024 Level II High Yield Notes LM06 Machine Learning Supervised machine learning, unsupervised machine learning, and deep learning Supervised machine learning makes use of labeled training data. It can be divided into two categories: • Regression: The target variable is continuous. • Classification: The target variable is categorical or ordinal. Unsupervised machine learning does not make use of labelled training data. The ML program has to discover structure within the data on its own. Two important types of problems well suited to unsupervised ML are: • Dimension reduction: Reducing the number of features (X variables). • Clustering: Sorting observations into groups. Deep learning refers to sophisticated algorithms which are used for highly complex tasks such as image classification, face recognition, speech recognition, and natural language processing. Overfitting Overfitting refers to an issue where the model fits training data perfectly but does not work well with out-of-sample data. There are two methods to reduce overfitting: • Preventing the algorithm from getting too complex: This is based on the principle that the simplest solution often tends to be the correct one. • Cross-validation: This is based on the principle of avoiding sampling bias. A commonly used technique is k-fold cross-validation. Here the data is shuffled randomly and then divided into k equal sub-samples, with k-1 samples used as training samples and one sample, the kth, used as a validation sample. The total out-of-sample error can be decomposed into: • Bias error: refers to the degree to which a model fits the training data. Underfitted models have high bias errors. • Variance error: refers to how much the model’s result change in response to new data. Overfitted models have high variance errors. • Base error: refers to errors due to randomness in the data. An overfitted model will have a low bias error but a high variance error. Generalization refers to the degree to which a model retains its explanatory power when predicting out-of-sample. A model that generalizes well has low variance error. © IFT. All rights reserved 1 Quantitative Methods 2024 Level II High Yield Notes Supervised machine learning algorithms In penalized regression, the regression coefficients are chosen to minimize the sum of squared residuals plus a penalty term that increases with the number of included variables. So, a feature must make a sufficient contribution to model fit to offset the penalty from including it. Because of this penalty, the model remains parsimonious and only the most important variables for explaining Y remain in the model. A popular type of penalized regression is LASSO. Support vector machine (SVM) is a linear classifier that aims to seek the optimal hyperplane – the one that separates the two sets of data points by the maximum margin. K-nearest neighbor (KNN) classifies a new observation by finding similarities (“nearness”) between it and its k-nearest neighbors in the existing data set. Classification and regression tree (CART) can be applied to predict a categorical variable or a continuous target variable. A binary CART tree is a combination of an initial root node, decision nodes, and terminal nodes. The root node and each decision node represent a single feature (f) and a cutoff value (c) for that feature. The CART algorithm iteratively partitions the data into sub-groups until terminal nodes are formed that contain the predicted label. A random forest classifier is a collection of many decision trees generated by a bagging method or by randomly reducing the number of features available during training. In ensemble learning, we combine predictions from a collection of models. This method typically produces more accurate and more stable predictions than the best single model. Unsupervised machine learning algorithms Principal components analysis (PCA) is used to reduce highly correlated features into a few uncorrelated composite variables. A composite variable is a variable that combines two or more variables that are statistically strongly related to each other. K-means algorithm repeatedly partitions observations into k non-overlapping clusters. The number of clusters, k, is a hyperparameter whose value must be set by the researcher before learning begins. Each cluster is characterized by its centroid and each observation is assigned to the cluster with the centroid to which that observation is closest. Hierarchical clustering algorithms create intermediate rounds of clusters in increasing or decreasing size until a final clustering is reached. Agglomerative clustering (or bottom-up) hierarchical clustering begins with each observation being treated as its own cluster. Divisive clustering (or top-down) hierarchical clustering starts with all the observations belonging to a single cluster. © IFT. All rights reserved 2 Quantitative Methods 2024 Level II High Yield Notes Neural networks, deep learning nets, and reinforcement learning Neural networks have layers of nodes connected by links. The three types of layers are: input layer, hidden layer and output layer. Learning takes place in the hidden layer through improvements in the weights applied to nodes. Neural networks with many hidden layers (at least 3 but often more than 20 hidden layers) are known as deep learning nets (DLNs). NNs and DLNs have been successfully applied to a wide variety of complex tasks characterized by non-linearities and interactions among features, particularly pattern recognition problems. Reinforcement learning (RL) algorithm involves an agent that should perform actions that will maximize its reward over time, taking into consideration the constraints of its environment. © IFT. All rights reserved 3 Quantitative Methods 2024 Level II High Yield Notes LM07 Big Data Projects Steps in a Data Analysis Project Exhibit 1 from the curriculum shows the steps involved in executing a data analysis project. The steps vary depending on whether we are working with structured data (data in tabular format) or unstructured data (textual data). Conceptualization: Define what the output of the model should be, (for example, will stock prices go up/down in a week from now), how this model will be used, who will use this model, and how this model will become part of the existing business process. Text problem formulation: Define the text classification problem, identify the exact inputs and outputs of the model. For example, computing sentiment scores (positive, negative, neutral) from the text data. Data Prep & Wrangling Structured data For structured data, data preparation and wrangling involve data cleansing and data preprocessing. Data cleansing involves resolving: • Incompleteness errors: Data is missing. • Invalidity errors: Data is outside a meaningful range. • Inaccuracy errors: Data is not a measure of true value. • Inconsistency errors: Data conflicts with the corresponding data points or reality. • Non-uniformity errors: Data is not present in an identical format. • Duplication errors: Duplicate observations are present. © IFT. All rights reserved 1 Quantitative Methods 2024 Level II High Yield Notes Data preprocessing involves performing the following transformations: • Extraction: A new variable is extracted from the current variable for ease of analyzing and using for training the ML model. • Aggregation: Two or more variables are consolidated into a single variable. • Filtration: Data rows not required for the project are removed. • Selection: Data columns not required for the project are removed. • Conversion: The variables in the dataset are converted into appropriate types to further process and analyze them correctly. Scaling is the process of adjusting the range of a feature by shifting and changing the scale of data. Two common methods used for scaling are: o Normalization: Xi(normalized) = o Xi −Xmin Xmax −Xmin Standardization: X i(standarized) = Xi −µ σ Unstructured data For unstructured data, data preparation and wrangling involve a set of text-specific cleansing and preprocessing tasks. Text cleansing involves removing the following unnecessary elements from the raw text: • Html tags • Most punctuations • Most numbers • White spaces Text preprocessing involves performing the following transformations: • Tokenization: the process of splitting a given text into separate tokens where each token is equivalent to a word. • Normalization: The normalization process involves the following actions: o Lowercasing – Removes differences among the same words due to upper and lower cases. o Removing stop words – Stop words are commonly used words such a ‘the’, ‘is’ and ‘a’. o Stemming – Converting inflected forms of a word into its base word. o Lemmatization – Converting inflected forms of a word into its morphological root (known as lemma). Lemmatization is a more sophisticated approach as compared to stemming and is difficult and expensive to perform. • Creating bag-of-words (BOW): It is a collection of distinct set of tokens that does not capture the position or sequence of the words in the text. • Organizing the BOW into a Document term matrix (DTM): It is a table, where each row of the matrix belongs to a document (or text file), and each column represents a token (or term). The number of rows is equal to the number of documents in the sample dataset. The number of columns is equal to the number of tokens in the final © IFT. All rights reserved 2 Quantitative Methods • 2024 Level II High Yield Notes BOW. The cells contain the counts of the number of times a token is present in each document. N-grams and N-grams BOW: In some cases, a sequence of words may convey more meaning than individual words. N-grams is a representation of word sequences. The length of a sequence varies from 1 to n. A one-word sequence is a unigram; a twoword sequence is a bigram, and a 3-word sequence is a trigram; and so on. Data Exploration Exploratory data analysis (EDA) is a preliminary step in data exploration that involves summarizing and observing data. The objectives of EDA include: • Serving as a communication medium among project stakeholders • Understanding data properties • Finding patterns and relationships in data • Inspecting basic questions and hypotheses • Documenting data distributions and other characteristics • Planning modeling strategies for the next steps Visualization techniques for EDA include: histograms, box plots, scatterplots, word clouds, etc. In general, EDA helps identify the general trends in the data as well as the relationships between data. These relationships and trends can be used for feature selection and feature engineering. Feature selection is the process of selecting only the relevant features from the data set to reduce model complexity. Feature selection methods used for text data include: • Term frequency (TF): Features with very high or very low term frequencies are removed. These represent noisy features. • Document frequency (DF): The DF of a token is defined as the number of documents (texts) that contain the respective token divided by the total number of documents. This measure also helps in identifying and removing noisy features. • Chi-square test: This test allows us to rank tokens by their usefulness to each class in text classification problems. Features with high scores can be retained whereas features with low scores can be removed. • Mutual information measure: This measure tells us how much information is contributed by a token to a class of texts. The MI score ranges from 0 – 1. A score close to 0 indicates that the token is not very useful and can be removed. Whereas, a score close to 1 indicates that the token is closely associated with a particular class and should be retained. Feature engineering is the process of creating new features by changing or transforming existing features. Feature engineering for text data includes: © IFT. All rights reserved 3 Quantitative Methods • • • • 2024 Level II High Yield Notes Converting numbers into tokens: To preserve the meaning conveyed by numbers of different length they are converted into different tokens. For example, four-digit numbers are replaced with “/number4/,” 10-digit numbers with “/number10/,”. Creating n-grams: Sometimes multi-word sequences can carry more meaning than an individual word. N-grams are used to keep the connection between words intact. Using name entity recognition (NER): NER is an algorithm that allows us to identify the names of organizations, date, money, time, etc. and tag certain tokens with these names. Using parts of speech (POS): Similar to NER, POS algorithms allows us to tag each token with the corresponding part of speech such as noun, verb, adjective, proper noun etc. Model Training Model training consists of three major tasks: method selection, performance evaluation, and model tuning. Method selection: This decision is based on the following factors: • Whether the data project involves labeled data (supervised learning)or unlabeled data (unsupervised learning) • Type of data: numerical, continuous, or categorical; text data; image data; speech data; etc. • Size of the dataset Performance evaluation: Commonly used techniques are: • Error analysis using confusion matrix: A confusion matrix is created with four categories - true positives, false positives, true negatives and false negatives. The following metrics are used to evaluate a confusion matrix: Precision (P) = TP/(TP + FP) Recall (R) = TP/(TP + FN) Accuracy = (TP + TN)/(TP + FP + TN + FN) F1 score = (2 * P * R)/(P + R) The higher the accuracy and the F1 score, the better the model performance. • Receiver Operating Characteristic (ROC): ROC curves and area under the curve (AUC) of various models are calculated and compared. An area under the curve (AUC) close to 1 indicates a perfect model. Whereas, an AUC of 0.5 indicated random guessing. In other words, a more convex curve indicates better model performance. • Calculating root mean squared error (RMSE): The root mean squared error is computed by finding the square root of the mean of the squared differences between the actual values and the model’s predicted values (error). The model with the smallest RMSE is the most accurate model. © IFT. All rights reserved 4 Quantitative Methods 2024 Level II High Yield Notes Model tuning • Involves managing the trade-off between model bias error (which is associated with underfitting) and model variance error (which is associated with overfitting). • ‘Grid search’ is a method of systematically training an ML model by using different hyperparameter values to determine which values lead to best model performance. • A fitting curve of in-sample error and out-of-sample error on the y-axis versus model complexity on the x-axis is useful for managing the trade-off between bias and variance errors. Evaluating the Fit of a ML algorithm Fitting describes the degree to which an ML model can be generalized to new data. • Underfit: Model does not fit the training data well. • Overfit: Model is complex and fits the training data too well. It is unlikely to perform well on out-of-sample data. • Good fit: Model fits the training data well and yet is simple enough so that it can generalize well to out-of-sample data. © IFT. All rights reserved 5