Chapter 07-08: Machine Learning and Big Data

CHAPTER 7. MACHINE LEARNING 1. Definition ▪ Statistical approaches rely on assumptions and models of structure → These can fail in reality. ▪ Machine learning seeks to extract knowledge from large amounts of data with no such restrictions. ▪ The goal of machine learning algorithms is to automate decision-making processes by generalizing (i.e., “learning”) from known examples to determine an underlying structure in the data. ▪ Machine learning techniques are better able than statistical approaches to handle problems with:  Many variables (high dimensionality)  High degree of nonlinearity. 2. Types of Learning ▪ ▪ Compare ML terms with regression terms:  Target variable = Dependent variable (i.e., the y variable).  Features = Independent variables (i.e., the x variables).  Training data set = the sample used to fit the model.  Hyperparameter: This is a model input specified by the researcher. Supervised Learning  ML algorithms that use labeled training data – the data set contains matched sets of observed inputs and the associated output.   ▪ Multiple regression is an example of supervised learning. Supervised learning can be divided into two categories of problems:  Regression: Focuses on making predictions of continuous target variables  Classification: Classification models are used in cases where the target variable is categorical (e.g., from low to high creditworthiness) or ordinal (fraud detection for credit card transactions) Unsupervised Learning  ML algorithms that do not use labeled training data; instead, inputs (i.e., features) are provided without any conclusions about those inputs.  In the absence of any target variable, the program seeks out structure or interrelationships in the data.  Unsupervised learning can be divided into two categories of problems: 1 ▪  Dimension reduction: Focuses on reducing the number of features while retaining variation across observations to preserve the information contained in that variation.  Clustering: Focuses on sorting observations into groups (clusters) such that observations in the same cluster are more similar to each other than they are to observations in other clusters. Deep Learning and Reinforcement Learning  Deep learning and reinforcement learning are themselves based on neural networks (NNs, also called artificial neural networks, or ANNs).  Deep Learning: Sophisticated algorithms address highly complex tasks, such as image classification, face recognition, speech recognition, and natural language processing.  In reinforcement learning, a computer learns from interacting with itself (or data generated by the same algorithm) 3. Overview of Evaluating ML Algorithm Performance ▪ An ML algorithm that fits the training data too well will typically not predict well using new data. This problem is known as overfitting, and it means that the fitted algorithm does not generalize well to new data. ▪ A model that generalizes well is a model that retains its explanatory power when predicting out-of-sample. 3.1. Measure Generalization of a Model ▪ ▪ To measure how well a model generalizes, data analysts create three nonoverlapping data sets:  (1) training sample (used to develop the model).  (2) validation sample (used for tuning the model).  (3) test sample (used for evaluating the model using new data). Measure errors: 2 ▪ ▪  Bias error: From (1). This is the in-sample error resulting from models with a poor fit.  Variance error: From (2) and (3). This is the out-of-sample error resulting from overfitted models that do not generalize well.  Base error: These are residual errors due to random noise. A learning curve:  Plots the accuracy rate (1 − error rate) in the validation or test sample versus the size of the training sample.  Models with high bias error (1st panel): the error rates converge, but far below the desired level.  Models with high variance error (2nd panel): the in-sample accuracy rate converge toward the desired level, while the out-of-sample accuracy rate lags far behind.  A robust, well-generalizing model (3rd panel): accuracy rate as the sample size is increased, and the in-sample and out-of-sample error rates will converge toward a desired accuracy level. Fitting curve:  Show the trade of between cost (model error) and complexity (number of features).  Variance error increases with model complexity, while bias error decrease with complexity. 3 3.2. Reduce Overfitting ▪ Complexity reduction: a penalty is imposed to exclude features that are not meaningfully contributing to out-of-sample prediction accuracy. The penalty value increases with the number of independent variables used by the model. ▪ Cross validation: Use when we don’t have a large enough data set → estimates out-of-sample error rates directly form validation sample.  K-fold cross-validation: sample is randomly divided equally into k parts. The training sample comprises (k-1) parts, with one left for validation. This process is repeated k times, the average of the k validation errors is then taken as a reasonable estimate of the model’s out-of-sample error. 4. Supervised Machine Learning Algorithms 4.1. Penalized Regression ▪ Penalized regression includes a constraint such that the regression coefficients are chosen to minimize the sum of squared residuals (SSE) plus a penalty term that increases in size with the number of included features. ▪ In one popular type of penalized regression, LASSO (least absolute shrinkage and selection operator), the penalty term has the following form, with λ > 0: 𝑲 ̂𝑲 | 𝑷𝒆𝒏𝒂𝒍𝒕𝒚 𝒕𝒆𝒓𝒎 = 𝝀 ∑|𝒃 𝒌=𝟏 ▪ Penalized regression models seek to minimize the sum of square errors (SSE) + penalty value: 𝒏 𝑲 ̂𝑲 | ̂ 𝒊 ) + 𝝀 ∑|𝒃 ∑(𝒀𝒊 − 𝒀 𝟐 𝒊=𝟏 𝒌=𝟏 4 ▪ Lambda (λ) is a hyperparameter - a parameter whose value must be set by the researcher before learning begins - of the regression model and will determine the balance between fitting the model versus keeping the model parsimonious. ▪ What if Lambda (λ) = 0? → OLS ▪ Regularization describes methods that reduce statistical variability in high dimensional data estimation problems. ▪ Parsimonious: a parsimonious model is one that accomplishes the required level of explanation using as few predictor variables as possible. 4.2. Support Vector Machine (SVM) ▪ Linear classifier - a binary classifier that makes its classification decision based on a linear combination of the features of each data point. ▪ SVM is a linear classifier that optimally separates the observations into two sets of data points. ▪ SVM maximizes the probability of making a correct prediction by determining the boundary that is farthest way from all the observations:  The straight line in the middle of the shaded strip is the discriminant boundary, or boundary.  The margin is determined by the observations closest to the boundary these observations are called support vectors → Maximize margin. 5 ▪ Soft margin classification: use to handle misclassified observations by adding a penalty to the objective function for observations in the training set that are misclassified → trade-off between wider margin and lower total error penalty. ▪ Non-linear SVM can be used to reduce classification error but requires more features and may result in overfitting. ▪ Applications of SVM in investment management:  Classifying debt issuers into likely-to-default versus not-likely-to-default issuers, stocks-to-short versus not-to-short. 4.3. K-Nearest Neighbor (KNN) ▪ Used to classify a new observation by finding similarities (“nearness”) between this new observation and the existing data. ▪ The researchers specify the value of k, the hyperparameter, triggering the algorithm to look for the k observations in the sample that are closest to the new observation that is being classified. ▪ The specification of k is important because if it is too small, it will result in a high error rate, and if it is too large, it will dilute the result by averaging across too many outcomes. Also, if k is even, there may be ties, with no clear winner. 6 ▪ ▪ Problems of KNN:  Meaning of “near”.  The choice of a correct distance measure may be even more subjective for ordinal or categorical data → need clear understanding of the data and underlying business.  K must be chosen with the understanding.  KNN results can be sensitive to inclusion of irrelevant or correlated features, so it may be necessary to select features manually → KNN algorithms tend to work better with a small number of features. Investment applications of KNN:  Predicting bankruptcy, assigning a bond to a ratings class, predicting stock prices. 4.4. Classification And Regression Trees (CART) ▪ Applied to either binary classification or regression. ▪ Example: whether or not a company increase their dividends to shareholders? Such a classification requires a binary tree: a combination of an initial root node, decision nodes, and terminal nodes. The root node and each decision node represent a single feature (f) and a cutoff value (c) for that feature. ▪ From top feature, there is a cutoff value c is estimated. Observations with feature values greater than c are assigned to on classification, and the remainder are assigned to the other classification and then repeat to second features. The tree stops when the error cannot be reduced further, resulting in a terminal node. 7 ▪ The CART algorithm chooses the feature and the cutoff value at each node that generates the widest separation of the labeled data to minimize classification error. 8 ▪ CART makes no assumptions about the characteristics of the training data, so if left unconstrained, potentially it can perfectly learn the training data → Overfitting. How to ovoid?  Regularization parameters such as maximum tree depth, maximum number of decision nodes may be used to avoid overfitting.  Pruning: sections of free with minimal explanatory power are pruned (removed). ▪ CART is popular because it provides a visual explanation of the prediction process, compared to other algorithms that are often described as black boxes. ▪ Investment applications of CART include detecting fraudulent financial statements and selecting stocks and bonds. 4.5. Ensemble Learning ▪ Ensemble learning is the technique of combining predictions from multiple models rather than a single model. The ensemble method results in a lower average error rate because the different models cancel out noise. ▪ Ensemble learning can be divided into two main categories:   Aggregation of heterogenous learners:  Different algorithms are combined together via a voting classifier.  A majority-vote classifier will assign to a new data point the predicted label with the most votes. Aggregation of homogenous learners:  The same algorithm is used but on different training data.  Bootstrap aggregating (or bagging) is a technique whereby the original training data set is used to generate n new training data sets or bags of data.  Each new bag of data is generated by random sampling with replacement from the initial training set. The algorithm can now be trained on an independent data sets that will generate n new models. Then use voting classifier. 4.6. Random Forest ▪ A random forest classifier is a collection of a large number of decision trees trained via a bagging method. ▪ How it works?  Use bagging to create many independent data sets.  For each data set, randomly select m < n total number of features then use CART to create decision tree → many decision tree → random forest.  Any new observation will be classified by majority vote → implementing a machine learning version of the “wisdom of crowds.” 9 ▪ Reduce of noise because errors cancel out across collection of slightly different trees. ▪ However, it lacks the ease of interpretability of individual trees → black box-type algorithm. ▪ Investment applications of random forest include factor-based asset allocation, and prediction models for the success of an IPO. 5. Unsupervised Machine Learning Algorithms 5.1. Principal Component Analysis (PCA) ▪ PCA is used to summarize or reduce highly correlated features of data into a few main, uncorrelated composite variables. ▪ A composite variable is a variable that combines two or more variables that are statistically strongly related to each other. ▪ PCA transforms the covariance matrix of the features and involves two key concepts: Eigenvectors and eigenvalues.  Eigenvectors (Principal components) define new, mutually uncorrelated composite variables that are linear combinations of the original features.  Eigenvalue gives the proportion of total variance in the initial data that is explained by each eigenvector. ▪ The PCA algorithm operates in such a way that it finds PC1 by selecting the line for which the sum of the projection errors for all data points is minimized and for which the sum of the spread between all the data is maximized. ▪ The next largest portion of the remaining variance is best explained by PC2, which is at right angles to PC1 and thus is uncorrelated with PC1. 10 ▪ Trade-off between lower-dimensional, more manageable view of a complex data set and some loss of information. ▪ Scree plots shows the proportion of total variance in the data explained by each principal component. ▪ The main drawback of PCA is that since the principal components are combinations of the data set’s initial features, they typically cannot be easily labeled or directly interpreted by the analyst → Black box. 5.2. Clustering ▪ Clustering organizes data points into similar groups called clusters ▪ A cluster contains a subset of observations from the data set such that all the observations within the same cluster are “similar.” ▪ A good clustering of the data need:  Cohesion: The observations inside each cluster are similar or close to each other  Separation: The observations in two different clusters are as far away from one another or are as dissimilar as possible. ▪ Requires expert human judgement: what it means to be “similar”? ▪ Euclidian distance: the straight-line distance between two points. 5.3. K-Means Clustering 11 ▪ An algorithm that repeatedly partitions observations into a fixed number, k, of nonoverlapping clusters. ▪ Each cluster is characterized by its centroid, and each observation is assigned by the algorithm to the cluster with the centroid to which that observation is closet. ▪ The k-means algorithm will continue to iterate until no observation is reassigned to a new cluster. ▪ Limitations:  ▪ k must be decided before k-means can be run → solution: run the algorithm using a range of values for k to find the optimal number of clusters. K-means clustering is used in investment management to classify thousands of securities based on patterns in high dimensional data. 5.4. Hierarchical Clustering ▪ Builds a hierarchy of clusters without any predefined number of clusters. ▪ Agglomerative (bottom-up) clustering begins with each observation being treated as its own cluster. Then the algorithm finds the two closet clusters, defined by some measure of distance, and combines them into one new larger cluster. This process is repeated iteratively until all observations are clumped into a single cluster. ▪ A divisive (top-down) clustering starts with one giant cluster, and then it partitions that cluster into smaller and smaller clusters. 12 13 CHAPTER 8. BIG DATA PROJECTS 1. Big Data in Investment Management ▪ ▪ Big data is characterized by the three Vs of volume, variety, and velocity:  Volume refers to the quantity of data. Big data refers to a huge volume of data.  Variety refers to data sources. Big data is collected from various sources: user-generated, traditional transactional, emails, images, clickstreams, and so on.  Velocity refers to the speed with which the data is created and collected (e.g., all the social media postings created during a specified time interval). When use big data to make prediction:  the veracity or validity of the data, needs to be considered. Not all data sources are reliable, and the researcher has to separate quality from quantity to generate robust forecasts. 2. Data Analysis Steps ▪ For structured data – Data that are organized in a systematic format that is readily searchable and readable by computer operations:  Step 1: Conceptualization of the modeling task: Define the problem  Step 2: Data collection: The researcher has to determine which sources (internal or external) to use to collect this data.  Step 3: Data preparation and wrangling:  Data preparation (Cleansing): Clean the data set and preparing it for the model. Cleaning the data set includes addressing any missing values or verification of any out-of-range values.  Data wrangling (Preprocessing): Preprocessing data may involve aggregating, filtering, or extracting relevant variables.  Step 4: Data exploration: This step involves feature selection and engineering as well as initial (i.e., exploratory) data analysis.  Step 5: Model training: Select appropriate ML algorithm to use, evaluating the algorithm using a training data set, and tuning the model. ▪ These five steps are iterative. Depending on the quality of the output, the researcher can go back and tweak any of the steps involved to improve the model. ▪ For unstructured data - Data are not organized into any systematic format that can be processed by computers directly. For example, suppose the researcher wants to also incorporate a borrower’s social media posts in the determination of credit scores. The first four steps would then be modified as follows:  Step 1: Text problem formulation: Determine the problem and identify the exact inputs and output of the model. 14 ▪  Step 2: Data curation (collection): Determine the sources of data to be used (e.g., web scouring, specific social media sites). If using supervised learning, annotating a reliable target variable is also necessary.  Step 3: Text preparation and wrangling. Convert unstructured data structured data.  Step 4: Text exploration. This involves text visualization as well as text feature selection and engineering. The output of a model using unstructured data may then be used in isolation, or combined with other structured variables as an input into another model. 3. Data Preparation and Wrangling ▪ ▪ Do this step after  Step 1. Define problem  Step 2. Data collection: Data collection involves downloading data from internal and external sources. While accessing a database, appropriate caution must be exercised to ensure data validity. README files associated with a database usually contain information about how, what, and where data is stored:  External data can also be obtained from third-party vendors using an application programming interface (API).  External data comes with a price: It saves time and money in the next steps, but other market participants can use the same data, diminishing the firm’s proprietary advantage. Once the data are collected, the data preparation and wrangling stage begins. This stage involves two important tasks: Cleansing (Data preparation) and Preprocessing (Data Wrangling), respectively. 3.1. For Structured Data 3.1.1. Data Preparation (Cleansing) ▪ Structured data are organized in a systematic format that is readily searchable and readable by computer operations for processing and analyzing. ▪ The data cleansing process mainly deals with identifying and mitigating all errors including:  Incompleteness error: missing data → omitted or replaced with “NA” for deletion or substitution. Example: rows 4 (ID 3), 5 (ID4), 6 (ID 5), and 7 (ID6) 15  Invalidity error: data are outside of a meaningful range → verifying other administrative data records. Example: Date of birth in Row 5.  Inaccuracy error: the data are not a measure of true value → rectified with the help of business records and administrators. Example: Credit card in Row 5.  Inconsistency error: the data conflict with the corresponding data points or reality → This contradiction should be eliminated by clarifying with another source. Example: Gender in row 3 (ID 2).  Non-uniformity error: the data are not present in an identical format → converting the data points into a preferable standard format. Example: Monetary unit in Salary and Other Income.  Duplication error: duplicate observations are present → removing the duplicate entries. Example: Row 6 and row 3 are identical. ▪ Example: ▪ Data cleansing is accomplished via automated, rules-based algorithms as well as human intervention. Metadata (summary data) for the raw data set may serve as a starting point for error identification. Observations with erroneous values that cannot be cleansed would be dropped. 3.1.2. Data Wrangling (Preprocessing) ▪ Data preprocessing primarily includes transformations and scaling of the data. ▪ Transformations:  Extraction (e.g., extracting number of years employed based on dates provided).  Aggregation, which involves consolidating two related variables into one, using appropriate weights.  Filtration, which involves removing irrelevant observations.  Selection, which involves removing features (i.e., data columns) not needed for processing.  Conversion of data of diverse types (e.g., nominal, ordinal). 16 ▪ Outlier:   ▪ Detect:  Use standard deviation: data value that is outside of 3 standard deviations from the mean may be considered an outlier.  Use interquartile range: IQR is the difference between the 75th and the 25th percentile values of the data. Handle:  Trimming: The highest and lowest x% of observations are excluded.  Winsorization: Extreme values may be replaced by the maximum value allowable for that variable. Scaling is a process of adjusting the range of a feature by shifting and changing the scale of data:  Normalization: Normalization scales variable values between 0 and 1. 𝑵𝒐𝒓𝒎𝒂𝒍𝒊𝒛𝒆 𝑿𝒊 =  𝑿𝒊 − 𝑿𝒎𝒊𝒏 𝑿𝒎𝒂𝒙 − 𝑿𝒎𝒊𝒏 Standardization centers the variables at 0 and scales them as units of standard deviations from the mean. 𝑺𝒕𝒂𝒏𝒅𝒂𝒓𝒅𝒊𝒛𝒆𝒅 𝑿𝒊 =  𝑿𝒊 − 𝝁 𝝈 Unlike normalization, standardization is not sensitive to outliers, but assumes that the variable is normally distributed. 3.2. For Unstructured Data ▪ Unstructured data constitute approximately 80% of the total data available today. ▪ Unstructured, text-based data is more suitable for human use rather than for processing by a computer. For analysis, unstructured data has to be converted into structured data. ▪ The cleansing and processing of text data is called text processing. ▪ Steps in Text Preparation (Cleansing):  1. Remove HTML tags. Text collected from web pages has embedded HTML tags, which may need to be removed before processing. A regular expression (regex) is a text string used to identify characters in a particular order.  2. Remove punctuations. Text analysis usually does not need punctuations, so these need to be removed as well. Some punctuations (e.g., %, $, ?) may be needed for analysis, and if so, they are replaced with annotations (i.e., textual expressions) for model training.  3. Remove numbers. Digits are removed or replaced with annotations to let the ML program know that a number is present, but its value is not 17 important in the analysis. If the value of a number is important for analysis, such values are first extracted via text applications.  ▪ 4. Remove white spaces. Extra formatting-related white spaces (e.g., tabs, indents) do not serve any purpose in text processing and are removed. Steps in Text Wrangling (Preprocessing):  Step 1. Tokenization: In text wrangling, a token is a word, and tokenization is the process of splitting a sentence into tokens. 18   Step 2. Normalization:  1. Lowercasing. So as to not discriminate between market and Market.  2. Removal of stop words. In some ML applications, stop words such as the, is, and so on do not carry any semantic meaning; hence, they are removed to reduce the number of tokens in the training data.  3. Stemming. This is a rules-based algorithm that converts all variations of a word into a common value. For example, integrate, integration, and integrating are all assigned a common value of integrat. While stemming makes the text confusing for human processing, it is ideally suited for machines.  4. Lemmatization. This involves the conversion of inflected forms of a word into its lemma (i.e., morphological root). Lemmatization is similar to stemming, but is more computationally advanced and resource intensive. For example: analyzed, analyzing, analyzes should be noted as analyze. Step 3. Create a bag-of-word (BOW): After the data is cleansed text is normalized, a bag-of-words (BOW) procedure is applied, which simply collects all the words or tokens without regard to the sequence of occurrence. 19  Step 4. Create a document term matrix (DTM): DTM is then used to convert the unstructured data into structured data. In this matrix, each text document is a row, and the columns are represented by tokens. The cell value represents the number of occurrences of a token in a document (i.e., row).  If the sequence of text is important, N-grams can be used to represent word sequences. A two-word sequence is a bigram, a three-word sequence is trigram, and so forth. Consider the sentence, “The market is up today.” Bigrams of this sentence include “the_market,” “market_is,” “is_up,” and “up_today.” BOW is then applied to the bigrams instead of the original words. N-gram implementation will affect the normalization of the BOW because stop words will not be removed. 4. Data Exploration ▪ Steps in data exploration include the following:  Exploratory data analysis (EDA) involves looking at data descriptors such as summary statistics, heat maps, word clouds, and so on. The objectives of EDA include: 20 ▪  Understanding characteristics.  Finding patterns or relationships, and evaluating basic questions and hypotheses. Planning modeling in future steps. data properties, distributions, and other  Feature selection is a process to select only the needed attributes of the data for ML model training. The higher the number of features selected, the higher the model complexity and training time.  Feature engineering is the process of creating new features by transforming or combining multiple features. Model performance depends heavily on feature selection and engineering, and it is common for the analyst to recursively go back to this step and apply tweaks until model performance is acceptable. 4.1. Structured Data 4.1.1. EDA ▪ When the number of features is large, dimension reduction models such as principal component analysis (PCA) can facilitate data exploration. ▪ For one-dimensional data, summary statistics include the mean, standard deviation, skewness, and kurtosis can be used. EDA visualizations include:  Histograms: Histograms capture the frequency of observations in a series of equal-width bins. Density plots are smoothed histograms for continuous data overlaid on top of histograms.  Bar charts show frequency distributions of categorical variables (e.g., proportion of the population in each of the five geographical zones of the country).  Box plots are used for continuous variables and highlight the median, quartiles, and outliers of a normally distributed feature.  Density plots are smoothed histograms and are commonly laid on top of histograms. 21  For two or more dimensions, the summary statistic can be a correlation matrix. Data can be visually graphed using a scatterplot, box plots, stacked bar or a line graph. 4.1.2. Feature selection ▪ With feature selection, we try to select only the features that contribute to the outof-sample predictive power of the model. ▪ A parsimonious model (i.e., a model with fewer features) reduces feature-induced noise and improves the model’s prediction accuracy. ▪ Feature selection requires a good understanding of the business environment and the interrelationships among the features identified in the EDA. ▪ For structured data, feature selection is an iterative, methodical process. Features can be assigned an importance score using statistical methods and then ranked and selected based on that score. Dimension reduction algorithms may be employed to reduce the number of features needed so as to reduce processing time during model training. 4.1.3. Feature engineering (FE) ▪ FE involves optimizing and improving the selected features: ▪ Model training results depend on how the features are presented to the algorithm. Feature engineering involves either decomposing a feature into multiple features or converting an existing feature into a new feature.  For continuous data: a new feature may be created – for example, by taking the logarithm of the product of two or more features.  For categorical data: a new feature can be a combination of two features or a decomposition of one feature into many.  One-hot encoding (OHE) is a process used to convert a categorical feature into a binary (dummy) variable suitable for machine processing. Feature engineering seeks to make model training faster and easier. 22 4.2. Unstructured Data 4.2.1. EDA ▪ Unstructured text can be tokenized, and summary statistics include:  Term frequency: number of times the word appears in the text.  Co-occurrence: where two or more words appear together.  A word cloud: a visual representation of all the words in a BOW, such that words with higher frequency have a larger font size. This allows the analyst to determine which words are contextually more important. Figure below shows an example of a word cloud. 4.2.2. Feature Selection ▪ Involves selecting a subset of tokens in the BOW. Reduction in BOW size makes the model more parsimonious and reduces feature-induced noise. ▪ Noisy features do not contribute to prediction accuracy. High- and low-frequency words are often eliminated, resulting in a more concise BOW.  High-frequency words tend to be stop words (if not removed during the data wrangling phase) or common vocabulary words.  Low-frequency words may be irrelevant. Consider a model to predict bankruptcy: to effectively separate defaulters from non-defaulters, tokens that are associated with both categories should be removed. 4.2.3. Future Engineering ▪ Techniques of Future Engineering include:  Numbers. Tokens with standard lengths are identified and converted into a token such as /numberX/. Four-digit numbers may be associated with years and are assigned a value of /number4/. 23  N-grams. These are multiword patterns, and if they are useful, the order is preserved. For example, the words expansionary monetary policy may be best kept as a sequence rather than broken into three different tokens, and therefore would be replaced by a single token, expansionary_monetary_policy.  Name entity recognition (NER). NER algorithms search for token values, in the context it was used, against their internal library and assign a NER tag to the token. For example, Microsoft would be assigned a NER tag of ORG and Europe would be assigned a NER tag of Place. NER object class assignment is meant to make the selected features more discriminatory.  Parts of speech (POS). This uses language structure dictionaries to contextually assign tags (POS) to text. For example, market could be a verb as in “to market” or a noun as in “in the market”. 5. Model Training ▪ The three tasks of ML model training are method selection, performance evaluation, and tuning. 5.1. Method Selection ▪ Method selection is the art and science of choosing the appropriate ML method (i.e., algorithm) given the objectives and data characteristics. Method selection is based on the following factors:  Supervised or unsupervised learning. Supervised learning is used when the training data contains the ground truth or the known outcome (i.e., the target variable). In such cases, available methods include regression, ensemble trees, support vector machines (SVMs), and neural networks (NN). Unsupervised learning occurs when there is no target variable. Unsupervised learning methods include clustering, dimension reduction, and anomaly detection.  Type of data. For numerical data (e.g., predicting earnings) we may use classification and regression tree (CART) methods. For text data, we can use generalized linear models (GLMs) and SVMs. For image data, neural networks and deep learning methods can be employed.  Size of data. Large data sets with many observations and features can be handled with SVMs. Neural networks work better with a large number of observations, but few features. ▪ Once a method is selected, the researcher has to specify appropriate hyperparameters (e.g., the number of hidden layers in a neural network). For mixed data sets (containing numerical and textual data), multiple methods are often used. ▪ For supervised learning, before model training begins, the data set is divided into three parts. The larger part (≈ 60%) is used for model training. A second part (≈ 20%) is used for validation and model tuning. The last part (≈ 20%) is the test set, 24 and is used to check the out-of-sample performance of the model. Due to the absence of labeled training data, no splitting of the data set is needed for unsupervised learning. ▪ Class imbalance occurs when one class has a large number of observations relative to other classes.  For example, in a model for predicting bond default, if the data set has a large number of high-grade bonds (i.e., those that would be less likely to default), then the model would be more likely to predict nondefault for a new observation. The training data set should have a variety of high- and lowgrade bonds so as to have enough diversity to make correct predictions.  One way to overcome class imbalance is to undersample the overrepresented class and oversample the underrepresented class. 5.2. Performance Evaluation ▪ It is important to measure the model training performance or goodness of fit for validation of the model. These techniques are particularly suited to evaluating binary classification models 25 ▪ 1. Error analysis. Errors in classification problems can be false positives (type I error) or false negatives (type II error). A confusion matrix shows the results of a classification problem as below. Think confusion matrix like a Covid-test: 0 = No covid, 1 = Have covid.   Metrics such as precision (the ratio of true positives to all predicted positives) and recall (the ratio of TPs to all actual positives) can be used.  Precision (P) = TP / (TP + FP)  Recall (R) = TP / (TP + FN)  While FP and FN are both errors, they may not be equally important and are subject to business decision. For example, a lender may want to avoid lending to potential defaulters, and so will want to maximize recall.  High precision is valued when the cost of a type I error is large, while high recall is valued when the cost of a type II error is large. Overall performance valuation metrics:  Accuracy is the percentage of correctly predicted classes out of total predictions. Accuracy = (TP + TN) / (TP + TN + FP + FN)  The F1 score is the harmonic mean of precision and recall. F1 score is more appropriate (than accuracy) when unequal class distribution is in the dataset. F1 score = (2 × P × R) / (P + R) 26 ▪ 2. Receiver operating characteristic (ROC). Also used for classification problems, the ROC plots a curve showing the tradeoff between FPs and TPs. The true positive rate (TPR) is the same as recall, and is plotted along the Y-axis. The false positive rate (FPR) is the ratio of FPs to all actual negatives, and is plotted along the X-axis. TPR = TP / (TP + FN) FPR = FP / (FP + TN) 27 ▪ 3. Root mean square error (RMSE). This is useful for data predictions that are continuous, such as regression models. The RMSE is a single metric summarizing the prediction error in a sample. 𝒏 𝑹𝑴𝑺𝑬 = √∑ 𝒊=𝟏 (𝑷𝒓𝒆𝒅𝒊𝒄𝒕𝒆𝒅𝒊 − 𝑨𝒄𝒕𝒖𝒂𝒍𝒊 )𝟐 𝒏 5.3. Model Tuning ▪ After model evaluation, the model needs to be revised until it reaches an acceptable performance level. It is necessary to find an optimum tradeoff between bias and variance errors, such that the model is neither underfitting nor overfitting. ▪ Parameters are estimated by the model (e.g., slope coefficients in a regression model) using an optimization technique on the training sample. 28 ▪ Hyperparameters (e.g., the number of hidden layers in NN, or the p-threshold in logistic regression) are specified by ML engineers, and are independent of the training sample. ▪ Tuning involves altering the hyperparameters until a desirable level of model performance is achieved. For each specification of hyperparameter(s), a confusion matrix is prepared based on the classification results, and accuracy and F1 scores are compiled. 29