Data Science Overview

Data Science Overview -​ -​ -​ -​ -​ -​ Week 1, data science overview Data science, also known as data-driven science, is an interdisciplinary field of scientific methods, processes, algorithms and systems to extract knowledge or insights from data in various forms, either structured or unstructured, similar to data mining. The three V's—Volume (amount of data), Velocity (speed of data processing), and Variety (different data types)—are fundamental characteristics that define Big Data. Data Science can be done on small data sets, and not everything done using Big Data would necessarily be called Data Science, but there is overlap. This overlap comes with the three V’s mentioned earlier and something bigger (unicorn? Not sure what this means in ppt) Big data without data science is machine learning, while data science without big data are just generic data sets. Python, R, SQL, and Excel are what are mainly used. Data Science Basics -​ -​ Week 1, machine learning lecture PPT Supervised learning is predictive learning. Predictive learning attempts to learn a model from data X which predicts a variable y (ie. type of movie, number of views would be X, y is your rating). (pages 16-22) -​ -​ -​ -​ Algorithms include: Linear Regression (Regression)/Logistic Regression(Classification). Random Forest/Decision Trees/Gradient Boosting. SVM (Support Vector Machines) and nonlinear kernels. -​ Unsupervised learning is descriptive. Try to infer a hidden structure in the data without proper training examples (no teacher, hence ‘unsupervised’). (pages 23-25) -​ -​ -​ -​ -​ -​ Algorithms Include: K-means clustering. Decision Tree Clustering. SVM Topic Models (LDA, etc) Gaussian Mixture Models -​ -​ -​ -​ (Pg 33-36) Lazy learner: Just store Data set without learning from it Start classifying data when it receives Test data, so it takes less time learning and more time classifying data -​ -​ Eager learner: When it receive data set it starts classifying (learning) Then it does not wait for test data to learn, so it takes a long time learning and less time classifying data Lazy examples: K - Nearest Neighbor, Case - Based Reasoning Eager examples: Decision Tree, Naive Bayes, Artificial Neural Networks -​ -​ Linear Regression basics -​ -​ -​ -​ -​ -​ -​ -​ -​ All of week 2 ppts cover this The goal is to learn a model from the training data and make predictions. How to train/learn model parameters: defining a cost function & minimizing it A cost function measures how bad (or good) a candidate model is. Specifically, measure the difference between predictions from our model and the training data. The objective is to minimize the cost function so as to minimize the difference between predictions and observations (Explained more in pages 21-26) The process of learning a model from training data is essentially the process of minimizing a cost function. Optimization: finding optimal parameter values that minimize a cost function Page 47 onwards is an example 2nd ppt: -​ -​ -​ -​ -​ -​ -​ -​ -​ Regression problems are supervised learning problems in which the response is continuous Classification problems are supervised learning problems in which the response is categorical Benefits of linear regression: widely used runs fast easy to use (not a lot of tuning required) highly interpretable basis for many other methods Examples on Page 6-32. Goes over stuff like .describe() -​ -​ -​ -​ -​ -​ -​ -​ -​ -​ -​ -​ Issues that can arise during the analysis: Linearity - the relationships between the predictors and the outcome variable should be linear Normality - the errors should be normally distributed - technically normality is necessary only for the t-tests to be valid, estimation of the coefficients only requires that the errors be identically and independently distributed Homogeneity of variance (homoscedasticity) - the error variance should be constant Independence - the errors associated with one observation are not correlated with the errors of any other observation Errors in variables - predictor variables are measured without error Model specification - the model should be properly specified (including all relevant variables, and excluding irrelevant variables) The following are unusual and infuenial data Outliers: In linear regression, an outlier is an observation with large residual. In other words, it is an observation whose dependent-variable value is unusual given its values on the predictor variables. An outlier may indicate a sample peculiarity or may indicate a data entry error or other problem. Leverage: An observation with an extreme value on a predictor variable is called a point with high leverage. Leverage is a measure of how far an observation deviates from the mean of that variable. These leverage points can have an effect on the estimate of regression coefficients. Influence: An observation is said to be influential if removing the observation substantially changes the estimate of coefficients. Influence can be thought of as the product of leverage and outlierness. Page 38 onwards is examples of categorical stuff Working with data -​ -​ -​ -​ -​ -​ -​ -​ -​ -​ -​ Week 3 lecture 1, data objects Curse of dimensionality: When dimensionality increases, data becomes increasingly sparse in the space that it occupies Dimensionality reduction Purpose: Avoid curse of dimensionality Reduce amount of time and memory required by data mining algorithms Allow data to be more easily visualized May help to eliminate irrelevant features or reduce noise Techniques of dimensionality reduction Principal Components Analysis (PCA) Singular Value Decomposition -​ -​ -​ -​ -​ -​ -​ -​ Others: supervised and non-linear techniques Check Pages 18-20 for attribute information An attribute transformation is a function that maps the entire set of values of a given attribute to a new set of replacement values such that each old value can be identified with one of the new values Normalization refers to various techniques to adjust to differences among attributes in terms of frequency of occurrence, mean, variance, range In statistics, standardization refers to subtracting off the means and dividing by the standard deviation Reasons for missing values: Information is not collected ​ (e.g., people decline to give their age and weight) Attributes may not be applicable to all cases ​ (e.g., annual income is not applicable to children) -​ -​ -​ -​ -​ -​ Handling missing values: Eliminate data objects or variables Estimate missing values Example: time series of temperature Example: census results Ignore the missing value during analysis -​ Duplicates: Data set may include data objects that are duplicates, or almost duplicates of one another. Major issue when merging data from heterogeneous sources -​ -​ -​ -​ Entropy: The entropy is a measure of the uncertainty associated with a random variable As uncertainty and or randomness increases for a result set so does the entropy Values range from 0 – 1 to represent the entropy of information Explained more in pages 30-37. Explanation for calculating in pages 35-41 -​ -​ -​ -​ -​ -​ -​ -​ -​ -​ -​ Motivation To better understand the data: central tendency, variation and spread Data dispersion characteristics median, max, min, quantiles, outliers, variance, etc. Numerical dimensions correspond to sorted intervals Data dispersion: analyzed with multiple granularities of precision Boxplot or quantile analysis on sorted intervals Dispersion analysis on computed measures Folding measures into numerical dimensions Boxplot or quantile analysis on the transformed cube Explained more from pages 46-51 -​ -​ -​ -​ -​ -​ -​ -​ -​ Pages 53-62 explain graphs more, but here’s some basic descriptions Boxplot: graphic display of five-number summary Histogram: x-axis are values, y-axis repres. frequencies Quantile plot: each value xi is paired with fi indicating that approximately 100 fi % of data are xi Quantile-quantile (q-q) plot: graphs the quantiles of one univariant distribution against the corresponding quantiles of another Scatter plot: each pair of values is a pair of coordinates and plotted as points in the plane Data discretization is a method of converting attributes values of continuous data into a finite set of intervals with minimum data loss. In contrast, data binarization is used to transform the continuous and discrete attributes into binary attributes. Explained more in 63-74 Data processing lecture -​ -​ -​ -​ -​ Missing data: There are various ways for us to handle it Ignore Fill manually Imputation replacing missing value Explained more in pages 10-13 -​ Categorical variables are basically the variables that are discrete and not continuous. Ex — color of an item is a discrete variables whereas its price is a continuous variable. Categorical variables are further divided into 2 types: Ordinal categorical variables — These variables can be ordered. Ex — Size of a T-shirt. We can say that M< L< XL. Nominal categorical variables — These variables can’t be ordered. Ex — Color of a T-shirt. We can’t say that Blue< Green as it doesn’t make any sense to compare the colors as they don’t have any relationship. Explained more in pages 15-17 -​ -​ -​ -​ -​ -​ Data reduction: Obtain a reduced representation of the data set that is much smaller in volume but yet produces the same (or almost the same) analytical results Explained more on pages 39-43 -​ All of the week 4 slides should be focused on in length