University:
California State University, Northridge
Course:
ME 209 | Programming for Mechanical Engineers
Academic year:

2011
Views:

285

Pages:

19
Author:

germanokrt

Data Preprocessing Why Data Preprocessing Is Important? • Welcome to the Real World! • No quality data, no quality mining results! • Preprocessing is one of the most critical steps in a data mining process 6 Major Tasks in Data P Preprocessing i Figure 2.1 Forms of data preprocessing 7 Why Data Preprocessing is Beneficial to D Data Mi Mining? i ? • Less data – data mining methods can learn faster • Higher Hi h accuracy – data mining methods can generalize better • Simple results – they are easier to understand • Fewer attributes – For the next round of data collection, saving can be made by removing redundant and irrelevant features 8 Data Cleaning 9 Remarks on Data Cleaning • “Data cleaning is one of the biggest problems in data warehousing” ‐‐ Ralph Kimball • “Data cleaning is the number one problem in data warehousing” ‐‐ DCI survey 10 Why Data Is “Dirty”? • IIncomplete, l noisy, i and d inconsistent i i data d are commonplace properties of large real‐world databases …. (p (p. 48) • There are many possible reasons for noisy data …. (p. 48) 11 Types of Dirty Data Cleaning Methods • Missing values – Fill in missing values • Noisy data (incorrect values) – Identify outliers and smooth out noisy data 12 Methods for Missing Values (1) • Ignore the tuple • Fill in the missing value manually • Use a global constant to fill in the missing value 13 Methods for Missing Values (2) • Use the attribute mean to fill in the missing value • Use the attribute mean for all samples belonging to the same class as the given tuple • Use the most probable value to fill in the missing value 14 Methods for Noisy Data • Binning • Regression • Clustering 15 Binning 16 Regression 17 Clustering Figure 2.12 A 2‐D plot of customer data with respect to customer locations in a city, showing three data clusters. Each cluster centroid is marked with a “+”, representing the average point i t on space that th t cluster. l t Outliers O tli may be b detected d t t d as values l that th t fall f ll outside t id off the sets of clusters. 18 Data Integration 19 Data Integration • Schema integration and object matching – Entity identification problem • Redundant data (between attributes) occur often when integration of multiple databases – Redundant attributes may be able to be detected by correlation l ti analysis, l i and d chi‐square hi method th d 20 Schema Integration and Object Matching • custom_id and cust_number – Schema conflict • “H” and ”S”, and 1 and 2 for pay_type in one database – Value conflict • Solutions – meta t data d t (data (d t about b td data) t ) 21 Detecting Redundancy (1) • If an attributed can be “derived” from another attribute or a set of attributes, it may be redundant 22 Detecting Redundancy (2) • Some redundancies can be detected by correlation analysis – Correlation coefficient for numeric data – Chi‐square test for categorical data • These can be also used for data reduction 23

Data Preprocessing

of 19

Description

Free up your schedule!

Take 5 seconds to unlock