University:
California State University, Northridge
Course:
ME 209 | Programming for Mechanical Engineers
Academic year:

2011
Views:

41

Pages:

25
Author:

cliceannaq71

Chi-square Chi square Test • For categorical (discrete) data, a correlation relationship between two attributes, A and B, can be discovered by a χ2 test • Given the degree of freedom, the value of χ2 is used to decide correlation based on a significance level 24 Chi-square Test for Categorical D t Data (Observed − Expected ) χ =∑ Expected 2 2 (o − e ) χ 2 = ∑∑ eij i =1 j =1 c r ij ij 2 count ( A = ai ) × count ( B = bj ) eij = N p. 68 The larger the Χ2 value, the more likely the variables are related. 25 Chi-square Chi square Test fiction non_ffiction Total male 250 female 200 Total 450 50 300 1000 1200 1050 1500 Table2.2 A 2 X 2 contingency table for the data of Example 2.1. Are gender and preferred_reading correlated? The χ2 statistic tests the hypothesis that gender and preferred_reading are independent. The test is based on a significant level, with (r ‐ 1) x (c ‐ 1) degree of freedom. 26 Table of Percentage Points of th χ2 the 2 Distribution Di t ib ti 27 Correlation Coefficient N N ∑ (a − A)(b − B) ∑ (a b ) − N AB i rA, B = i =1 i NσAσB − 1 ≤ rA, B ≤ +1 i i = i =1 NσAσB p. 68 28 29 Data Transformation 30 Data Transformation/Consolidation • • • • • Smoothing √ Aggregation Generalization Normaliza on √ Attribute construc on √ 31 Smoothing • Remove noise from the data • Binning, regression, and clustering 32 Data Normalization • Min Min-max max normalization v − minA v' = (new _ maxA − new _ minA) + new _ minA maxA − minA • z-score normalization v' = v − μA σ A • Normalization by decimal scaling v v' = j 10 where j is the smallest integer such that Max(|ν′|) < 1 33 Data Normalization • Suppose that the minimum and maximum values for attribute income are $12,000 and $98,000, respectively. We would like to map income to the range [0.0, [0 0 1.0]. 1 0] Do Min Min‐max max normalization, z‐score normalization, and decimal scaling for the attribute income 34 Attribution Construction • New attributes are constructed from given attributes and added in order to help improve accuracy and understanding of structure in high‐dimension data • Example – Add the attribute area based on the attributes height and width 35 Data Reduction 36 Data Reduction • Data reduction techniques can be applied to obtain a reduced representation of the data set that is much smaller in volume volume, yet closely maintains the integrity of the original data 37 Data Reduction • • • • • • (Data Cube)Aggregation Attribute (Subset) Selection Dimensionality Reduction Numerosity Reduction Data Discretization C Concept t Hierarchy Hi h G Generation ti 38 “The The Curse of Dimensionality”(1) Dimensionality (1) • Size – The size of a data set yielding the same density of data points in an n‐dimensional space increase exponentially with dimensions • Radius – A larger radius is needed to enclose a faction of the data points in a high‐dimensional space 39 “The The Curse of Dimensionality”(2) Dimensionality (2) • Distance – Almost every point is closer to an edge than to another sample point in a high‐dimensional space • Outlier – Almost every point is an outlier in a high‐dimensional space 40 Data Cube Aggregation • Summarize (aggregate) data based on dimensions • The resulting data set is smaller in volume, without loss of information necessary for analysis task • Concept hierarchies may exist for each attribute, allowing the analysis of data at multiple levels of abstraction 41 Data Aggregation Figure 2.13 Sales data for a given branch of AllElectronics for the years 2002 to 2004. On the left, the sales are shown per quarter. On the right, the data are aggregated to provide the annual sales 42 Data Cube • Provide fast access to pre‐computed, summarized data, thereby benefiting on on‐line line analytical processing as well as data mining 43 Data Cube - Example Figure 2.14 A data cube for sales at AllElectronics 44 Attribute Subset Selection (1) • Attribute selection can help in the phases of data mining (knowledge discovery) process – By attribute selection, • we can improve data mining performance (speed of l learning, i predictive di i accuracy, or simplicity i li i off rules) l ) • we can visualize the data for model selected • we reduce dimensionality and remove noise noise. 45 Attribute Subset Selection (2) • Attribute (Feature) selection is a search problem – Search directions • (Sequential) Forward selection • (Sequential) Backward selection (elimination) • Bidirectional selection • Decision D i i ttree algorithm l ith (i (induction) d ti ) 46 Attribute Subset Selection (3) • Attribute (Feature) selection is a search problem – Search strategies • Exhaustive E h ti search h • Heuristic search – Selection criteria • Statistic significance • Information gain g • etc. 47 Attribute Subset Selection (4) Figure 2.15. Greedy (heuristic) methods for attribute subset selection 48

Chi-square Test

of 25

Description

Free up your schedule!

Take 5 seconds to unlock