Lecture 11 - Median & Monotonic Transformation

Lecture 11 Median and monotonic transformation If g(y) is monotonic and not decreasing, the original order is kept If g(y) is monotonic and increasing, the order is inverse Linear transformations An important monotonic transformation is linear: g(y) = a + by Example Celsius to Fahrenheit: F = 1.8C + 32 Data: Celsius (18, 20, 21), Fahrenheit (64.4, 68, 69.8) Mean: C_mean = 19.6, F_mean = 67.4 Median: C_median = 20, F_median = 68 Variance: σ^2_C = 1.56, σ^2_F = 5.04 The mean of the transformation is linked to the linear function The median of the transformation is linked to the linear function The variance of the transformation is not linked to the linear function, but it can be expressed in terms of the variances of the original and transformed variables. Table: Celsius Fahrenheit 18 64.4 20 68 21 69.8 Not equivalence of the mean in general Mean is a transformation T(g(y)) non-linear on the data It is not, in general, equal to the same transformation applied to the mean: T(g(y)) ≠ g(T(y)) Logarithmic transformation When the data are highly asymmetric, a transformation such as logarithm can be useful to improve visualization. Graph A graph showing the distribution of Facebook friends and Twitter followers. Variability concept: mutability or heterogeneity Units are homogeneous with respect to a given variable if they all assume the same value. Minimum mutability We obtain minimum mutability when all the units present the same identical value of a variable. In this case, the statistical units are perfectly homogeneous with respect to the phenomenon considered. The distribution of the absolute frequencies is: modality yi absolute frequencies 0 0 N The corresponding distribution of relative frequencies is: modality absolute frequencies 0 1 Maximum mutability We obtain maximum mutability when all the modalities present the same identical frequency. The distribution of the absolute frequencies is: modality y1 y2 ... yk absolute frequencies N/k N/k ... N/k The corresponding distribution of relative frequencies is: modality y1 y2 ... yk absolute frequencies 1/k 1/k ... 1/k Gini index e_s = 1 - Σ(fi - f_i)^2 = 1 - Σf_i^2 e_n = Σ(fi * (1 - fi)) = 1 - Σf_i^2 Relative frequencies table Modality | n | f | f^2 A | 18 | 0.3 | 0.09 B | 12 | 0.2 | 0.04 C | 30 | 0.5 | 0.25 Example e_n = 1 - Σf_i^2 = 1 - 0.38 = 0.62 The Gini index cancels out with a table of minimum mutability It assumes maximum value when mutability is maximum, that is when all relative frequencies take on the value f_i = 1/k. e_n ≤ Σ(1/k * (1 - 1/k)) = (1 - 1/k) Example Modality | n | f | f^2 A | 20 | 0.33 | 0.1089 B | 20 | 0.33 | 0.1089 C | 20 | 0.33 | 0.1089 e_n = 1 - Σf_i^2 = 1 - 0.3267 = 0.6733 To analyze variables We have analyzed the distribution of a variable Y conditioning it by different values observed for another variable X We will see statistical tools to analyze this case (bivariate analysis) When we itemize N pairs of modalities or values, we will talk of double disaggregated distributions As for distribution for one single variable, we will talk of double frequency distribution, when observations are grouped in modalities or classes or intervals Example: hours of sleep Suppose a double variable (Y, X) = (hours of sleep, sex) Absolute frequency distribution is given by: [A table with rows and columns representing different combinations of hours of sleep and sex] This double frequency distribution "includes several frequency distribution" The center of the distribution shows the number of units that present a precise value for the couple (Y, X) The first column shows the distribution of the hours of sleep among females The second column shows the distribution of the hours of sleep among males The third row shows, among people who sleep 7 hours a night, how many are females and how many males The last column shows the distribution of sleep hours regardless of sex The last line shows the distribution of the sex variable Contingency table A double frequency distribution is usually called a two-way contingency table. A two-way contingency table is like this: x1 ... xk Total y1 n11 ... n1k n1. ... ... ... ... ... ys ns1 ... nsk ns. Total n.1 ... n.k N X and Y are the two variables under analysis xi are the modalities of X yi are the modalities of Y nij is the frequency for Yi and Xj n.j is the marginal frequency of the modality xj of X n1. is the marginal frequency of the modality yi of Y