Probability Theory

Lecture 3 : Probability Theory 1. Terminology and review We consider real-valued discrete random variables and continuous random variables. A discrete random variable X is given by its probability mass function P which is a non-negative real valued function fX : Ω → R≥0 satisfying x∈Ω fX (x) = 1 for some finite domain Ω known as the sample space. For example,   2/3 (x = 1) fX (x) = 1/3 (x = −1) ,   0 (otherwise) denotes the probability mass function of a discrete random variable X which takes value 1 with probability 2/3 and −1 with probability 1/3. A continuous random variable Y is given by its probability density function which is a non-negative real valued function fY : Ω → R≥0 satisfying ´ f (y) dy = 1 (we will mostly consider cases when the sample space Ω is Y Ω the reals R). For example, ( 1 (y ∈ [0, 1]) fY (y) = , 0 (otherwise) denotes the probability density function of a continuous random variable Y which takes a uniform value in the interval [0, 1]. For a given set A ⊆ Ω, one can compute the probability of the events X ∈ A and Y ∈ A as ˆ X P(X ∈ A) = fX (x), P(Y ∈ A) = fY (y)dy. A x∈A The expectation (or mean) of a random variable is defined as ˆ X E[X] = xfX (x), E[Y ] = yfY (y)dy. y x A cumulative distribution function of a random variable is defined as FX (x) = P(X ≤ x). Two random variables X and Y are independent if P({X ∈ A} ∩ {Y ∈ B}) = P(X ∈ A)P(Y ∈ B) 1 Lecture 3 2 for all sets A and B. Random variables X1 , X2 , · · · , Xn are mutually independent if P({X1 ∈ A1 } ∩ {X2 ∈ A2 } ∩ · · · ∩ {Xn ∈ An }) = n Y P(Xi ∈ Ai ), i=1 for all sets Ai . They are pairwise independent if Xi and Xj are independent for all i 6= j. Two random variables are uncorrelated if E[XY ] = E[X]E[Y ]. 1.1. Normal and lognormal distributions. We first review one of the most important distributions in probability theory. Definition 1.1. For reals −∞ < µ < ∞ and σ > 0, the normal distribution (or Gaussian distribution) denoted N (µ, σ 2 ), with mean µ and variance σ 2 is a continuous random variable with probability density function 1 2 2 f (x) = √ e−(x−µ) /(2σ ) (−∞ < x < ∞). σ 2π Exercise 1.2. Verify, by computation, that the mean of the normal distribution is µ. To model the stock market, it is more reasonable to assert that the rate of change of the stock price has normal distribution (compared to the stock price itself having normal distribution). Log-normal distribution can be used to model such situation, where a log-normal distribution is a probability distribution whose (natural) logarithm has normal distribution. In other words, if the cumulative distribution function of normal distribution is F (x), then that for log-normal distribution is F (ln x). To obtain the probablility distribution of the log-normal distribution, we can use the change of variable formula. Theorem 1.3. (Change of variable) Suppose that X (resp. Y ) is a random variable over reals with probability distribution fX (x) (resp. fY (y)) and cumulative distribution function FX (x) (resp. FY (y)). Further suppose that FX , FY are differentiable, and there exists a function h such that FY (y) = FX (h(y)). Then fY (y) = dFY dFX = = fX (h(y)) · h0 (y). dy dy The term h0 (y) is called the Jacobian. (to avoid unnecessary technicality, we assume that all functions are differentiable) We can now compute the probability density function of log-normal distribution (or define it using this formula). Definition 1.4. For reals −∞ < µ < ∞ and σ > 0, the log-normal distribution with parameters µ and σ is a distribution with probability density Lecture 3 3 function g(x) = 1 √ xσ 2π e−(ln x−µ) 2 /(2σ 2 ) , x > 0. Exercise 1.5. (i) Compute the expectation and variance of log-normal distribution (HINT: it suffices to compute the case when µ = 0. You can use the fact that f (x) given above is a p.d.f. in order to simply the computation). (ii) Show that the product of two independent log-normal distributions is also a log-normal distribution. Some examples of other importatnt distributions that will repeatedly occur throughout the course are Poisson distribution and exponential distribution. All these distributions (including the normal and log-normal distribution) are examples of exponential families which are defined as distributions whose probability distribution function can be written in terms of a vector parameter θ as follows: ! k X (θ) f (x) = h(x)c(θ)exp wi (θ)ti (x) , i=1 for some c(θ) ≥ 0 and real-valued functions w1 (θ), · · · , wk (θ). For example, the pdf of log-normal distribution can be written as   (ln x)2 µ(ln x) 1 1 µ2 g(x) = · √ · exp − + − 2 , x σ 2π 2σ 2 σ2 2σ and thus for θ = (µ, σ), we may let h(x) = 12 , c(θ) = µ , σ2 2 2 √1 e−µ /2σ , σ 2π w1 (θ) = and t2 (x) = ln x. Exponential families − 2σ1 2 , t1 (x) = (ln x)2 , w2 (θ) = are known to exhibit many nice statistical properties. Exercise 1.6. In this course, we will mostly be interested in the ‘statistics’ of a random variable. Here are two topics that we will address today. (1) Moment generating function : the k-th moment of a random variable is defined as EX k . The moments of a random variable contain essential statiscial information about the random variable. The moment generating function is a way of encoding these information into a single function. (2) Long-term (large-scale) behavior : we will study the outcome of repeated independent trials (realizations) of a same random variable. 2. Moment generating function Definition 2.1. (Moment generating function) A moment generating function of a given random variable X is a function MX : R → R defined as MX (t) = E[etX ]. We note that not all random varaibles allow a moment generating function, since the sum on the right hand side might not converge. For example, Lecture 3 4 the log-normal distribution does not have a moment generating function. In fact, E[etX ] does not exist for all t 6= 0. The name ‘moment generating function’ comes from the fact that the k-th k moment can be compted as EX k = d dxMkX (0) for all k ≥ 0. Therefore, the moment generating function (if exists) can be also written as MX (t) = ∞ k X t k=0 k! mk , where mk = EX k is the k-th moment of X. Theorem 2.2. (i) Let X and Y be two random variable with the same moment generation function, i.e., MX (t) = MY (t) for all t. Then X and Y have the same distribution, i.e., the cumulative distribution functions are the same. (ii) Suppose that X is a random variable with moment generating function MX (t) and continuous cumulative distribution function FX (x). Let X1 , X2 , · · · , be a sequence of random variables such that lim MXi (t) = MX (t), i→∞ for all t. Then Xi converges to X in distribution, i.e., for all real x, we have lim FXi (x) = FX (x). i→∞ We will not prove this theorem here. See [1] for its proof. One should be careful when considering part (i) of this theorem. It does not imply that all two random variables with same moments have the same distribution (see [1] page 106-107). This can happen because the moment generating function need not exist. 3. Law of large numbers Theorem 3.1. (Weak law of large number) Suppose that i.i.d. (independent identically distributed) random variables X1 , X2 , · · · , Xn of mean µ and variance σ 2 are given, and let X = n1 (X1 + · · · + Xn ). Then for all positive ε, P(|X − µ| ≥ ε) → 0 as n → ∞. Proof. By linearity of expectation, we see that n 1X E[X] = E[Xi ] = µ, n i=1 and thus E[X − µ] = 0. Next step is to compute the variance of X. We have " # n 1 X 2 2 V[X] = E[(X − µ) ] = E (Xi − µ) , n i=1 Lecture 3 5 which by the fact that Xi are independent, becomes " n # X 1 σ2 2 2 E[(X − µ) ] = 2 E (Xi − µ) = . n n i=1 Since ε2 P(|X − µ| ≥ ε) ≤ E[(X − µ)2 ] = σ2 , n we see that V[X] σ2 = , ε2 nε2 where the right hand side tends to zero as n tends to infinity. P(|X − µ| ≥ ε) ≤  The theorem stated above is in a very weak form; (1) the conditions can be relaxed and still give the same conclusion, (2) a stronger conclusion can be deduced from the same assumption (strong law of large number). See [1] for more information. Example 3.2. All the games played in the casino that you play against the casino (for example, roulette) are designed so that casino has about 1˜5% advantage over the player when the player plays the optimal strategy. They are also designed so that the variance σ is large compared to this average gain. Hence from the player’s perspective, it looks as if anything can happen over a small period of time and that one has a reasonable chance of making money. From the casino’s perspective, as long as they can provide independent outcomes for each game, the law of large number tells that they are going to make money. This does not apply to games that player plays with each other (e.g, poker). For these games, the casino makes money by charging certain amount of fee for playing each game. 4. Central limit theorem In the law of large numbers, we considered the random variable Yn = Pn σ2 i=1 Xi , which has variance V[Yn ] = n . Suppose that the mean of each X √i is zero so that Yn has mean 0, and consider the random variable Zn = nYn . Note that Zn has mean 0 and variance σ 2 , which is the same as Xi . Will the random variable Zn behave similarly to Xi ? If not, can we say anything interesting about it? The following theorem provides a surprising answer to this question. 1 n Theorem 4.1. (Central limit theorem) Let X1 , X2 , · · · be i.i.d. random variables with mean µ and variance σ 2 and moment generating function
√  1 Pn M (t). Then as n tends to infinity, the random variable Zn = n n i=1 Xi −  µ converges in distribution to the normal distribution N (0, σ 2 ). Lecture 3 6 Proof. We may replace Xi with Xi − µ, and assume that µ = 0. Consider the moment generating function of Zn : MZn (t) = E[etZn ] = E[e t √1n Pn i=1 Xi ]. Since Xi are independent random variables, we have n Y √ t MZn (t) = E[etXi / n ] = M ( √ )n . n i=1 Since ∞ 2 X tk t 1 2 2 3/2 k 2 t EX = 1 + σ M(√ ) = + O( 3/2 ) = eσ t /(2n)+O(1/n ) , k/2 2n n n k! n k=0 we have n  22 t 3/2 M ( √ )n = eσ t /(2n)+O(1/n ) n = eσ 2 t2 /2+O(1/n1/2 ) . 2 2 Therefore, for all t, we have MZn (t) → eσ t /2 as n → ∞. This is the moment generating function of N (0, σ 2 ). Therefore, the distribution of Zn converges to the normal distribution N (0, σ 2 ).  Example 4.2. (Estimating the mean of a random variable) Suppose that we observed n independent samples X1 , X2 , · · · , Xn of some random variable X, and are trying to estimate the mean of X based on this random variable. A reasonable guess is to take 1 Yn = (X1 + X2 + · + Xn ). n (we will see in future lectures how to justify this choice for certain random variables, and how this is not the best choice for certain random variables). Note that this is itself is a random variable. Law of large number tells us that this converges √ to the actual mean. Central limit theorem tells us the distribution of nYn . References [1] R. Durrett, Probability: Theory and Examples, 3rd edition.