Overview of Some Probability Distributions

Lecture 1 Overview of some probability distributions. In this lecture we will review several common distributions that will be used often throughtout the class. Each distribution is usually described by its probability function (p.f.) in the case of discrete distributions or probability density function (p.d.f.) in the case of continuous distributions. Let us recall basic definitions associated with these two cases. Discrete distributions. Suppose that a set X consists of a countable or finite number of points, X = {a1 , a2 , a3 , · · ·}. Then a probability distribution P on X can be defined via a function p(x) on X with the following properties: 1. 0 ≈ p(ai ) ≈ 1, �� 2. i=1 p(ai ) = 1. A function p(x) is called the probability function. If X is a random variable with distribution P then p(ai ) = P(X = ai ) - a probability that X takes value ai . Given a function � : X � R, the expectation of �(X) is defined by E�(X) = � � �(ai )p(ai ). i=1 (Absolutely) continuous distributions. Continuous distribution P on R is defined via a probability density function (p.d.f.) p(x) on R such that � � p(X) ∼ 0 and p(X)dx = 1. −� If a random variable X has distribution P then the probability that X takes a value in the interval [a, b] is given by � b P(X ∞ [a, b]) = p(x)dx. a Clearly, in this case for any a ∞ R we have P(X = a) = 0. Given a function � : X � R, the expectation of �(X) is defined by � � E�(X) = �(x)p(x)dx. −� Notation. The fact that a random variable X has distribution P will be denoted by X � P. Normal (Gaussian) Distribution N (�, π 2 ). Normal distribution is a continuous dis­ tribution on R with p.d.f. p(x) = ≤ (x−�)2 1 e− 2�2 for x ∞ (−→, →). 2ϕπ Here −→ < � < →, π > 0 are the parameters of the distribution. Let us recall some properties of a normal distribution. If a random variable X has a normal distribution N (�, π 2 ) then the r.v. X −� Y = � N (0, 1) π has a standard normal distribution N (0, 1). To see this, we can write, � bα+� �X − � � (x−�)2 1 ≤ P ∞ [a, b] = P(X ∞ [aπ + �, bπ + �]) = e− 2�2 dx π 2ϕπ aα +� � b y2 1 ≤ e− 2 dy, = 2ϕ a where in the last integral we made a change of variables y = (x − �)/π. This, of course, means that Y � N (0, 1). The expectation of Y is � � y2 1 EY = y ≤ e− 2 dy = 0 2ϕ −� since the integrand is an odd function. To compute the second moment EY 2 , let us first note y2 that since �12� e− 2 is a probability density function, it integrates to 1, i.e. � � y2 1 ≤ e− 2 dy. 1= 2ϕ −� If we integrate this by parts, we get, � � � � 2 �� y2 1 − y2 1 y − y2 � 2 ≤ e dy = ≤ ye � ≤ (−y)e− 2 dy 1 = − −� 2ϕ 2ϕ 2ϕ −� −� � � 2 y 1 = 0+ y 2 ≤ e− 2 dy = EY 2 . 2ϕ −� Thus, the second moment EY 2 = 1. The variance of Y is Var(Y ) = EY 2 − (EY )2 = 1 − 0 = 1. It is now easy to compute the mean and the variance of X = � + πY � N (�, π 2 ), EX = � + πEY = �, EX 2 = E(�2 + 2�πY + π 2 Y 2 ) = �2 + π 2 , Var(X) = EX 2 − (EX)2 = �2 + π 2 − �2 = π 2 . Thus, parameter � is a mean and parameter π 2 is a variance of a normal distribution. Let us recall (without giving a proof) that if we have several, say n, independent random variables Xi , 1 ≈ i ≈ n, such that Xi � N (�i , πi2 ) then their sum will also have a normal distribution X1 + . . . + Xn � N (�1 + . . . + �n , π12 + . . . + πn2 ). Normal distribution appears in one of the most important results that one learns in probabil­ ity class, namely, a Central Limit Theorem (CLT), which states the following. If X1 , . . . , Xn is an i.i.d. sample such that π 2 = Var(X) < →, then n ≤ 1 � ≤ (Xi − EXi ) = n(X̄ − EX1 ) �d N (0, π 2 ) n i=1 converges in distribution to a normal distribution with zero mean and variance π 2 , where convergence in distribution means that for any interval [a, b], P �≤ � n(X̄n − EX1 ) ∞ [a, b] � � b a ≤ x2 1 e− 2�2 dx. 2ϕπ This result can be generalized for a sequence of random variables with different distributions and it basically says that the sum of many independent random variables/factors approxi­ mately looks like a normal distribution as long as each factor has a small impact on the total sum. A consequence of this phenomenon is that a normal distribution gives a good approxi­ mation for many random objects that by nature are affected by a sum of many independent factors, for example, person’s height or weight, fluctuations of a stock’s price, etc. Bernoulli Distribution B(p). This distribution describes a random variable that can take only two possible values, i.e. X = {0, 1}. The distribution is described by a probability function p(1) = P(X = 1) = p, p(0) = P(X = 0) = 1 − p for some p ∞ [0, 1]. It is easy to check that EX = p, Var(X) = p(1 − p). Binomial Distribution B(n, p). This distribution describes a random variable X that is a number of successes in n trials with probability of success p. In other words, X is a sum of n independent Bernoulli r.v. Therefore, X takes values in X = {0, 1, . . . , n} and the distribution is given by a probability function � � n k p(k) = P(X = k) = p (1 − p)n−k . k It is easy to check that EX = np, Var(X) = np(1 − p). Exponential Distribution E(�). This is a continuous distribution with p.d.f. � �e−�x x ∼ 0, p(x) = 0 x < 0. Here, � > 0 is the parameter of the distribution. Again, it is a simple calculus exersice to check that 1 1 EX = , Var(X) = 2 . � � This distribution has the following nice property. If a random variable X � E(�) then probability that X exceeds level t for some t > 0 is � � P(X ∼ t) = P(X ∞ [t, →)) = �e−�x dx = e−�t . t Given another s > 0, the conditional probability that X will exceed level t + s given that it will exceed level t can be computed as follows: P(X ∼ t + s, X ∼ t) P(X ∼ t + s) = P(X ∼ t) P(X ∼ t) −�(t+s) −�t −�s = e /e =e = P(X ∼ s), P(X ∼ t + s|X ∼ t) = i.e. P(X ∼ t + s|X ∼ t) = P(X ∼ s). If X represent a lifetime of some object in some random conditions, then the above property means that the chance that X will ”live” longer then t + s given that it will ”live” longer than t is the same as the chance that X will live longer than t in the first place. Or, in other words, if X is “alive” at time t then it is ”like new”. Therefore, some natural examples that can be described by exponential distribution are the lifetime of high quality products (or, possibly, soldiers in combat). Poisson Distribution �(�). This is a discrete distribution with X = {0, 1, 2, 3, . . .}, p(k) = P(X = k) = �k −� e for k = 0, 1, 2, , . . . k! It is an exercise to show that EX = �, Var(X) = �. Poisson distribution could be used to describe the following random objects: the number of stars in a random area of the space; number of misprints in a typed page; number of wrong connections to your phone number; distribution of bacteria on some surface or weed in the field. All these examples share some common properties that give rise to a Poisson distribution. Suppose that we count a number of random objects in a certain region T and this counting process has the following properties: 1. Average number of objects in any region S ≥ T is proportional to the size of S, i.e. ECount(S) = �|S |. Here |S | denotes the size of S, i.e. length, area, volume, etc. Parameter � > 0 represents the intensity of the process. 2. Counts on disjoint regions are independent. 3. Chance to observe more than one object in a small region is very small, i.e. P(Count(S) ∼ 2) becomes small when the size |S | gets small. PSfrag replacements We will show that under these assumptions will imply that the number Count(T ) of objects in the region T has Poisson distribution �(�|T |) with parameter �|T |. 0 X1 T n T ....... X2 Xn − Counts on small subintervals Figure 1.1: Poisson Distribution For simplicity, let us assume that the region T is an interval [0, T ] of length T. Let us split this interval into a large number n of small equal subintervals of length T /n and denote by Xi the number of random objects in the ith subinterval, i = 1, . . . , n. By the first property above, �T EXi = . n On the other hand, by definition of expectation � EXi = kP(Xi = k) = 0 + P(Xi = 1) + αn , k�0 � where αn = k�2 kP(Xi = k), and by the last property above we assume that αn becomes small with n, since the probability to observe more that two objects on the interval of size T /n becomes small as n becomes large. Combining two equations above gives, P(Xi = 1) � � Tn . Also, since by the last property the probability that any count Xi is ∼ 2 is small, i.e. P(at least one Xi ∼ 2) ≈ no �T � n � 0 as n � →, Count(T ) = X1 + · · · + Xn has approximately binomial distribution B(n, �|T |/n) and we can write � �� � � n �T k �T �n−k P(Count(T ) = X1 + · · · + Xn = k) � 1− n k n (�T )k −�T � e . k! The last limit is a simple calculus exercise and this is also a famous Poisson approximation of binomial distribution taught in every probability class. Uniform Distribution U[0, λ]. This distribution has probability density function � 1 , x ∞ [0, λ], λ p(x) = 0, otherwise. Matlab review of probability distributions. Matlab Help/Statistics Toolbox/Probability Distributions. Each distribution in Matlab has a name, for example, normal distribution has a name ’norm’. Adding a suffix defines a function associated with this distribution. For example, ’normrnd’ generates random numbers from distribution ’norm’, ’normpdf’ gives p.d.f., ’norm­ cdf’ gives c.d.f., ’normfit’ fits the normal distribution for a given dataset (we will look at this last type of functions when we discuss Maximum Likelihood Estimators). Please, look at each function for its syntax, input, output, etc. Type ’help normrnd’ to quickly see how the normal random number generator works. Also, there is a graphic user interface tools like ’disttool’ (to run it just type disttool in the main Matlab window) that allows you to play with different distributions, or ’randtool’ that generates and visualizes random samples from different distributions.