The Delta-Method and Asymptotics of Some Estimators

THE DELTA-METHOD AND ASYMPTOTICS OF SOME ESTIMATORS The delta-method gives a way that asymptotic normality can be preserved under nonlinear, but differentiable, transformations. The method is well known; one version of it is given in J. Rice, Mathematical Statistics and Data Analysis, 2d. ed., 1995. A simple form of it using only a first derivative, for functions of one variable, will be given here. (A multidimensional version is used in Section 3.7 of Mathematical Statistics, 18.466 course notes by R. Dudley, on the MIT OCW website.) Theorem. Let Yn be a sequence of real-valued random variables such that for some µ and √ ∞ to N (0, σ 2). Let f be a function from σ, n(Yn − µ) converges in distribution as n → √ R into R having a derivative f
(µ) at µ. Then n[f (Yn ) − f (µ)] converges in distribution as n → ∞ to N (0, f
(µ)2 σ 2 ). Remarks. In statistics, where µ is an unknown parameter, one will want f to be differentiable at all possible µ (and preferably, for f
to be continuous, although that is not needed in the proof). √ Proof. We have Yn −µ = Op (1/ n) as n → ∞. Also, f (y) = f (µ)+f
(µ)(y−µ)+o(|y−µ|) as y → µ by definition of derivative. Thus f (Yn ) = f (µ) + f
(µ)(Yn − µ) + op (|Yn − µ|), so √ √ √ √ n[f (Yn ) − f (µ)] = f
(µ) n(Yn − µ) + nop (1/ n). � The last term is op (1), so the conclusion follows. Let’s say a distribution function F has a good median if F has a continuous density F = f with f (m) > 0 at m, the median of F . More precisely, f (m) > 0 and f continuous at m imply that F is strictly increasing in a neighborhood of m, so m is the unique x with F (x) = 1/2 and so the unique median. Let’s find the asymptotic distribution of the sample median. First let n = 2k + 1 odd, so the nth sample median mn = X(k+1) . If F is the U [0, 1] distribution, let its order statistics be U(1) < · · · < U(n) . Recall that U(j) has a beta distribution βj,n−j+1 for each j, so the sample median U(k+1) has a βk+1,k+1 distribution. Its density is xk (1 − x)k /B(k + 1, k + 1) for 0 ≤ x ≤ 1 and 0 elsewhere. The distribution has mean 1/2 and variance 1/[4(2k + 3)] = 1/[4(n + 2)]. This beta distribution is asymptotically normal with its mean and variance as n → ∞ or equivalently k → ∞. This fact is a special case of facts known since about 1920, but lacking a handy reference, I’ll indicate a proof. Let y = x − (1/2), so |y| ≤ 1/2 where the density is non-zero. On that interval,
k k x (1 − x) = � 1 +y 2 �k � 1 −y 2 �k � = 1 − y2 4 �k = 4−k (1 − 4y 2 )k . We have√(1 − 4y 2 )k ≤ exp(−4ky 2 ) for all y with |y| ≤ 1/2, and for any constant c and |y| ≤ c/ k, k log(1−4y 2 )+4ky 2 = O(k(4y 2 )2 ) = O(1/k) = O(1/n) as n → ∞ and k → ∞, 1 so for such y (depending on k), (1 − 4y 2 )k is asymptotic to exp(−4ky 2 ). It follows that βk+1,k+1 is asymptotically√normal with mean 1/2 and variance 1/(8k) which is asymptotic to 1/(4n). In other words n[U(k+1) − 12 ] converges in distribution as n → ∞ to N (0, 1/4). Now for any distribution function F with a good median m, and n = 2k + 1 odd, the sample median mn = X(k+1) has the distribution of F ← (U(k+1) ) because F ← is monotonic (non-decreasing, and strictly increasing in a neighborhood of 12 ). We have √ ← F (1/2) = m. √So by the delta-method theorem above, n(mn − m), being equal in distribution to n(F ← (U(k+1) ) − F ← (1/2)), converges in distribution as n → ∞ to N (0, (F ← )
(1/2)2 /4) = N (0, 1/(4f (m)2)), as stated in Randles and Wolfe, p. 227, line 2, for symmetric distributions. For n = 2k even, U(k) and U(k+1) have βk,k+1 and βk+1,k distributions respectively, and |U(k+1) − U(k) | = Op (1/n). For the sample median mU,n = [U(k) + U(k+1) ]/2, we then also have |mU,n − U(k) | = Op (1/n). By a small adaptation of the argument for the n odd √ case, we get that n(U(k) − 12 ) converges in distribution to N (0, 1/4) as n = 2k → ∞, and √ 1 so does n(m √U,n − 2 ). So, for a distribution F with a good median m and sample medians mn , we get n(mn − m) converging in distribution as n → ∞ to N (0, 1/(4f (m)2)), just as when n is odd and as stated by Randles and Wolfe. Next, let’s consider the Hodges-Lehmann estimator. In this case, beside assuming F has a good median m, we’ll assume the distribution is symmetric around m. (If a distribution is symmetric around a point θ, then θ must be the median.) In other words, there is a density f0 with f0 (−x) = f0 (x) for all x, f0 (0) > 0, f0 is continuous at 0, and the density f is fm (x) ≡ f0 (x − m), which is then symmetric around m. Given X1 , ..., Xn i.i.d. with a distribution F satisfying the given conditions, but otherwise unknown, the HodgesLehmann estimator θ̂HL is the median of the numbers (Xi +Xj )/2 for 1 ≤ i ≤ j ≤ n. There are n(n + 1)/2 of these numbers (which are called Walsh averages). The sample median is an estimator of the unknown m, and θ̂HL is another which is often better. To look into it we’ll consider some U -statistics. For any real x, x1 , and x2 let hx (x1 , x2 ) = Ψ(2x−x1 −x2 ). This kernel is symmetric under interchanging x1 and x2 for each x. We want to find the asymptotic behavior of θ̂HL − m, specifically, that it’s asymptotically normal with mean 0 and variance C/n for some C depending on F . In doing this, we can assume m = 0, because subtracting m from all the observations makes m = 0 and doesn’t change the distribution of the difference. So we can assume F is symmetric around 0. Let G be the distribution function � ∞ of X1 + X2 . Then G has a density g given by the convolution of f with itself, g(x) = −∞ f (x − y)f (y)dy. We have for all x Ehx (X1 , X2 ) = P (X1 + X2 < 2x) = G(2x). The quantity called ζ1 , entering into the asymptotic variance of the U -statistic formed from the kernel hx , is given by ζ1 = P (X1 + X2 < 2x, X1 + X3 < 2x) − G(2x)2 . We are interested especially in x = 0 since that is now the median and center of symmetry of F and of G. For x = 0 we get � ∞ F (−u)2 dF (u) = P (X1 + X2 < 0, X1 + X3 < 0) = −∞ 2 � ∞ −∞ � 2 [1 − F (u)] dF (u) = 0 1 (1 − t)2 dt = 1/3, and Eh0 = 1/2, so ζ1 = 1/12. We have a kernel of order r = 2, and the asymptotic variance of a U -statistic is r2 ζ1 . Defining a U -statistic depending on x we have (n) U(x) � �−1 n = 2 � Ψ(x − Xi − Xj ). 1≤i