Hinkley, David

University of Minnesota Digital Conservancy

-, 
• 
4 
LIKELIHOOD AS APPROXIMATE PIVOTAL DISTRIBUTION 
By 
David Hinkley 
University of Minnesota 
Technical Report No. 360 
January 1980 
Research supported by NSF Grant MCS-7904558. 
LIKELIHOOD AS APPROXIMATE PIVOTAL DISTRIBUTION 
By D.V. Hinkley 
University of Minnesota 
SUMMARY 
For a single-parameter curved exponential family, it is shown that the con-
ditional distribution of the maximum likelihood estimator is the normalized like-
lihood function, except for a relative error proportional to inverse sample size. 
Key Words: Ancillary statistic; Information; Edgeworth expansion; Maximum 
likelihood estimate; Sufficient statistic; Exponential family. 
L INTRODUCTION 
Recently it has been shown (Efron & Hinkley, 1978; Peers, 1978) that there 
is a conditional normal approximation for the distribution of the maximum likeli-
hood estimator in the one-parameter case. This conditional normal approximation 
uses an observed characteristic of the likelihood as a variance approximation. 
In principle the approximation is unnecessary for the special case of estimating 
a location parameter, because Fisher (1934) derived the exact, fully-informative 
conditional distribution of the maximum likelihood estimator. That distribution 
is the normalized likelihood function. In this paper we present theoretical 
evidence that Fisher's special case result is a good approximation in general. 
Specifically, let lik(8) denote the likelihood for a sample of size n from 
a curved exponential family distribution indexed by a single parameter 0 and let 
8 denote the maximum likelihood estimator. Then we find that the probability 
density of the pivot 8-8, conditional on a natural approximate ancillary a, is 
2 
lik(8) { -1} 
f lik(u)du l+O(n ) ' 
provided that 8 is chosen to make the Fisher information constant. The proof 
is based on an Edgeworth expansion for the joint distribution of the sufficient 
statistic, which is developed in Section 2. The final step to the conditional 
"' distribution of 0 is outlined in Section 3, where we find it essential to choose 
8 so that the expected Fisher information is constant. Section 4 contains 
some comments on the result, its implications and its possible generalization. 
2. JOINT DISTRIBUTION OF MAXIMUM LIKELIHOOD ESTIMATE AND ANCILLARIES 
We suppose that x1 , ... ,~ are independent, p-dimensional identically dis-
tributed vectors each with probability density 
e e n, 
where n e R. Thus f 6(x) is in a one-dimensional curved subset of the p-dimen-
sional canonical exponential family. The minimal sufficient statistic is 
S = I:Xj, whose probability density we denote by g6 (s). Our first aim is to find 
an expansion for g8 (s) in terms of the log likelihood t 6 (s) = I: log f 0(xj) and 
its derivatives at the maximum. 
Before proceeding we introduce the following definitions and notations. Let 
(1) 
and 
A = C' ' 'eC·P>>, e /\a 'l\e ' • • • ' I\ 
where dots refer to differentiation with respect to 0. Simple calculations 
show that 
3 
(2) 
(3) 
and so on, where 
is the expected Fisher information. We assume that the maximum likelihood estimate 
8 is a solution of t 0 = O, and we define the observed Fisher information I= -t0. 
Finally let 
(4) 
For stylistic convenience we often drop the subscript 0, and use a circumflex 
to denote evaluation at 0; for example, V = Va and V = V9. 
The first step in expanding the joint density g0(s) is to note that suffi-
ciency implies 
(5) 
-for any Sen. A saddle-point expansion of g8(s) corresponds to the optimal 
choice 0 = 0(s) used in the Edgeworth expansion 
Intel-½ T -1 
= ½ [exp {-½(s-n89) (nl:e) (s-n8e) } ](l+r ) , 
(27T) P n 
(6) 
where 
r = _!_ (HTp')[J](.s') + O(n-1), (7) 
n 6/ii 
as described in detail by Barndorff-Nielsen & Cox (1979). The term (HTp')[J](s') 
in (7) involves Hermite polynomials of degree up to 3 with arguments' which is 
an orthogonalised, standardized version of s. An important property is that 
(HTp')[3l(s') is invariant under affine transformation of s. 
4 
-With the choice 8 = S(s) in (6), an initial simplification is obtained by 
,... "T " 
s = A (s-nB). (8) 
Note that ;j is the deviation of i~·j) from E8{i~·j)} evaluated at 8 = 8, and 
that in particular s1 = 0. 
accordingly as 
"T " We shall writes = (O,u), and partition Vin (4) 
where nv11 = J. Then writing 
(9) 
and using (8), expression (6) yields 
lnEI-½ "T " l" g8(s) =½ exp {-½u (nW)- u} {l+r (s')}. (2n) p n (10) 
The next step is to rewrite (10) in terms of approximately ancillary statistics, 
which are transformations of 18, ... ,ig·p) To do this we choose the unique lower 
triangular matrix B = Ba such that 
-BWB T = identity (11) 
and define 
T 1 "" 
a = (a2 , ••• ,ap) = -- Bu. In 
(12) 
The latter statistic is asymptotically ancillary, having a normal limiting 
distribution with zero mean and identity variance matrix. This can be proved 
by first applying a centr~l limit theorem to the linear Taylor expansion of 
n-½~, whose limiting variance matrix is 
5 
cf. Lemma 2 of Efron & Hinkley (.1978) for the case p = 2. The particular choice 
.. ( .. ) 
of a in (12), in which aj depends only on 1a,···,1e J , corresponds to a rele-
vance ordering with a2 the dominant ancillary. If Ye denotes the statistical 
curvature per observation (Efron, 1975), one can see from (12) that 
(13) 
which was denoted by q in §5 of Efron & Hinkley (1978). 
Returning to expression (10), we now substitute from (12) to obtain 
lnEI-½ T A g6(s) =½p {exp{-½a a)}{l + r (a)}, (21T) n (14) 
where in r (a) we have used the affine invariance property of the Hermite poly-
n 
nomials to replaces' by a. The constant coefficients p' in (7) are evaluated 
at 0, hence the circumflex on r. 
n 
In order to find an expansion for the joint distribution of (0,a) it 
A 
remains only to derive the Jacobian J = ja(0,a)/asj. This can be obtained by 
a geometric argument, or by direct calculus, and is 
-1 0 A I 
J = IAI (15) 
0 
1 AT 
-B 
In 
see Appendix for details. Using (15) together with (11), (9), and (4) in (14) 
leads to 
A 89 (0 ,a) -½p I T = (21T) JA½ {exp(-½a a)}{l+rn(a)}. 
0 
This can be simplified, by substitution for I from (13), to give 
6 
= (.2-rr) -½p J ~{ exp (-½a Ta)}{ 1+;~ (a)}, (16) 
where 
"' 1 +..l. 1 T"' [3] "' 1 r*(a) = {6 (H p) (a)+ya2} + O(n-) n In 
1 + ..l. "' + O(n-1), = n1 (a) In 
(17) 
say. Finally, substituting (16) in (5), we have the general density expansion 
(18) 
where~ is the standard normal density. 
Of course by carrying further terms in the Edgeworth expansion, as in 
Barndorff-Nielsen & Cox (1979), one can obtain a more detailed expansion of the 
final term in (18). However, such a finer development would be of no value in 
the approximate analysis of the next section. 
3. CONDITIONAL DISTRIBUTION OF THE MAXIMUM LIKELIHOOD ESTIMATE 
The main result we seek involves the expansion for the conditional density 
"' h8(ela) obtained from (18). In Fisher's (1934) analysis for the location model, 
where xj = 8 + ej with known distribution for the ej, the ancillary a is the 
"' 
ordered set of residuals xj - 8. Fisher showed that, because t 8(s) = t 8(6,a) = 
t 8_8(0,a), 
• 
7 
"' 
exp{t8(8,a)} exp{t8(s)} =~--------=~--_;_ ___ _ I expU.0 (t ,a) }dt / exp{it (s) }dt • (19) 
This is the normalized likelihood function, a function only of the pivot 8-0 
and ancillary a. We wish to show that (19) is correct, to within a O(n-1) 
relative error, for the curved exponential family model. 
It turns out to be not only convenient but also crucial to choose 8 so 
that J 0 = n. We now assume this choice, and attempt to duplicate Fisher's result 
for the pivot 8-0. The first step is to obtain the expansion 
"' ,... 
,... exp{10(8,a)-1a(0,a)} 
h (a I a) - ----------.----e - I exp{t8(t,a)-1t(t,a)}dt (20) 
from (18). This follows by noting that ~1 (a) = n1 (a) + O(n-½) for 9-8 = O(n-½), 
and then formally computing the integral of (18). The term n1(a) cancels in 
the resulting expression for h0 (ela), leaving the O(n-l) term in (20). 
Notice that in Fisher's problem t 0(0,a) is ancillary and therefore has can-
celled out in (19). But this is not possible in our problem. Nor is the last 
step in (19) VGlid for us. Instead we must resort to approximations using the 
relationships 
"' I= n +/nya2 = n +/nya2 + Op(l) 
(j ~ 3), (21) 
where the first relationship depends on the fact that J0 = n. Now for 8-t = 
O(n-½) and 8-8 = O(n-½), (21) implies that the Taylor expansion for the exponent 
in the integral in C2O) may be written 
8 
2 Iii n 3 -1 i(t,a)-i (t,a) = -½(t-0) (n + ny a2) - 6 (t-0) µ +O(n ) t t t 
½ 2 /'ii- n 3 -1 = - (t-0) (n + ny a) - - (t-8) µ +o(n ) a 2 6 a 
2 In n 3 -1 
= -½(t-8) (n + ny§a2) - 6 (t-8) µa+O(n ) 
" " 
= i8-t+0 (8,a) - i0(8,a) + O(n-1), (22) 
where nµ 8 = E0(t8). But the integral in (20) may be truncated at t-8 = O(n-½) 
with relative error O(n-1), so that we may substitute (22) in th~ integrand and 
obtain 
" he(ela) 
the analog of Fisher's result (19), valid for 0-8 = O(n-½). 
" Expressed in pivotal form, the result for T = 8-8 is 
" lik (8-t) 
ha(tla) = f li~ , .. ,..:a .. {l+o(n-1)}. 
s 
(23) 
(24) 
In principle the normalized likelihood is, then, a better approximation than 
Efron & Hinkley's (1978) normal approximation, since the latter involves a 
relative error of order n-½. The result (24) also more closely resembles the 
" 
Bayes posterior distribution obtained for the pivot 8-0 when using the Jeffreys' 
prior density proportional to J~½. 
4. COMMENTS 
The result linking normalized likelihood to the conditional distribution of 
8 is very similar to a result obtained by Cox (1980) for the 11 confidence dis-
. 
tribution" based on the score statistic t 8; Cox's second-order ancillary seems 
to be an expansion of our a2. 
; 
I 
9 
A referee has rightly questioned the value of our theoretical result, 
particularly in view of the restriction on parameterization. To take the 
theoretical implications first, it would appear that likelihood per~ has a 
stronger interpretation than was previously believed, and tpat the oft-advocated 
information-stabilizing choice of 0 does indeed correspond to a (local) loca-
tion parameter. I believe that (23) can be used to show that the log likelihood 
ratio statistic 2(19-18) is equal to Xi+ Op(n-
1) conditional on a,·generalizing 
Lemma 3 of Efron & Hinkley (1978). For the particular class of problems con-
sidered, aTaas defined by (12) is the usual chi-squared goodness of fit statis-
tic, whose distribution can be expanded in asymptotic series by use of (18). 
The practical use of (23) requires choosing 8 so that J 8 is constant. 
Section 6 of Efron & Hinkley (1978) gives details of one example where this is 
relatively easy. Cox's (1980) Poisson process example is also straightforward. 
But the example discussed by Hinkley (1979) appears intractable. 
One suggestion to be drawn from the restriction J8 = n is that the result 
(23), and Efron & Hinkley's (1978) conditional normal approximation, will not 
hold for the multiparameter case in general. To shed a little more light on 
this point, consider the following example due to George Cobb. Let Yi= 02zi + E1 , 
where the si are independent N(O,1) and the Zi are independent N(61 ,1). The 
minimal sufficient statistic is (!Zi,!Z~,EZiYi), and A= EZi (EZi) 2/n is 
ancillary with respect to 8T = (81 ,02). The expected Fisher information matrix 
~ 2 is J6 = diag(n,n+n81), and the observed Fisher information matrix is I= 
A2 A 
diag(n,a+n81). Evidently the conditional distribution of e2 given a is not 
approximately N(02,r
22 ), which would generalize Efron & Hinkley's (1978) result, 
A2 
but this is the conditional distribution given a+ ne 1. Note, however, that 
10 
" T " 2 (8-8) I(S-8) c 2(t8-t8) = x2 given a. (25) 
One might speculate that (25) holds approximately in general, for an appropriate 
approximate "simultaneous ancillary" a, so that a simple approximate conditional 
theory exists for simultaneous estimation. 
ACKNOWLEDGEMENTS 
I thank Patricia Grambsch for discussions relating to Section 4. The 
research described here was supported by a grant from the National Science 
Foundation. 
.-
' 
11 
APPENDIX 
To evaluate the Jacobian given in (15), we begin with (8) and (13) which 
combine to give 
T if it is understood that the zero first element of A9(s-nB8) is deleted; the 
A A 
statistic 0 = 0(s) satisfies the likelihood equation 
A 
Conside~ a small perturbation os of s, with corresponding perturbations 06 and 
oa. Then first expanding the likelihood equation we find that, retaining 
first-order terms, 
.T. 
Therefore, because ~ = nA8B0 , and because of (3), 
which implies 
A 
06/as 
A similar linear expansion for a+oa leads to 
say, wherein A0 has its first column A§ removed. Combining the expressions 
for ae/as and aa/as we have 
" lo(8,a)/osl 
12 
-1. 1 T . 
= 1 r A0, - A0B0 + A0n01 In 
-1· 1 T 
= II A0, - A0B81 In 
T . -1 1 T 
= I Aa diag (.I , - Be) I , 
In 
where the last expression A8 now includes its first column A§·· 
This result can be achieved simply by decomposing os into a perturbation 
•T 
0S1 in the plane {s: A9(s-nBa) = O} and a perturbation os2 parallel to the 
curve {nS8}, and noting that oa = _!_ A0B§(os1), 08 = ii(os2), where Ag has its In 
first column removed. 
• 
, 
.; 
0 
13 
REFERENCES 
Bamdorff-Neilsen, O. and Cox, D.R. (1979). Edgeworth and saddle-point 
approxi~tions with statistical applications. J. Roy. Statist. Soc. 
]!, 41 (to appear). 
Cox, D.R. (1980). Local ancillarity. Biometrika, ~ (to appear). 
Efron, B. (1975). Defining the curvature of a statistical problem (with 
applications to second order efficiency) (with Discussion). Ann. Statist. 
l, 1189-242. 
Efron, B. and Hinkley, D.V. (1978). Assessing the accuracy of the maximum 
likelihood estimator: Observed versus expected Fisher information. 
Biometrika, 65, 457-82. 
Fisher, R.A. (1934). Two new properties of mathematical likelihood. · 
Proc. R. Soc.!!:_, 144, 285-307. 
Hinkley, D.V. (1979). Likelihood inference for a simple growth model. 
Biometrika, 66 (to appear). 
Peers, H.W. (1978). Second-order sufficiency and statistical invariants. 
Biometrika, 65, 489-96. 


Likelihood as Approximate Pivotal Distribution

https://conservancy.umn.edu/bitstream/handle/11299/199363/Technical%20Report%20360%20Likelihood%20as%20Approximate%20Pivotal%20Distrobution%20David%20Hinkley.pdf?sequence=1&isAllowed=y

Likelihood as Approximate Pivotal Distribution

Abstract

Similar works

Full text

Available Versions

University of Minnesota Digital Conservancy