We want to minimize a function f (usually a negative- log-likelihood or related function) over a parameter region which we believe contains at least a sub-region over which the function is locally convex. In large-sample settings, we expect a very sharp peak near which the function behaves like a quadric surface. The calculus-based theory leads to several important remarks for statistical problems. • Search for parameters with ∇ f(ϑ) = 0; • Newton-Raphson (NR) gives one-step solution in case f is quadratic; • Newton-Raphson converges quadratically, i.e. with distances from the local maximizer squaring at each iteration, if we start close enough; • step-lengths for gradient ascent are essentially arbitrary but may have to be made artifically small in order to avoid overflows and numerical instabilities; • NR steps may also be wild and numerically unstable away from the immediate neighborhood of a local max. • at a not-too-large computational cost, it makes sense to avoid unstable steps by searching along the ray provided by either the gradient or the NR increment to ensure that the function-value decreases at each iteration (reduction of multivariate to univariate search). The last suggestion, together with the requirement to approximate gradients and Hessians via finite-difference schemes, is characteristic of Quasi-Newton methods. References for all of these topics: Numerical Recipes, plus general books on optimization like Luenberger, Optimization by Vector Space Methods, or general numerical-analysis books like the text of Stoer & Bulirsch often used in MAPL 466 or 666. 52 6.1 Coding & Splus Functions Related to Newton-Raphson The multivariate Newton-Raphson (NR) method of solving an equation g(x) = 0, where g is a smooth (k-vector-valued) function of a k-dimensional vector variable x whose Jacobian matrix Jg(x)

Year: 2012

OAI identifier:
oai:CiteSeerX.psu:10.1.1.210.5610

Provided by:
CiteSeerX

Download PDF:To submit an update or takedown request for this paper, please submit an Update/Correction/Removal Request.