A field of applied statistics called survival analysis has been developed over several decades to deal with censored data, which occur in astronomical surveys when objects are too faint to be detected. How these methods can assist in the statistical interpretation of astronomical data are reviewed

Feigelson, Eric D.

Isobe, Takashi

English

NASA Technical Reports Server

i fij —
SURVIVAL ANALYSIS, OR WHAT TO DO WITH UPPER LIMITS
IN ASTRONOMICAL SURVEYS
TAKASHI ISOBE and ERIC D . FEIGELSON
Department of Astronomy
Pennsylvania State University
University Park, PA 16802
Abstract; A field of applied statistics called 'survival analysis' has been
developed over several decades to deal with 'censored data1, which occur in astro-
nomical surveys when objects are too faint to be detected. We review here briefly,
and elsewhere in more detail, how these methods can assist in the statistical inter- ^
pretation of astronomical data. . a? w o
<N <s r-
I H ^ o
CO O ;*
oo c? i—
Astronomers often need to analyze samples of objects for which incomplete in-
formation is available. For example, a study of the X-ray properties of optically j ^
selected quasars might find more than half of the X-ray observations are upper »
limits. In statistics, these are called left censored data points. Some , n
researchers have omitted these censored data from consideration. Others divided ' u
them into detected and undetected subsamples and compared them by two sample tests. <H <
These methods can introduce bias to the results and do not make efficient use of the 33 o
data. The presence of upper limits make it difficult to measure even simple 9 Q,^
statistical quantities, such as the mean X-ray luminosity of the sample. o^ tn
Similar cases occur in many fields of research Ce.g. biomedical research, U U
actuarial science, industrial reliability testing, econometrics) and many statis- ^^ ^
ticians have studied these problems. The methods they have developed comprise a M O ^
field generally known as 'Survival Analysis*. Although it may seem that observa- PH o-H
tions in astronomy are very different from data in these fields, we find that the ^ S S
mathematical problems are the same. The field of survival analysis is now quite , w <*»
extensive, and is'described in several recent books (e.g. Miller 1981, Lawless 1982). +»
We have summarized some of the methods that might be particularly useful to < H +>
astronomers in two recent papers- CFeigelson and Nelson 1985, Isobe et al. 1986; •2 «• w
Papers I and II) . . P» «n <o
The method used for several centuries by actuaries to determine the distribu- § S o
tion function of a censored data set is in fact identical to the "fractional "> H «
luminosity function" frequently used by radio astronomers Ce.g. Auriemma et al. <-\
1977). Its limitations are that accuracy is lost due to binning, error analysis is ^o^ at
not simple (YN errors are not correct, .the actuarial "Greenwood's formula" must be ' JJJ JJJ 2
used), and few useful statistical tests are available. The situation changed <N => «
dramatically in the 1950-70's. With the advent of Kaplan and Meier's unbinned 22 3. £i
maximum-likelihood estimator for the distribution function, the discovery of several ^ jj*
non-parametric 2-sample tests, and the development of several tests for correlation y <& to
and linear regression. Some of this progress has been paralleled in astronomy with «i o w
the independent discovery of th£ Kaplan-Meier estimator (Avn± et al. 1980; Hummel Jg Q ^
1981; Pfleiderer and Krommidas 1982) and of the EM-algorithm linear regression (Avni jf.9^
and Tananbaum 1986). Avni's procedures are now widely used in X-ray astronomy. We
believe that the power and reliability of astronomers* interpretation of censored
data will be greatly improved if we have the full range of survival analysis
techniques available to us. :
https://ntrs.nasa.gov/search.jsp?R=19880015167 2020-03-20T07:14:20+00:00Z
We can briefly review the principal elements of survival analysis discussed in
Papers I and II. The Kaplan-Meier estimator finds the maximum-likelihood distribu-
tion of a censored data set. For example, we can find the X-ray luminosity function
of optically selected quasars, its mean value and its error. Assuming there are no
ties among the data z , the Kaplan-Meier estimator is given by
1
 «,
S(zt) = z nz (l-l/n..) J where zfz^
S(z.) = 1 where zi<z1»
where 6^=1 if zj is detected, 6^=0 if z* is undetected, and n^ = number of data
points in R(z^ ). R(z^ ) is called the risk set, which consists of all data points
which have not been detected before z^. The Kaplan-Meier function has simple
analytic formulae for error analysis, and gives .the mean and standard deviation of
the distribution.
If we need to test the hypothesis that two or more populations have the same
distributions (e.g., the X-ray luminosity functions of optically selected and radio
selected quasars), several procedures are available (Paper I). Gehan's extension of
the Wilcoxon test and the logrank test are most commonly used in biomedical applica-
tions. The former is more effective than the latter when the underlying distribu-
tion function is normal. The logrank test is more effective at finding differences
in the two samples at the censored end of the distribution, while the Gehan test is
more sensitive at the uncensored end.
Now if we want to see whether a correlation exists between two variables
(e.g., the optical and X-ray luminosities), Cox regression and a generalization of
Kendall's T correlation coefficient are very useful (Paper II). The former one is
very popular in many fields of study if only the dependent variable contains cen-
sored data. The latter method is not yet widely used, but it permits any type of
censoring. If a correlation is present, linear regression can be performed. The
EM (estimate and maximize) algorithm with a normal distribution (Wolynetz 1979) or
with a Kaplan-Meier distribution (Buckley and James 1980) can treat this problem.
Schmitt (1985) has developed a linear regression method for data censored in both
variables.
The main advantages of survival analysis over methods traditionally used by
astronomers are its wide variety of statistical tools and their mathematical
robustness well-established by professional statisticians. The principal disadvan-
tages are its complexity and inconvenience compared to previous methods. We have
used a combination of programs in commercial statistical software packages (BMDP
for the Kaplan-Meier estimator, 2-sample tests and Cox regression), published
FORTRAN codes (the Kaplan-Meier and 2-sample tests in Lee 1976, and the EM algorithm
linear regression in Wolynetz 1979), and codes written by ourselves (generalized
Kendall's T and Schmitt's linear regression for dual censored data). We are glad to
share our experience and codes with others and, if sufficient interest is present,
may produce a more coherent package for astronomical use. Please feel free to
contact us.
While astronomers can substantially benefit from the survival analysis methods
already developed, some additional statistical work needs to be done. An obvious
deficiency is that existing methods assume the censored value is known exactly,
while most astronomical upper limits are estimated from the absence of a signal in
(assumed) Gaussian noise. A weighting scheme based- on the noise value, rather than
an artificial limit such as "3-o", should be developed. Study is also needed on the
statistics of truncated rather than censored data. Truncated data sets in astronomy
are those where objects fainter than a given flux level are missing entirely from
the sample. Statisticians have only recently realize (Woodruffe 1985) that the
"C-method" derived by Lynden-Bell (1971) to overcome selection effects in quasar
surveys is the analog of the Kaplan-Meier estimator for truncated data. We are
beginning to study statistics for truncated data in some detail.
This work was supported in part by NASA grant NAG 8-555.
References
Auriemma, C. et al., Astr. Ap. 57, 41 (1977).
Avni, Y., et al., Ap.J. 238, 800 (1980).
Avni, Y. and Tananbaum, H., Ap.J. 305, 83 (1986),
Buckley, J., and James, I., Bioraetrika 66, 429 C1979).
Feigelson, E.D. and Nelson, P.I., Ap.J. 293, 192 (1985). [Paper I]
Hummel, E., Astr. Ap. 93, 93 Q-981).
Isobe, T., Feigelson, E.D. and Nelson, P.I., Ap.J. 306, 490 (1986). [Paper II]
Lawless, J.F. Statistical Models and Methods for Lifetime Data, New York:Wiley
(1982).
Lee, E.T. Statistical Methods for Survival Data Analysis, Belmont CArLifetime
Learning Pub. (1980).
Lynden-Bell, D., Mon. Not. R. Astr. Soc. 155, 95 (1971).
Miller, R.G., Jr., Survival Analysis, New York:Wiley (1981).
Pfleiderer, J. and Krommidas, P., Mon. Not. R. Astr. Soc. 198, 281 (1982).
Schmitt, J.H., Ap.J. 293, 178 (1985).
Wolynetz, M.S. Appl. Statistics 28, 185 (1979).
Woodroofe, M., Ann. of Statistics 13, 163 (1985),
Authors' address
Eric D. Feigelson and Takashi Isobe
Department of Astronomy
525 Davey Laboratory
The Pennsylvania State University
University Park, PA 16802
U.S.A.
Submitted to Bull. Inform. CDS.


Survival analysis, or what to do with upper limits in astronomical surveys

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server