This presentation applies a covariance realism technique to the National Aeronautics and Space Administration (NASA) Earth Observation System (EOS) Aqua and Aura spacecraft based on inferential statistics. The technique consists of three parts: collection calculation of definitive state estimates through orbit determination, calculation of covariance realism test statistics at each covariance propagation point, and proper assessment of those test statistics

Tracewell, David

Zaidi, Waqar H.

NASA Technical Reports Server

Mission Operations Working Group
1
International Earth Science Constellation Mission Operations Working Group
April 13-15, 2016
Earth Observation System Flight Dynamics System 
Covariance Realism
Waqar H. Zaidi, EOS FDS Analysis Lead/Code 595
David Tracewell, EOS Flight Dynamics Engineer/Code 595
esmo-eos-fds@lists.nasa.gov
https://ntrs.nasa.gov/search.jsp?R=20160005170 2019-08-31T03:21:03+00:00Z
Mission Operations Working Group
Motivation
• At its User’s Forum on 14 Apr 2015, CARA recommended its users to 
begin delivering realistic covariances.
• This presentation is a response to that recommendation. 
• Aqua and Aura’s covariances have been tuned during times without 
maneuvers. 
• The impact on the probability of collision (on select conjunctions) using 
a tuned covariance was examined. 
• A method to tune covariances through maneuvers is being adopted and 
will be ready for presentation by the next MOWG.  
3/1/2016 2
Mission Operations Working Group
Introduction
• Covariance Realism: 
– Study the evolution of a set of covariances (propagated into the future beginning with a pre-
determined definitive state estimate error) by examining its behavior at equally spaced 
propagation points. 
– Uses inferential statistics in which behavioral conclusions for a large population are drawn using 
sample data. 
– The Mahalanobis distance of a covariance at a particular propagation point represents the ratio 
of the predicted minus definitive position difference to the covariance’s prediction. 
– A group of the squares of such calculations should conform to a chi-squared distribution with 3 
Degrees-of-Freedom (DoF)
• Involves the following 3 phases:
– Collection/calculation of definitive state estimates through orbit determination. 
– Calculation of covariance realism test statistics at each propagation point. 
– Proper assessment of those test statistics using a hypothesized distribution. 
3/1/2016 3
Mission Operations Working Group
Purpose
• Pc sensitive to Scaling of Primary Covariance:
– Graph below was presented at the 14 Apr 2015 CARA User’s Forum. 
– Depicts Pc differences between nominal value and recalculation with primary covariance 
rescaled (Scale Factors 0.5 to 2).
• ~2–5% of cases show Scaled Pc is greater than the Nominal Pc. 
– Impacts operational conclusions.
• A realistic covariance is important. 
3/1/2016 4
Mission Operations Working Group
Definition of Terms
Definitive State Estimate: 
– Best known position and velocity at an epoch time; obtained by passing observations through a 
Filter or Batch estimator. 
Definitive State Estimate Error: 
– Uncertainty in the definitive state estimate produced by a Filter or Batch estimator. 
– Contained in a Definitive Covariance Matrix. 
Predicted State Estimate: 
– Position and velocity are propagated to a time t using a state transition matrix and definitive state 
estimate at an epoch time t0. 
Predicted State Estimate Error: 
– Uncertainty in the Predicted State Estimate propagated using a force model. 
– Contained in a Predictive Covariance Matrix. 
Epoch Covariance: 
– State Estimate Error at a specific epoch. 
Predicted – Definitive State Estimate: 
– The difference between the predicted state estimate (propagated from epoch time t0) and the 
definitive state estimate (obtained through orbit determination) at any time t.
3/1/2016 5
Mission Operations Working Group
Gaussian versus 3 DOF Chi-Square Distribution
3/1/2016
34.1%34.1%
13.6% 13.6%
2.1% 2.1%
41%
22%
20%
17 %
• A Normal or Gaussian 
Distribution has a Mean of 
0 and Standard Deviation 
of 1:
• 50% of values are 
distributed above and 
below a Mean of 0
• A Chi-Square Distribution
is a multi-variate 
distribution of the sum of 
the squares of k
independent standard 
normal random variables.
• A k degree-of-freedom 
(DOF) Chi-Square 
Distribution has a mean 
value of k. 
• A Chi-Square Distribution 
with 3 DOF has a Mean of 
3 and a Standard 
Deviation of 8/3: 
• 61% of values are 
distributed below a 
Mean of 3
61%
50%
C
u
m
u
la
ti
v
e 
D
is
tr
ib
u
ti
o
n
 F
u
n
ct
io
n
P
ro
b
ab
il
it
y
 D
en
si
ty
 F
u
n
ct
io
n
Normal Distribution 3 DOF Chi-Square Distribution
6
Mission Operations Working Group
Chi-Square Statistic
• The Chi-Square statistic is computed using the vector of predicted –
definitive state estimates, ε, and the predicted state estimate error or 
covariance matrix, C:
• A perfectly sized covariance should have a Chi-Square equal to 3.
• In fact, this first moment of distribution test provides an initial idea of a 
covariance's departure from reality.  
• However, a more rigorous Empirical Cumulative Distribution Function 
(ECDF) Method is adopted for this covariance realism analysis.
3/1/2016
222
12
3 

















 
C
C
I
I
R
RT
dof C





















definitivepredicted
definitivepredicted
definitivepredicted
CC
II
RR

Inverse of the 
covariance matrix 
obtained directly 
from propagation, 
thus containing 
correlation terms
Difference between 
predicted and definitive 
state estimates
Predicted State 
Estimate Error
assuming no 
correlation between 
variables
7
Mission Operations Working Group
Empirical Cumulative Distribution Function 
Method
Quadratic Statistics1:
– An ECDF method that evaluates how 
well an empirical distribution 
corresponds to a parent distribution by 
examining the summation of a function 
of the squares of the deviations between 
the empirical and parent distributions. 
– Examples are the Cramér – von Mises, 
Watson, and Anderson-Darling statistics. 
– This analysis uses the more permissive 
Cramér – von Mises statistic due to the 
likelihood of outliers.  
P-value and Confidence Interval:
– P-value: The likelihood an empirical 
distribution is drawn from a parent 
distribution. 
– Confidence Interval: A p-value threshold 
that indicates a “pass” or “fail”. Normally 
2% or higher are accepted. 
3/1/2016
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 2 4 6 8 10C
u
m
u
la
ti
v
e
 D
is
tr
ib
u
ti
o
n
 F
u
n
c
ti
o
n
Chi-Square, εC-1εT
Deviation between 
empirical and 
parent or 
hypothetical 
distribution
Parent 
Distribution
 


 dxxxFxFnQ n )()()(
2

Empirical 
CDF 
Hypothetical 
CDF 
Empirical 
Distribution
Weighting 
Function
EDF 
Statistic
A p-value can be obtained for each Q-Statistic 
using a published table of p-values; one that is 
generated using Monte Carlo simulations.  
**Note in this example the Parent Distribution is a 3 DOF Chi-
Square Distribution
8
Mission Operations Working Group
Chi-Square Statistic Grouping
3/1/2016 9
• Collect bins of Chi-Square Statistics at each propagation point and 
examine their Chi-Square distribution . 
• The number of Chi-Square Statistics in each bin should be equal to the 
number of total propagations. 
Each red rectangle 
or “bin” here 
contains information 
of a set of 
covariances at a 
unique propagation 
point. 
Mission Operations Working Group
Process Noise Effect and Implementation
• State Noise Compensation (SNC) - process 
noise is added to the propagation of the 
definitive state estimate in order to account 
for uncertainty in the force model.
• The predicted state estimate error, 𝑃(𝑡), is 
propagated using linear mapping as 
follows:
• The process noise matrix Q is built using 
variances in acceleration as follows:
3/1/2016
𝑃 𝑡 = 𝜑𝑃 𝑡0 𝜑
𝑇 + 𝛤𝑄𝛤𝑇
𝜑, 𝛤 = state transition matrices
𝑃 𝑡0 = definitive state estimate error
Predicted –
Definitive State 
Estimates
Predicted 
State 
Estimate 
Error without 
Process 
Noise
Predicted State 
Estimate Error with
Process Noise



















accacc
accacc
RIC
qITqI
T
qI
T
qI
T
Q
2
3
34
2
23











C
I
R
accq

















100
010
001
I
∆𝑇 = propagation step size
𝜎𝑅,𝐼,𝐶= acceleration variances
Predicted State 
Estimate Error 
without Process 
Noise
Predicted –
Definitive State 
Estimates
Predicted State 
Estimate Error with
Process Noise
10
Mission Operations Working Group
Aqua Covariance Realism Case Study
• The following assumptions are made in the study:
– Propagation Date Span: 2 Aug 2014 to 6 Nov 2014. 
– Propagation Time Span: 3 Days. 
– Maneuvers occur on 27 Aug, 17 Sep, 8 Oct, and 21 Oct 2014 – propagation over 
these dates are avoided. 
– Process noise is kept constant for all propagations.
• The study is conducted as follows:
– Select an arbitrary set of acceleration variances and propagate all definitive state 
estimates using the corresponding process noise. 
– Examine the deviation between the ECDF and CDF of the 3 DOF Chi-Square 
Distribution without outlier identification. 
– Perform an outlier identification test and eliminate propagations that contain 
outliers.
– Resize (by adjusting the process noise) the predicted state estimate error using 
the total mean RMS error of all remaining propagations (after outlier identification). 
– Examine the deviation between the ECDF and CDF of the 3 DOF Chi-Square 
Distribution (post outlier identification and resized predicted state estimate error).
3/1/2016 11
Mission Operations Working Group
RMS Error and Uncertainty 
(Without Outlier Identification)
3/1/2016
• Propagation Time Span – 2 Aug 2014 to 
6 Nov 2014
• Maneuver Dates – 27 Aug 2014 
17 Sep 2014 
08 Oct 2014 
21 Oct 2014
Mean RMS
Predicted State 
Estimate Error
10/22/14
10/25/14
9/13/14
10/28/15
9/28/14
12
Mission Operations Working Group
Goodness-Of-Fit Results 
(Without Outlier Identification)
3/1/2016
Ideal or 
Hypothesized 
3 DOF Chi-
squared CDF
A 0.26 p-value 
represents an 
excellent result
• 80 Bins containing Chi-
Square statistics equal to 
the number of 
propagations (29) are 
tested by computing their 
CDF across the 3-day 
propagation time span. 
• A p-value threshold of 
0.02 or 2% is set to 
determine a statistical 
pass. 
• 54 out of 80 Chi-Square 
Bins (63.75%) produce p-
value larger than 0.02. 
• Statistical failures occur 
between 2.2 and 3 days 
in the propagation time 
span. 
• Heavy upper-tail 
distribution implies 
covariance is undersized.
2% p-value threshold
13
Mission Operations Working Group
Rosner Outlier Test
• Identify the following potential outliers based on the Normalized In-Track 
Error at the end (largest disparity in error) of each 3 day propagation:
– 13 Sep 2014 
– 28 Sep 2014 
– 22 Oct 2014 
– 25 Oct 2014 
• Perform the Rosner Outlier Identification test using the preceding 
normalized in-track error values.
• For a 2% significance level, the outlier test results indicate all 4 
propagations are outliers and therefore can be eliminated. 
OutlierPositions = [24 25 14 18];
SigOut = [0.0353 0 0.0775 0];
3/1/2016
Empirical 
Distribution  
conforms to the 
parent distribution 
after outlier 
removal
Parent 
Distribution
Position Start Date (εI-μI)/σI
1 2-Aug-14 0.631
2 5-Aug-14 0.671
3 8-Aug-14 0.250
4 11-Aug-14 0.396
5 14-Aug-14 0.164
6 17-Aug-14 0.276
7 20-Aug-14 0.456
8 23-Aug-14 0.497
9 29-Aug-14 0.637
10 1-Sep-14 0.180
11 4-Sep-14 0.263
12 7-Sep-14 -0.261
13 10-Sep-14 0.335
14 13-Sep-14 -1.159
15 19-Sep-14 0.910
16 22-Sep-14 0.638
17 25-Sep-14 -0.367
18 28-Sep-14 -1.078
19 1-Oct-14 0.425
20 4-Oct-14 0.987
21 10-Oct-14 0.395
22 13-Oct-14 0.723
23 16-Oct-14 -0.308
24 22-Oct-14 -3.220
25 25-Oct-14 -2.889
26 28-Oct-14 -0.781
27 31-Oct-14 0.289
28 3-Nov-14 0.594
20 6-Nov-14 0.347
14
Mission Operations Working Group
0
50
100
150
200
250
7/20/2014 8/9/2014 8/29/2014 9/18/2014 10/8/2014 10/28/2014 11/17/2014
S
o
la
r 
A
c
ti
v
it
y
Date
F10.7 Solar Flux Geomagnetic Index (Ap)
Outlier Rejection Correlation to Solar Activity
3/1/2016
• The Outliers identified by the Rosner test show a direct correlation to solar 
activity on those dates. 
• At this time, it appears FDS Propagation is not equipped to predict 
persistently high solar activity or a dramatic drop in the geomagnetic index. 
10/22/14
10/25/14
9/28/14
9/13/14
15
Mission Operations Working Group
RMS Error and Uncertainty
(With Outlier Identification)
3/1/2016
• Propagation Time Span – 02 Aug 2014 to 
06 Nov 2014
• Outliers – 13 Sep 2014 28 Sep 2014
22 Oct 2014  25 Oct 2014 
• Maneuver Dates – 27 Aug 2014 
17 Sep 2014 
08 Oct 2014 
21 Oct 2014
Mean RMS
Predicted State 
Estimate Error
16
Mission Operations Working Group
Goodness-Of-Fit Results 
(With Outlier Identification)
3/1/2016
Ideal or 
Hypothesized 
3 DOF Chi-
squared CDF
A 0.26 p-value 
represents an 
excellent result
• 80 Bins containing Chi-
Square statistics equal to 
the number of 
propagations (25) are 
tested by computing their 
CDF across the 3-day 
propagation time span. 
• A p-value threshold of 
0.02 or 2% is set to 
determine a statistical 
pass. 
• 70 out of 80 Chi-Square 
Bins (87.50%) produce p-
value larger than 0.02. 
• Statistical failures occur 
between 0.6 and 0.75 
days in the propagation 
time span. 
2% p-value 
threshold
17
Mission Operations Working Group
0.00E+00
1.00E-10
2.00E-10
3.00E-10
4.00E-10
5.00E-10
6.00E-10
7.00E-10
11/22/2013 3/2/2014 6/10/2014 9/18/2014 12/27/2014 4/6/2015 7/15/2015 10/23/2015
In
-T
ra
c
k
 A
c
c
. 
V
a
ri
a
n
c
e
Date
Aqua and Aura Seasonal Covariance Tuning 
Jan 2014 – Oct 2015
3/1/2016
0
50
100
150
200
250
11/22/2013 3/2/2014 6/10/2014 9/18/2014 12/27/2014 4/6/2015 7/15/2015 10/23/2015
S
o
la
r 
A
c
ti
v
it
y
Date
F10.7 Solar Flux Geomagnetic Index (Ap)
Magnitude of the In-Track 
Acceleration Variance required for 
realistic covariances appears to 
follow the seasonal change in solar 
activity
PM – 73.75% 
CH – 77.5%
PM – 75% 
CH – 67.5%
PM – 81.25% 
CH – 82.5%
PM – 93.75% 
CH – 85%
PM – 70% 
CH – 77.5%
PM – 86.25% 
CH – 86.25%
PM – 81.5% 
CH – 70%
Percentage of Chi-Square Statistics above 2%
PM – 92.5% 
CH – 81.25%
CH
PM
18
Mission Operations Working Group
Impact of a Realistic Covariance on the 
Probability of Collision (PC)
• Selected several conjunctions with similar primary and secondary object 
uncertainties (a rare occurrence) 2.5 – 3 days to TCA. 
• Replaced the OCM ASSET covariance with a Tuned O/O covariance.
• Kept miss distance equal to the OCM ASSET solution.
• Examined the changes in uncertainties and their impact on the PC. 
3/1/2016
Time of 
Closest 
Approach 
(UTC)
OCM 
Creation 
Time
(UTC)
OCM 
Age at 
TCA 
(Days)
O/O Cov. 
Start Date
(UTC)
O/O 
Cov. 
Age at 
TCA 
(Days)
Radial
Miss 
(m)
In-Track
Miss 
(m)
Cross-
Track
Miss 
(m)
O/O 
Radial 
Cov 
(m)
O/O 
In-
Track 
Cov 
(m)
O/O 
Cross-
Track 
Cov 
(m)
OCM 
ASSET 
Radial 
Cov 
(m)
OCM 
ASSET 
In-
Track 
Cov 
(m)
OCM 
ASSET
Cross-
Track 
Cov 
(m)
Sec,  
Object 
Radial 
Cov 
(m)
Sec,  
Object 
In-
Track 
Cov 
(m)
Sec,  
Object 
(Cross-
Track) 
Cov 
(m)
Pc w/ 
O/O 
Cov.
Pc w/ 
OCM 
ASSET 
Cov.
03/26/14 
03:45:13
03/23/14 
03:02:57
3.03
03/23/14 
12:00:00
2.66 94.1 2807.6 -1771 5.0 518.2 8.6 13.2 1303.5 3.4 5.3 836 6.8 1.0E-19 1.0E-14
04/07/14 
00:22:40
04/04/14 
03:27:10
2.87
04/04/14 
12:00:00
2.52 -299.3 9766.4 2619.4 4.5 546.7 7.3 8.8 942.3 6.9 10.9 292 16.7 1.3E-71 1.1E-87
12/22/14 
18:32:31
12/20/14 
03:10:30
2.64
12/20/14 
12:00:00
2.27 380.6 -2325.2 -1989.9 4.9 513.6 6.9 17.4 755.4 4.4 13.4 526 7.4 4.9E-67 2.2E-63
03/22/15 
15:12:10
03/20/15 
02:51:29
2.51
03/20/15 
12:00:00
2.13 416.6 -3288.1 976.9 4.9 434 7.9 39.0 647.2 21.0 17.7 380 13.3 5.9E-119 9.0E-42
05/11/15 
14:08:45
05/08/15 
14:28:10
2.99
05/08/15 
12:00:00
3.09 -33.9 984 994.6 4.2 246.4 7.9 10.0 427.4 2.8 8.3 153 6.5 4.7E-12 1.4E-06
05/11/15 
14:08:45
05/08/15 
17:37:35
2.85
05/08/15 
12:00:00
3.09 -29.4 1020.9 1029.4 4.2 246.4 7.9 9.6 394.7 2.8 8.3 153 6.5 6.4E-13 2.4E-07
05/11/15 
14:08:45
05/09/15 
01:00:08
2.55
05/08/15 
12:00:00
3.09 -26.3 995.9 1002.8 4.2 246.4 7.9 9.8 321.6 2.8 7.2 116 6.2 1.1E-13 1.6E-09
19
Mission Operations Working Group
Conclusion & Future Work
Conclusion:
– Aqua and Aura are ready to start delivering tuned covariances. 
– Covariance realism tuning is sensitive to outliers but can be tuned for up to 3 
months at a time.
– Provided the primary and secondary object uncertainties are similar, an impact on 
the Pc is evident – similar uncertainties involve well-tracked secondary objects. 
Future Work:
– Interpolate definitive state estimates and add them to prediction – definitive state 
estimate. 
– Resampling Investigation – Take 1,000 random subsets of a passed Chi-Square 
Bin and determine the p-values of each test. 
– Complete analysis for covariance propagation through maneuvers
– Complete analysis for Terra and GPM. 
– Test tuned results with 7-day propagations.
– Group together outliers during high solar events and determine if they conform to a 
Gaussian distribution – Look at the possibility of increasing process noise during high solar 
events to more accurately model the predicted state estimate error. 
3/1/2016 20
Mission Operations Working Group
References
1. M. D. Hejduk, “Covariance Realism Evaluation Approaches.” Flight Dynamics Support 
Services II Technical Memorandum. 10 Jul 2015. FDSS-II-16-0049.
2. M. Duncan, A. Long, “Realistic Covariance Prediction for the Earth Science Constellation.”
AAS/AIAA Astrodynamics Specialist Conference and Exhibit. Keystone, Colorado. 21-24 
August 2006. AIAA 2006-6293.
3. B. Rosner, "On the Detection of Many Outliers." Technometrics Vol 17 No 2 (May 1975, pp. 
221-227.
4. B. Rosner, "Percentage Points for the RST Many Outlier Procedure." Technometrics Vol 19 
NO 3 (August 1977). pp. 307 – 312.
3/1/2016 21


Earth Observation System Flight Dynamics System Covariance Realism

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server