To better inform sense and avoid research needs and to understand ongoing investigation of potential solutions that ultimately lead to the assisting the FAA with their Congressional mandate to fly UAS in the NAS

Consiglio, Maria Cristina

Randall, Debra K.

Santiago, Confesor

NASA Technical Reports Server

National Aeronautics and Space Administration 
www.nasa.gov 
28-29 October, 2014 
SARP Well Clear Industry Day 
Laurel, MD 
 
Unmanned Aircraft Systems (UAS) Integration in 
the National Airspace System (NAS) Project 
NASA Contributions to the SARP WC Definition 
Debra Randall 
UAS UAS Chief Systems Engineer 
 
Maria Consiglio 
Confesor Santiago 
UAS NAS Sense and Avoid Co-Project Engineers 
 
 
https://ntrs.nasa.gov/search.jsp?R=20140017297 2019-08-31T15:04:57+00:00Z
NASA Controller Acceptability 
 Study and ACES Simulation 
2 
  
Final Agreed Operational Acceptability Metrics 
Acceptability Metric 
NASA Controller 
Acceptability 
Study 
MIT/LL USAF Open/Closed 
Loop Simulation 
NASA ACES 
Monte Carlo Simulation 
TCAS II RA Rate    
Controller 
Acceptability 
Considerations 

  

  
Well Clear Volume 
Penetration Rate 
  
  
Crosstrack Deviation     
Vertical Deviation     
Maneuver Initiation 
Point 
  
  
CPA Miss 
Distance/Time Given 
Well Clear Violation 
  

  

Mitigated Risk Ratio        
Benefits of Controller Acceptability Study (CAS1) 
• Provides a direct assessment of Well Clear Boundary (WCB) 
operational acceptability by air traffic controllers (i.e., operators) 
o CAS1 simulates lower altitude, moderate-complexity Class E airspace operations 
o These operations are challenging for WCB definitions and DAA: more well-clear 
encounters, more traffic flow constraints 
o Controllers for this airspace are arguably most accustomed to well clear encounters 
and accommodating them along with other separation services
 
• Provides insight into both minimum and maximum acceptable WCB 
sizes 
o Air Traffic Controllers (ATC) require a WCB that works within the existing operations 
of the National Airspace System (NAS) 
o A WCB that is too small will cause safety concerns for ATC, and potentially distract 
attention from other encounters in the sector, but 
o A WCB that is too big will disrupt traffic flow, and increase ATC workload due to 
secondary conflicts and cross-sector coordination 
 
• Provides an additional look at TCAS II Corrective RA Rates between 
WCB definitions, using the CAS1 encounter set 
3 
Benefits of ACES Simulations 
• Airspace Concept Evaluation System (ACES) Simulations 
o Simulate UAS traffic from various proposed missions with “UAS-like” aircraft 
models 
o Collect NAS-wide encounter data within various classes of airspace 
o Simulate 100s of thousands of flight hours in a matter of days 
o Evaluate well-clear violation rates against historical VFR radar data 
(RADES) 
 
 
4 
CAS1 Research Approach  
• Construct a set of simulated “well clear encounters”  
- Different miss distances, encounter geometries, relative speeds  
 
• Embed these encounters into simulated background traffic 
scenarios representative of moderate-workload TRACON 
traffic (IFR and VFR) on a calm, clear-weather day 
 
• Ask a series of ATC volunteers to “control” the simulated 
traffic scenarios, and measure the results  
- Direct query after each encounter  
- Workload and performance measures throughout  
 
• Perform statistical analysis of all data afterward to assess 
the range(s) of acceptable horizontal miss distances 
- 14 controllers, 84 simulation hours, 1176 queries 
 5 
CAS1 Traffic Scenario Area 
6 
Scenarios focused on ATC sector 
handling arrivals to Collin County 
Regional (McKinney – TKI), ~28 
nmi NE of DFW 
 
Feature-rich airspace: 
•Class B, 4000-11000’ above TKI 
•VFR & IFR aircraft all under positive 
control, all cooperative 
•Class D, SFC-2900’ around TKI 
•VFR & IFR cooperative aircraft 
receiving Class D ATC services 
Class E, 700’ or 1200’ AGL up to 
FL180 and outside Class B and D 
•IFR aircraft 
•VFR aircraft, some receiving ATC 
services, some not 
•CAS1 Well Clear encounters 
occurred in this airspace 
•Class G, SFC to overlying airspace 
•Nearby non-towered airports 
Ratings for Opposite Direction encounters  
Mean of 14 ATC subjects for each encounter 
The plot above shows Mean 
Ratings for opposite direction  
encounters. 
Plot of frequency of Rating 
responses for opposite direction 
encounters. 
7 
0
1
2
3
4
5
0 0.5 1 1.5 2 2.5 3 3.5
R
a
t
i
n
g
 
Horizontal Miss Distance (nmi) 
Ratings by HMD (Opposite Direction) 
1
2
3
4
5
0
5
10
15
20
25
0.5 1 1.5 2 2.5 3
R
a
t
i
n
g
 
1
-
5
 
b
y
 
0
.
5
 
F
r
e
q
u
e
n
c
y
 
Horizontal Miss Distance (nmi) 
Ratings by HMD (Opposite Direction) 
Note: All Horizontal Miss Distances required a UAS lateral maneuver (initially a collision course) 
 
Rating Scale: After each “well clear encounter” test subjects 
were asked to rate the horizontal miss distance (HMD) on a five-
point scale:  From 1=Too Close to 5-Too large 
8 
0.0
10.0
20.0
30.0
40.0
50.0
60.0
70.0
80.0
0 0.5 1 1.5 2 2.5 3 3.5
P
e
r
c
e
n
t
a
g
e
 
Horizontal Miss Distance (nmi) 
Rating % by HMD for < 3 Rating (OD) 
0.0
10.0
20.0
30.0
40.0
50.0
60.0
70.0
80.0
90.0
100.0
0 0.5 1 1.5 2 2.5 3 3.5
P
e
r
c
e
n
t
a
g
e
 
Horizontal Miss Distance (nmi) 
Rating % by HMD for Rating of 3 (OD) 
0.0
10.0
20.0
30.0
40.0
50.0
60.0
70.0
80.0
90.0
100.0
0 0.5 1 1.5 2 2.5 3 3.5
P
e
r
c
e
n
t
a
g
e
 
Horizontal Miss Distance (nmi) 
Rating % by HMD for > 3 Rating (OD) 
OD = Opposite Direction encounters 
Ratings for Opposite Direction encounters  
Rating % by HMD for <3, 3, and >3 Ratings 
“Too Close” 
“Too Wide” 
“Okay” 
•Horizontal Miss Distance (HMD) 
of ~1.5 nmi appears optimal for 
ATC acceptability 
•HMD range of 1-2 nmi looks 
good  
CAS1 Approach to ATC Acceptability  
Assessment for WCB Candidates 
• For each CAS1 “well clear encounter” and each WCB 
candidate: 
 
– Assess HMD value 
 
– Use encounter geometry and HMD to determine CAS1 ATC 
measured acceptability rating (interpolate as necessary) 
9 
CAS1 ATC Acceptability Assessment for WCB 
Candidates: Encounters Exactly at WCB 
• Average HMD ratings: 
10 
Model HMD Rating HMD Rating HMD Rating 
USAF 0.87 2.9 * 0.72 2.6 * 1.1 – 1.8 2.8 – 3.2 * 
MIT LL 0.67 2.3 * 0.67 2.5 * 0.67 2.3 * 
NASA 1.0 3.0 1.0 2.9 1.0 2.8 
Opposite Direction Overtake Crossing 
* Interpolated 
CAS1 ATC Acceptability Assessment for WCB 
Candidates: Encounters Exactly at WCB 
• Percent < 3 (“too close”) rating: 
11 
Model HMD < 3 HMD < 3 HMD < 3 
USAF 0.87 30% * 0.72 40% * 1.1 – 1.8 22% – 5% * 
MIT LL 0.67 55% * 0.67 46% * 0.67 55% * 
NASA 1.0 19% 1.0 18% 1.0 25% 
Opposite Direction Overtake Crossing 
• Percent > 3 (“too wide”) rating: 
Model HMD > 3 HMD > 3 HMD > 3 
USAF 0.87 7% * 0.72 2% * 1.1 – 1.8 2% – 34% * 
MIT LL 0.67 4% * 0.67 1% * 0.67 1% * 
NASA 1.0 11% 1.0 4%  1.0 2% 
Opposite Direction Overtake Crossing 
* Interpolated 
* Interpolated 
CAS1 Approach to ATC Acceptability of  
Vertical Deviation 
• Controller assessment of acceptable vertical 
deviations was performed during debrief sessions 
 
o All preferred 500’ or less for level VFR-IFR encounters 
 
o Opinions were negative when asked to assess effects of 
WCB “taller” than 500’, e.g.: 
 “Problematic” 
 “Would be a significant factor in congested airspace, if you’re 
working other aircraft it [<=500’] gives you another altitude” 
 “I don’t think it would be pretty” 
 “I would expect 500’ to be sufficient for a manned aircraft, it 
should be sufficient for an unmanned aircraft” 
 “Would be pretty disruptive” 
 
 
12 
CAS1 Approach to TCAS RA Rate Evaluation  
• For each CAS1 “well clear encounter” and each WCB 
candidate: 
 
– Assess minimum HMD values 
 
– Use encounter geometry (including the respective WCB candidate’s 
HMD) as input to simple CAS1 TCAS model  
 
– Determine whether a corrective RA would have occurred for each 
well clear encounter geometry and WCB candidate  
13 
Tau and HMD Threshold Values for TCAS 
14 
• TCAS Tau and HMD threshold values for RA issuance are 
altitude-dependent (lower values at lower altitudes): 
• Self Separation HMD distances should be no smaller than TCAS 
HMD values but may need to be larger for controller acceptability 
Own Altitude (ft) Tau (sec) HMD (nmi) HMD + 50% 
< 1000 AGL N/A N/A N/A 
1000-2350 AGL 15 0.20 0.30 
2350-5000 20 0.35 0.53 
5000-10000 25 0.55 0.83 
10000-20000 30 0.80 1.2 
20000-42000 35 1.1 1.7 
>42000 35 1.1 1.7 
CAS1 TCAS RA Rate Evaluation Results 
• For the 42 CAS1 encounters as simulated, TCAS model, and 
WCB candidate HMD values, one TCAS RA each occurred for 
the USAF and LL WCB candidates 
o HMD values are generally sufficient for all WCB candidates 
o TCAS sensitivity level and aircraft speeds are also low 
 
• Results if all encounters flown between 5000-10000’ (e.g., 6000’): 
o USAF: 2 RAs 
o LL: 36 RAs (all but the overtake encounters) 
o NASA: 0 Ras
 
• Results if all encounters flown above 10000’: 
o USAF: 12 RAs 
o LL: 36 RAs (all but the overtake encounters) 
o NASA: 36 RAs (all but the overtake encounters) 
 
• HMD > 1.2 nmi avoids RAs below Class A with this TCAS model 
 
15 
4-DOF Trajectory Model 
 Aerodynamic models of aircraft  
 Models replicate pilot behavior 
 User-definable uncertainty characteristics 
Modeling and Simulation: ACES 
NAS-wide Simulation 
• Gate-to-gate simulation 
of ATM operations   
• Full flight schedule with 
flight plans 
• Sector and center 
models with some 
airspace procedures 
Simulation Agents 
• Air traffic controller decision making 
• Traffic flow management models 
• Individual aircraft characteristics 
• IFR Flight Tracks from ASDI data 
• VFR Flight Tracks from 84th 
Squadron RADES data 
16 
National Traffic Management Regional Traffic Management 
Local Approach 
and Departure 
Traffic 
Management 
Airport and Surface 
Traffic Management 
ACES Well-Clear Simulation and Analysis  
ACES 
UAS 
Missions 
Eight Days* 
of  VFR 
Traffic 
Autoresolver 
NAS 
Distributed System 
USAF WC 
NASA WC 
 MIT LL WC 
Database: 
- State Data 
- Intruders 
- Resolutions 
- WCVs 
Matlab Post 
Processing 
PPT Results 
• Webinar Meetings 
• F2F Workshops 
* Over 25k UAS flight hours simulated in the NAS per day 
UAS Missions Characteristics 
18 
UAS group Duration 
(per flight) 
Flights per 
day 
Cruise Alt.  Flight Pattern 
Air Quality Monitoring  Shadow-B  1-4 hrs. 104-1044 4k,5k, and 6k  
ft AGL 
Radiator Grid Pattern 
Cargo Transport 
 
Cessna 208  
 
varies 1.4k 2k-16k Point to Point 
Atmospheric Sampling Global Hawk 1.5-13 hrs. 2352 5k-35k ft 
AGL 
Radiator Grid Pattern 
On-demand Remote Air 
Taxi -Cirrus 
 
Cirrus 
SR22T 
varies 8k 6k-11k Point to Point 
On-demand Remote Air 
Taxi - Mustang 
 
Cessna 
Mustang 
varies 2k-4k 9k-20k Point to Point 
 
Strategic  Fire 
Monitoring 
 
Predator-B 20 hrs. 74-324 31k ft MSL Radiator Grid Pattern 
Tactical Fire Monitoring 
 
Shadow-B 1-1.5 hrs. varies varies Circular Loitering Orbit 
Flood Inundation 
Mapping 
 
Aerosonde 1-4 hrs. varies 4k ft AGL Radiator Grid Pattern 
Point to Point 
Flow Stream 
Monitoring  
 
Aerosonde 1-4 hrs. 20-200 4k  Radiator Grid Pattern 
Point to Point 
Cooperative VFR Traffic 
• The 84th Rader Evaluation Squadron (RADES) data were used. 
o The data contain the radar hits collected from hundreds of radar sites in U.S, and 
each hit provide timestamp, latitude, longitude, Mode 3 code, Mode C code, and 
others. 
o There is no explicit information that could be used to determine whether radar hits 
come from IFR flights or VFR flights. 
 
 
• Criteria for filtering out cooperative VFR traffic (for each tracked flight): 
o All tracks are below 18,000 ft, 
o At least one track has the Mode 3 code of 1200, 
o Average speed ranges from 50 knots to 250 knots. 
 
19 
Method for Extracting VFR Traffic 
• Input: RADES data for a certain time period. 
 
• Output: Flight plan file for a fast-time simulation system, Advanced Concept 
Evaluation System (ACES). 
 
• Method (three steps): 
1. Generate tracks using a minimum spanning tree based clustering algorithm, 
2. Generate smooth tracks using a Kalman filter, 
3. Generate a flight plan file after reducing the number of waypoints and adding 
airports which are closest to start/end waypoints. 
 
 
This becomes the basis of the traffic the UAS’s encounter for measuring 
well-clear violation rates. 
 
20 
Cooperative VFR Traffic Used for SARP 
21 
No. of 
Coop. VFR 
Flights 
Self-Separation Algorithm (Autoresolver) 
• If a WCV is predicted to occur within a parametric time, e.g. 60 seconds, 
Autoresolver is engaged to command a resolution that solves the problem 
 
• Autoresolver iterates through different horizontal and altitude maneuvers options 
that avoid WCV 
o During this process it tries to ensure an extra safety margin by putting buffers around 
horizontal miss distance and time threshold 
o If ownship is level, prefers the minimum left and right turn that avoids WCV over climb 
or descent 
o If ownship is transitioning vertically, prefers the closest temporary altitude hold that 
avoids WCV 
 
 
22 
Altitude Horizontal 
TCAS-II RA Metric 
Probability of Well-Clear Violation (WCV) with TCAS RA prior to WCV 
 
• Assumption: Intruders (manned) experiencing TCAS-RA’s while UAS DAA 
system detects it as well-clear is undesirable.  (Unmitigated) 
• The smaller the better 
 
 
 
 
• While detecting and resolving WCVs (mitigated), at what rate do we trigger a 
TCAS-RA?   (Mitigated) 
 
  
 
 
23 
Number of WCVs with TCAS-RA prior to WCV 
Total Number of WCVs 
# of TCAS-RA’s 
Total UAS Flight Hour 
Probability of WCV with TCAS RA prior to WCV 
24 
Probability 
Undesirable 
Desirable 
TCAS-RA Rate 
• MIT-LL:   8.47 x 10-4  RAs / flt-hour (fewest) 
 
• NASA:  3.9 x 10-3  RAs / flt-hour  (middle) 
 
• USAF:  1.52 x 10-2  RAs / flt-hour (most) 
25 
Well Clear Violation Penetration Rate 
• When analyzing well clear violation rates between the three different models, it 
wasn’t clear if higher or lower rates were better. 
 
• As a group, SARP weighted this metric of lower importance. 
 
• All three models had about the same WCV rate. 
 
• However, what was found to be interesting is verifying there were no unusual 
trends in WCV rate for the three WC models by different VFR traffic days or 
UAS mission profiles. 
26 
WCV Rate by Day 
27 
Well-Clear 
Violation 
Rate per 
1 x 102 flight 
hours 
WCV Rate reveals similar trends within each day for each WC definition. 
WCV Rate by UAS Mission Type 
28 
Well-Clear 
Violation 
Rate per 
1 x 102 flight 
hours 
*Aggregate 
at ~3 WCV 
per 100 flt-
hours 
Other Metrics 
• We also collected the following metrics using ACES to support SARP Well-Clear 
Workshop: 
 
o Maneuver Initiation Point – points to surveillance performance requirements for 
detecting WCVs 
 
o CPA Miss Distance for WCVs – evaluates severity of WCVs 
 
o Minimum Time from WCV to NMAC – identifies worse case scenarios where little time 
is available from violating well-clear to having an NMAC 
29 
Summary of CAS1 Results 
• Controller Acceptability Considerations 
o NASA WCB rated highest, followed by USAF WCB, followed by LL WCB 
o LL HMD of 4000’ (0.67 nmi) elicits a higher “too close” rating from 
controllers, but actual HMD values would likely be higher in practice due 
to horizontal buffers added for DAA sensor uncertainty 
o LL WCB “height” of 700’ rated low by controllers but subsequent 
FAA/RTCA modification of WCB to 450’ fixes this controller acceptability 
concern 
 
• TCAS II RA Rate 
o For the CAS1 encounter geometries and TCAS model, the NASA model 
had the fewest RAs below 10000’ and the USAF model had the fewest 
above 10000’ 
o Small HMD values cause higher RA rates, especially at higher altitudes 
and high closure rates 
o HMD values > 1.2 nmi minimizes RA issuance below Class A airspace 
with this TCAS model 
 30 
Summary of ACES Simulations 
• Collected encounter data with characteristics that resemble envisioned UAS 
missions interacting with historically cooperative VFR traffic in different 
classes of airspace with different “UAS-like” aircraft models 
• Results complimented the analyses provided by USAF’s stressing cases and 
MIT LL’s Monte Carlo simulations 
• Each well-clear model had areas of high and low performance with respect 
to the SARP-accepted metrics 
• Overall, over 200,000 UAS flight hours of envisioned missions were 
simulated NAS-wide and used to help SARP determine a well-clear definition 
recommendation 
 
 


Unmanned Aircraft Systems (UAS) Integration in the National Airspace System (NAS) Project. NASA Contributions to the SARP WC Definition

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server