Automated Multi-Aircraft Control System scenario generation for Human-in-the-Loop evaluations of air traffic management concepts is described. Methods for analyzing and comparing the seed-scenario generated using the automated process and the Human-in-the-Loop-scenario designed to meet the experiment objectives are discussed. The main findings are: (1) many of the characteristics of the seed-scenario used for constructing the Human-in-the-Loop-scenario are preserved in the Human-in-the-Loop Scenario, (2) landing characteristics of the traffic generated by the Multi-Aircraft Control System using the input scenario compare reasonably well with that intended in the input scenario, and (3) many of desired characteristics of the Human-in-the-Loop-scenario can be achieved by further automation

Chatterji, Gano

Nguyen, Jimmy

Palopo, Kee

Zheng, Yun

NASA Technical Reports Server

1Automated Scenario Generation for 
Human-in-the-Loop Simulations
Gano Chatterji and Yun Zheng
Crown Consulting, Inc.
Kee Palopo
NASA Ames Research Center
Jimmy Nguyen
Optimal Synthesis, Inc.
https://ntrs.nasa.gov/search.jsp?R=20180005212 2019-08-31T14:42:35+00:00Z
2Motivation
• Need for realistic scenarios to study diverse operations
– Unmanned Aerial Vehicles 
– Urban Air Mobility
– Supersonic aircraft
• Manual creation of realistic scenarios for generating 
traffic for Human-in-the-Loop simulation is difficult  
– Missing and erroneous data
– Repeated creation of scenario and testing in simulation is time 
consuming
– Difficulties cause studies to be limited to few scenarios
• Automated scenario generation has potential for 
overcoming limitations 
– Use real air traffic data to create scenario
– Remove flights with erroneous data
– Mix data from different days to achieve desired traffic volume
3Research Questions
• Can initial traffic scenario be generated using an automated process 
that runs in Multi-Aircraft Control System (MACS)?
• Can this initial traffic scenario be used as a starting point for building a 
Human-in-the-Loop (HITL) scenario?
• How does one compare the initial traffic scenario created using the 
automated process with the manually altered HITL-scenario?
• Can an automated process be used to directly create a HITL-scenario?
4Outline
• Air Traffic Management (ATM) Testbed
• Automated scenario generation using ATM Testbed
• Approach
• Results
• Conclusions
5Air Traffic Management Testbed
ATM Services
Conflict 
Detection
Trajectory 
Generator
Conflict
Resolution
Scheduler
Component N
Component A
Data Distribution Service
Test Bed
Support Services
Including 
Housekeeping
Traffic 
Generators
GovCloud
ATM SimulationReal Aircraft
Test Bed Visualization 
Services
Stakeholder
Tools
High-fidelity Simulator
6Air Traffic Management Testbed
ATM Services
Conflict 
Detection
Trajectory 
Generator
Conflict
Resolution
Scheduler
Component N
Component A
Data Distribution Service
Test Bed
Support Services
Including 
Housekeeping
Traffic 
Generators
GovCloud
ATM SimulationReal Aircraft
Test Bed Visualization 
Services
Stakeholder
Tools
High-fidelity Simulator
Scenario 
Generation
7Creation of Seed-scenario
Automated Scenario Generation
Seed-scenario
Input Data for Scenario Generation
8Input Data for Scenario Generation
• Reduced Record (RD)
– Single record for each flight
• Beacon-code, flight-plan, takeoff/landing runway, departure/arrival time, 
sector/center transition list
• Event Data (EV)
– Multiple records related to events for each flight
• Event time and type- landing, crossing
• Example: center crossing from ZOA to ZLA
• Integrated Flight Format (IFF)
– Multiple records for each flight
• All flight plans including amended flight plans
• Position data
• EV and RD useful for filtering and IFF for data augmentation
9Simulation Architect View for Composing 
MACS Traffic Scenario
Data Loader
Airline Filter:
AAL, DAL
Landing Runway
Filter: 22L
Airport Arrival
Filter: KEWR
Airport Arrival
Filter: KLGA
Landing Time
Filter: start-time, 
end-time
EV Filter
MACS Traffic
Scenario Builder
Other inputs
RD Filters
10
MACS Scenario Generation Steps
Load & filter input data 
11
MACS Scenario Generation Steps
Preprocess flight data
Load & filter input data 
12
MACS Scenario Generation Steps
Preprocess flight data
Load & filter input data 
Flights remaining?
Yes
13
MACS Scenario Generation Steps
Preprocess flight data
Load & filter input data 
Flights remaining?
Find entry track data
Yes
14
MACS Scenario Generation Steps
Preprocess flight data
Load & filter input data 
Flights remaining?
Find entry track data
Find last flight-plan prior 
to entry track time
Yes
15
MACS Scenario Generation Steps
Preprocess flight data
Load & filter input data 
Flights remaining?
Find entry track data
Find last flight-plan prior 
to entry track time
Process flight-plan data
Yes
16
MACS Scenario Generation Steps
Preprocess flight data
Load & filter input data 
Flights remaining?
Find entry track data
Find last flight-plan prior 
to entry track time
Process flight-plan data
Compute entry state data
Yes
17
MACS Scenario Generation Steps
Preprocess flight data
Load & filter input data 
Flights remaining?
Find entry track data
Find last flight-plan prior 
to entry track time
Process flight-plan data
Compute entry state dataUpdate comment fields
Yes
18
MACS Scenario Generation Steps
Preprocess flight data
Load & filter input data 
Flights remaining?
Find entry track data
Find last flight-plan prior 
to entry track time
Process flight-plan data
Compute entry state dataUpdate comment fields
Assign values to data fields
Yes
19
MACS Scenario Generation Steps
Preprocess flight data
Load & filter input data 
Flights remaining?
Find entry track data
Find last flight-plan prior 
to entry track time
Process flight-plan data
Compute entry state dataUpdate comment fields
Assign values to data fields
Yes
20
MACS Scenario Generation Steps
Preprocess flight data
Load & filter input data 
Flights remaining?
Find entry track data
Find last flight-plan prior 
to entry track time
Process flight-plan data
Compute entry state dataUpdate comment fields
Assign values to data fields
Yes
No
Output scenario data
21
MACS Scenario Generation Steps
Preprocess flight data
Load & filter input data 
Flights remaining?
Find entry track data
Find last flight-plan prior 
to entry track time
Process flight-plan data
Compute entry state dataUpdate comment fields
Assign values to data fields
Yes
No
Stop
Output scenario data
22
Approach: Seed-scenario versus HITL-scenario
Automated Scenario Generation
Seed-scenario
Input Data for Scenario Generation
23
Approach: Seed-scenario versus HITL-scenario
Automated Scenario Generation
Manual Refinement
Seed-scenario
Input Data for Scenario Generation
24
Approach: Seed-scenario versus HITL-scenario
Automated Scenario Generation
Manual Refinement
Seed-scenario
HITL-scenario
Input Data for Scenario Generation
25
Traffic Scenarios
• Seed-scenario generated using automated process
– June 6, 2016 RD, EV and IFF files 
– Arrivals to Newark, New Jersey 
– Six-hours traffic starting at 17:00 UTC
– 299 flights; 274 landing on 22L, six on 22R, one on 29 and 18 not 
assigned
• Manually altered HITL-scenario
– Entry times altered to squeeze six-hours of traffic to five-hours to 
exceed arrival capacity of 40 aircraft/hour
– Flights within 40 nautical miles surrounding airport removed
– Some flights at the beginning of scenario removed
– Flights removed to maintain ratio of internal (400 nautical miles) to 
total number of flights
– 191 flights, all landing on 22L
26
Data Analysis I: Number of Flights with 
Same Parameter Value
# Parameter
1. Call-sign
2. Aircraft-type
3. Destination airport
4. Landing runway
5. MACS flight-plan
6. ATC flight-plan
7. Beacon-code
8. Departure airports
9. Entry point altitude
10. Entry point airspeed
11. Entry point sector-ID
12. Aircraft weight
Number of
flights
Aircraft-type
27
Seed-scenario Results
# Parameter Once Repeated Total
1. Call-sign 281 9 290
2. Aircraft-type 11 24 35
3. Destination airport 0 1 1
4. Landing runway 1 3 4
5. MACS flight-plan 148 47 195
6. ATC flight-plan 174 41 215
7. Beacon-code 256 21 277
8. Departure airports 50 68 118
9. Entry point altitude 73 61 134
10. Entry point airspeed 77 58 135
11. Entry point sector-ID 47 23 70
12. Aircraft weight 5 24 29
28
HITL-scenario Results
# Parameter Once Repeated Total
1. Call-sign 191 0 191
2. Aircraft-type 10 20 30
3. Destination airport 0 1 1
4. Landing runway 0 1 1
5. MACS flight-plan 64 41 105
6. ATC flight-plan 80 40 120
7. Beacon-code 181 5 186
8. Departure airports 41 50 91
9. Entry point altitude 35 46 81
10. Entry point airspeed 23 14 37
11. Entry point sector-ID 0 3 3
12. Aircraft weight 3 16 19
29
Seed-scenario versus HITL-scenario
Seed-scenario HITL-scenario
# Parameter Once/
Total (%)
Repeated/
Total (%)
Once/
Total (%)
Repeated/
Total (%)
1. Call-sign 97 3 100 0
2. Aircraft-type 31 69 33 67
3. Destination airport 0 100 0 100
4. Landing runway 25 75 0 100
5. MACS flight-plan 76 24 61 39
6. ATC flight-plan 81 19 67 33
7. Beacon-code 92 8 97 3
8. Departure airports 42 58 45 55
9. Entry point altitude 54 46 43 57
10. Entry point airspeed 57 43 62 38
11. Entry point sector-ID 67 33 0 100
12. Aircraft weight 17 83 16 84
30
Data Analysis II: Distribution
# Parameter
1. Route length
2. Cruise speed
3. Cruise altitude
4. Actual landing time
5. Predicted landing time
6. Aircraft weight
7. Entry time
8. Entry point airspeed
9. Entry point altitude
Number of
flights
Entry point airspeed (knots)
31
Data Analysis III: Seed-scenario Landing Rate
Expected landing time
Entry time
Route length
V Average cruise speed
R
L E
CR
L
E
R
CR
lt t
V
t
t
l
= +
-
-
-
-
Landing rate
(flights/hour)
Landing time (hours past 17 UTC)
32
Seed-scenario v/s HITL-scenario 
Landing Rate Results
274 flights 191 flights
Landing time (hours past 17 UTC)
Landing rate
(flights/hour)
Landing time (hours past 17 UTC)
Landing rate
(flights/hour)
33
Approach: HITL-scenario versus MACS Output
Automated Scenario Generation
Manual Refinement
Seed-scenario
HITL-scenario
Input Data for Scenario Generation
34
Approach: HITL-scenario versus MACS Output
Automated Scenario Generation
Manual Refinement
Seed-scenario
HITL-scenario
MACS Simulation
Input Data for Scenario Generation
35
Approach: HITL-scenario versus MACS Output
MACS Simulation Output
Automated Scenario Generation
Manual Refinement
Seed-scenario
HITL-scenario
MACS Simulation
Input Data for Scenario Generation
36
HITL-scenario v/s MACS Output 
Landing Rate Results
Landing rate
(flights/hour)
Landing time (hours past 17 UTC)
• Expected landing rate graph 
is sensitive to cruise speed
• Faster shifts left
• Slower shifts right
• Errors due to 
• 18 aircraft did not land in 
MACS
• Aircraft performance models 
in MACS 
• Conversion of Mach to 
cruise speed using standard 
atmosphere
37
Conclusions
• MACS simulations can be run with seed-scenario created 
using the ATM Testbed 
• Seed-scenario was found to be a good starting point for 
creating HITL-scenario
• Duplicate flight and distribution analysis useful for data 
quality and eliminating flights with unreasonable parameter 
values
• Analysis showed that many of the manual adjustments can 
also be included in the automated process to directly 
create the HITL-scenario
38
Extra Slides
39
Future Work
Enhance Scenario Generation to create Human-in-the-Loop 
scenarios
• Eliminate flights with erroneous parameter values
• Select flights to achieve the internal to total ratio
• Alter the landing times to achieve the desired landing rate


Automated Scenario Generation for Human-in-the-Loop Simulations

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server