Contents include the following: Computing Information and Communications Technology (CICT) Systems Analysis. Our modeling approach: a 3-part schematic investment model of technology change, impact assessment and prioritization. A whirlwind tour of our model. Lessons learned

Biegel, Bryan

Coughlan, Joe

Hansson, Othar

Hayes, Jordan

Pell, Barney

Stevens, Ken

NASA Technical Reports Server

Othar Hansson 
The CICT Earth Science 
Systems Analysis Model 
Barney Pell, Joe Coughlan, 
Bryan Biegel, Ken Stevens, 
Othar Hansson, Jordan Hayes 
NASA Ames Research Center 
& Thinkbank, Inc. 
April 2004 
The ESSA Team 
0 
Task leads: 
Barney Pel1 (Lead), Bryan Biegel (Co-lead), 
Joe Coughlan (Science Lead), 
Walt Brooks (Science Co-Lead) 
Subcontractor : 
Othar Hansson &Jordan Hayes, Thinkbank 
ARC team: 
Ken Stevens, Peter Cheeseman, Chris Henze, 
Samson Cheung, et al. 
46 
https://ntrs.nasa.gov/search.jsp?R=20050185582 2019-08-29T20:03:32+00:00Z
Enouqh About Me 
Research collaborations with NASA Ames since 1989 
(heu ristic search, data-mi ni ng , plan ni ng/sc hedu li ng ) , 
PhD (Computer Science), Berkeley. 
Using decision analysis techniques for search control 
decisions in science planning/scheduling systems. 
Thinkbank: 
custom software development, 
software architecture consu Iti ng , 
tech nology d ue-d i ligence for investors. 
Agenda 
CICT Systems Analysis 
Our modeling approach 
- a 3-part schematic investment model of 
technology change, impact assessment and 
prioritization 
A whirlwind tour of our model 
Lessons learned 
Systems Analysis in CICT 
Demonstrate "systematic and thorough investment decision 
process" to HQ, OMB and Congressional Decision Makers 
Increase awareness and substantiate CICT's impact to 
missions. Road map CICT projects to missions and 
measurement systems 
4 teams in FY03: 
- 2 pilot studies (Earth Science [me]; Space Science [Weisbin]): 
- TEAM: map from NASA Strategic Plan to IT capability 
- Systems Analysis Tools (COTS/GOTS) 
explore models for ROI of IT. 
requirement; technology impact assessment 
Earth Science Pilot Studv 
How do we characterize and quantify a 
science process? 
Can we build a model of how CICT 
technology investments impact ROI in a 
NASA science process? 
What modeling approach is suitable for 
making such analyses understandable and 
repeatable? 
48 
Current State 
What have we learned? (FY03) 
Decision analysis modeling techniques can be 
applied to systems analysis of CICT project areas. 
Built model of weather-prediction data pipeline. 
What don’t we know? (FY04) 
How much time/expense needed 
to build a full model 
How such a full model fits into a real 
NASA program context 
(CDS: Collaborative Decision Systems) 
Pilot Studv Focus 
Criteria for science process to study 
- Important to a major customer base, 
- Significantly drives technology investments 
- Generalizes to a class of related processes 
- Amenable to quantitative analysis. 
- Critical Earth Science process with relevance not only to 
- Stretch goals require technology breakthroughs. 
- Strong technology driver for other science problems 
- Starting point: analyses from ESE 
2010 Weather Prediction process 
NASA scientists but to the nation at  large. 
computational technology requirements workshop (4/02) 
49 
Pilot Study Accomplishments 
Identified modeling formalism (influence 
d i a g ra m s) 
- Clear semantics accessible to both ES & CICT experts 
- Tools exist for sensitivity analysis, decision-ma king, 
etc. 
We chose Analytica as our modeling tool. 
study as well. 
- Successfully transferred/applied to Space Science pilot 
Built a model with an understandable, simple 
structure (after much research and many 
iterations). 
Demonstrated the kinds of analyses made 
possible bv the model 
Agenda 
CICT Systems Analysis 
Our modeling approach 
- a 3-part schematic investment model of 
technology change, impact assessment 
and prioritization 
A whirlwind tour of our model 
Lessons learned 
50 
Methodolouv: Decision Model 
Overall 
System Value 
Ql: Which technology investments should I make? 
42: How does each technology investment improve 
overall system/mission value (including cost 
considerations)? Choose investments with highest 
value. 
illina in the Decision Mode I m m  m m  
System 
Performance & 
C Overall 
Model 
System value is a function of a set of metrics (accuracy, 
fidelity, cost, etc.). We can model the priority among 
the metrics independent of the technologies used. 
Technology investments have value in that they improve 
these metrics. 
51 
Filling in the Decision Model 
System System 
Characteristics Performance & 
Overall 
System Value 
Priorities 
Model 
Assessment 
Model 
The metrics can be modeled in terms of abstract system 
characteristics (data volume, algorithm accuracy, 
processing speed, model fidelity, . . .). 
Filling in the Decision Model 
System System 
Characteristics Performance & 
Assessment 
------- 
/
System 
Priorities 
Model 
Overall 
System Value 
Model 
Technology investments, together with some mission- 
specific parameters, influence the system characteristics. 
A technology investment (such as data visualization 
research) has value in that it improves system 
characteristics (such as model fidelity). 
52 
Methodologv: Influence Diaarams 
Overall 
Priorities 
Model 
We‘ve sketched an “influence diagram’’ model of the 
decision. 
Q: What tech. investments maximize expected overall system value? 
Q: Value of model refinement: How sensitive to assumption A? 
Q: Value of information: what if we knew that project P would succeed? 
Q: Value of control: what if we could reduce risk of project P failinq? 
Influence Diaqram Details 
System 
Performance 
Cost Metric 
\ \ -  
’system- 
Assessment 
/ 
System 
Priorities 
Model 
Overall 
System Value 
Model 
Influence diagram tools (such as Analytica) allow you to specify and 
evaluate these models. Diagram structure and decision analysis 
techniques speed specification of required parameters. 
“What-if” and optimization questions reduce to the problem of 
computing functions of conditional prob. distributions: 
“best” technology investment is: 
argmax [E(Overall System Value I Technology Investments)] 
53 
Agenda 
CICT Systems Analysis 
Our modeling approach 
- a 3-part schematic investment model of 
technology change, impact assessment and 
prioritization 
A whirlwind tour of our model 
Lessons learned 
The ESSA Model 
5 System 
Performance & 
Cost Metrics Overall 
Model 
Our set of 5 metrics include: 
development cost, operations cost, accuracy, model fidelity, etc. 
54 
The ESSA Model 
12 System 5 System 
Characteristics Performance & 
Cost Metrics 
Overall 
Model 
Model 
Our 12 System Characteristics include: 
observation density, assimilation efficiency, cpu efficiency, etc. 
The ESSA Model 
Overall 
Model 
Model 
Model 
Our 13 technology investments include: data-miningf launching a new data 
source, targeted observing, etc. 
Each represents a research area, summarizing a range of individual 
research tasks or proposals. 
5 5  
ivina Do he Mode I wn into t m .  
5 System 
Cost Metrics 
I c-r:ics Performance & 
System 
Priorities 
Model 
Model 
System-Assessment Model: the most stable part of the model, 
owned/designed by a customer domain expert who understands the 
behavior of the system/mission being analyzed. 
System-Assessment model computes System Metrics from System 
Characteristics 
System-Assessment Model 
data seledion 
assi rn i I ati o n 
simulation 
characterlstlcs 
forecast skill 
(E3 characteristics 
Here is the model of how system 
Lharacteristics drive sys tem 
sys tem metrics (cost & other 
utility attributes) W e  have tried 
l o  break the  model Into modules 
corresponding t o  the processes in 
the  underlying dataflow Each 
process (tan rounded rects) has 
several outpiits (blue ovals) that 
are in turn inputs t o  subsequent 
processes These intermediate 
values decouple the  models o f the  
indiVidiJal proressps 
Sys tem performance and cost rnetrics 1 
56 
Example Svstem Characteristics 
~ -~ - -  
Assimilation efficiency 
CPU efficiency 
Data efficiency 
Ensemble efficiency 
Model framework 
Observation density 
Postprocessing 
effectiveness 
Simulation efficiency 
-- ~- _ _  
0-1 scale: how much information is retained 
despite approximations in data assimilation? 
>O : percentage speedup in CPUs due to R&D 
investments 
0-1 scale: how much information is present in each 
bit of data selected? 
0-1 scale: how much improvement in forecast skill 
do we get from using ensemble algorithms? 
0-1 scale: how much fidelity is present in our 
models? 
0-1 scale: how many of the available observations 
do we make? 
0-1 scale: how much improvement in forecast skill 
do we get from using post-processing? 
> 0: percentage speedups in simulation due to 
R&D investments 
Instantiating the Model 
1 12System 5 System 
Characteristics Performance & 
Cost Metrics 
uveraii 
Priorities 
Model 
Mod 'el Assessment 
Model 
System-Change Model: owned/designed by a program manager who 
understands the feasibility and impact of different research areas. 
System-Change model computes System Characteristics from the set 
of Technology Investments chosen (and system/mission config 
parameters) 
57 
System-Change Model 
"Impact matrix" quantifies the changes to system 
characteristics that will occur if individual research 
projects succeed. 
research area. 
"Cost matrix" quantifies cost breakdown for each 
Portfolio of research areas determines what 
impacts will be felt. 
(In an extended model, cost and impact could vary 
over ti me. ) 
Svstem-Change: Research Areas 
Data-efficient simulations (same data size) 
Data-efficient simulations (less data) 
choose a more informative set of observations to improve forecast skill at 
the same computational cost 
reduce number of observations (and reduce computational cost) w/o 
reducing forecast skill 
ditto, but also gather more targeted observations based on ensemble 
accuracy estimates (e. g . , the Sen sorWe b concept) 
reduce number of grid points by using regional forecast as boundary 
conditions 
reduce number of ensembles needed to get similar accuracy estimates 
(e.g., through use of particle filter technology) 
Data-mining of model outputs 
increased skill from same model output via data analysis & visualization 
(i ntel I igent data understanding ) 
Targeted Observing 
Adaptive grid methods 
Improvements in ensemble methods 
58 
System-Chanqe: Research Areas 
Modeling tools 
ESMF and other initiatives to make modeling efforts more 
productive 
Auto or Semi-Automatic Parallelization tools, Benchmarking, 
Cluster management, etc. 
tools for creating more accurate instrument models. 
collect additional types of observation data by launching a new 
instrument. 
collect a new type of observation data, but keep the total amount 
of data processed the same. 
develop higher resolution models and move to higher resolution 
simulation 
System Management/Tuning tools 
Instrument models 
Launch new data source 
Launch replacement data source 
Higher resolution models 
Research Area Impact 
Impact matrix has a value for each pair (13 research areas x 12 
system characteristics): 156 possible, but only 18 are nonzero. 
Impact can be positive or negative: 
Impact(targeted observing, observation density) = low neg. 
Impact(1aunch new data source, observation density) = low 
Some more examples: 
Impact(targeted observing, targeting efficiency) = low 
Impact(system mgmt/tuning, cpu efficiency) = low 
Impact(adaptive grid, simulation efficiency) = medium 
59 
Impact Matrix 
g 
._ 
0 
E 
same data size 
(less data) 
targeted obsewing 
improved ensemble 
launch replacement data 
I 
+ 
Qualitative 3 Quantitative 
Impact is parameterized qualitatively (lo, med, hi). This 
qualitative scale is then quantified inside the model. 
Each of the parameters has a different interpretation 
under the four scenarios (pessimistic, consensus, 
optimistic, ideal). This allows us to compare in a best- 
case vs. worst-case manner. 
pess. cons. optim. ideal 
Lo .05 .I .I5 1.0 
Med .2 .3 .4 I .o 
Hi .3 .5 .7 I .o 
60 
lnstantiatinq the Model 
Overall 
Model 
System Priorities Model: designed/owned by program 
manager cognizant of NASA priorities 
System Priorities Model computes overall System Val1 e 
given the System Metrics. 
System Priorities Model 
science 
understanding 
value to  
coast-dwelling 
stakeholder 
e spo n sivenes s 
to charter 
............................................................... 
. . . . .  
. . . . . .  
. . . . . .  
61 
Review: Combining the Models 
Characteristics Performance & 
Cost Metrics Overall 
Model 
Assessment 
Priorities 
Model 
Model 
ults: Ca veat 
Remember: results (evaluations, ROI, etc.) 
must be understood as a function of the inputs used 
to calculate the results: 
f (  model, assumptions, priorities) 
Priorities depend on perspective: 
we model basic (science value only) 
versus applied (economic value only) 
of mt improvement in *+he .. 
9: NASA :Ea 
.......................................................................... .... optimbm o n r  research outcomcr 0 
0; .. none " " i n a  ..................................... 
proposal 1 r Tdals - 1  
~b r Tot& 
367M 1592M 
Basic: launch new data source (35M) & targeted observing (22Mj 
Armlied: data-mining (2.5B) & improved ensemble methods (1.5B) 
Evaluating Research Areas 
proposal 1 
63 
Sensitivity to "optimism" variable: two research areas have vastly higher 
potential impact under ideal assumptions. Pessimistic view of data- 
lining exceeds optimistic assessment of other areas. r 
Mid Value of nei improvement in value 
~~c,spplied &&er 9 ..... &"G ... 
icsd Pr-7 -... ....... ............................ : 
[3 ....... 
-. iDSY 
.- 
milRy model 
....... ............... .... 
KCF optrnkmaarese8rchotHcomerr 
xcuh: pr-2 v 
Synergy Between Research Areas 
We can look for synergies by finding pairs of research 
areas with much higher value than the two areas 
individually ... 
Under the applied research focus: 
Biggest synergies 
Launch new data source ($I.SB) 
+ targeted observing ($IB) 
yields a synergy of $700MM 
Launch new data source ($1.5B) 
+ data-efficient simulations ($800MM) yields a 
synergy of $400MM 
64 
Understanding the Model 
I dataselection I 
r 7 assimilation - -  process 
( - T I  
characteristics 
BLUE OVALS summarize 
the way that system changes 
flow through the assessment 
characteristics 
model. We can diagnose our 
assumptions by analyzing 
how these variables vary as 
we vary research area. 
imp1 process 
, I  
characteristics , 
System performance and cost metrics 
65 
Agenda 
CICT Systems Analysis 
Our modeling approach 
- a 3-part schematic investment model 
of technology change, impact 
assessment and prioritization 
A whirlwind tour of our model 
Lessons learned 
Modelina lessons learned. . 
Model and modeling technology should be: 
understandable and easy to use 
and should supporf= 
va ryi ng levels of deta i I (q ua I ita t ive3 qua ntitative) 
varying scope 
development of models by distributed stakeholders 
multiple uses / answer multiple questions 
varying assumptions/priorities 
communication/debate/collaboration 
(cross-cutting value as well as mission-specific value) 
Lessons learned. . . 
Model preferences of different stakeholders 
Allow for easy variation in assumptions (“what if 
ex p I i ci t I y 
our model is wrong? ... our estimates overly 
opt i m i st i c?‘? 
Compare impact of each technology to a no- 
investment baseline 
Make models modular and decoupled: 
technology investments 3 
system characteristics 3 
performance metrics 3 
“return” or “mission value” 
(three arrows == three submodels) 
66 
End of workshop talk. 
Full report is available at 
http://support. thinkbank. com/essa-final 
67 


The CICT Earth Science Systems Analysis Model

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server