Beth F Wheeler Atkinson

Jennifer Pagan

Melissa M Walwanis

Nelson

CiteSeerX

Evaluating Behavior Modeling Toolsets 
Jennifer Pagan  
Kaegan Corporation 
12350 Research Parkway 
 Orlando, FL 32826  
jennifer.pagan1.ctr@navy.mil 
 
Beth F. Wheeler Atkinson 
 Melissa M. Walwanis Nelson 
Naval Air Warfare Center Training Systems Division (NAWCTSD) 
12350 Research Parkway 
 Orlando, FL 32826 
beth.atkinson@navy.mil, melissa.walwanis@navy.mil 
 
1. Introduction 
 
Simulation-based training is increasingly important in 
Navy training. However, replicating real-world 
environments has inherent challenges such as the 
necessity to provide realistic human behaviors in the 
simulated environment.  One solution is to use human 
role-players for friendly and enemy forces.  However, 
using role-players is costly in terms of money used to hire 
outside contractors, operational time foregone by 
volunteer role-players, and the added equipment for role-
players.  Semi-Automated Forces (SAFs) provide a less 
costly alternative to replicating friendly, enemy, and 
neutral platforms in the virtual environment.  They are 
controlled and monitored by a human that pre-scripts 
command processes (Department of Defense, 1998).  
Although SAFs decrease the costs associated with using 
human-role players, the pre-scripted nature of their 
behaviors presents some inherent challenges.  This paper 
provides an overview of the current state-of-the-art in 
human behavior modeling and outlines remaining 
challenges. The authors then provide a practical 
framework for evaluating rapid human behavioral 
modeling toolsets to overcome the presented challenges. 
 
2. Challenges of Pre-scripted Behaviors  
 
While SAF behavior significantly contributes to the 
realism of training scenarios, limited behaviors provide an 
unrealistic situation that may hinder training transfer 
(Gelenbe, Hussain, & Kaptan, 2005).  This lack of realism 
is often because SAFs must be scripted prior to the 
training event.  For this reason, many mission variations 
are preprogrammed to facilitate realistic tactical 
behaviors.  Further, some training scenarios require 
thousands of SAF entities that must be pre-scripted to 
successfully execute training. However, pre-scripting this 
many entities with several behavioral variations is 
impractical due to time constraints and increased 
manpower requirements (Cox & Fu, 2005).   
 
Even when SAF entities are scripted with few behavior 
variations, scripting large numbers of SAFs in short 
periods of time also presents challenges.  There is often 
an increase in manpower to support scenario generation, 
(albeit, less than using live role-players) and instructors 
work long hours to ensure that training events are kept on 
schedule. Increased work hours contribute to cognitive 
fatigue and thus could limit the quality of training 
provided by an instructor (Whelan, Loftus, Perme, & 
Baldwin, 2002).  Finally, as large scale simulation-based 
training events become more common and increase in 
scale, additional instructors are required to monitor SAF 
behaviors, causing training costs to increase (Furness & 
Tyler, 2001).   
 
3. Behavior Modeling Evaluation 
 
The previously mentioned challenges to SAFs limit 
fidelity and increase costs, showing a need to practically 
evaluate current human behavior modeling toolsets in a 
manner that can help overcome these challenges. A 
review of current behavior modeling technologies 
indicates two prominent technical approaches for creating 
more realistic SAFs: algorithms and hierarchies. While 
algorithmic approaches use behavioral instances to 
capture demonstrated behaviors, hierarchal approaches 
decompose high level tasks or goals into primitives to 
elicit behaviors.  Both approaches of behavior modeling 
have shown to be effective methods of producing more 
realistic behaviors (Banks & Stytz, 2003).  While these 
approaches are effective means of modeling realistic 
behavior, toolsets using these approaches should be 
evaluated on several criteria to practically increase Return 
on Investment and drive future scientific inquiry. 
 
Proceedings of the 19th Conference on Behavior Representation in Modeling and Simulation, Charleston, SC, 21 - 24 March 2010
319
We have developed a behavior modeling toolset 
evaluation framework which can be divided into three 
categories: cost, schedule, and performance.  Each 
category has its own set of evaluation criteria. 
 
3.1 Cost 
 
The cost category is broken into three criteria thought to 
reduce the cost of implementing a behavior modeling 
toolset.  The three evaluation criteria are: 
1) Domain Independence. Can entities be reused in a 
variety of training scenarios and simulations regardless of 
developmental domain? 
2) Technology Readiness Level (TRL). What is the level 
of maturity of the technology?   
3) Resource Requirements.  How much funding is 
required to increase product maturity? 
 
3.2 Schedule 
 
The time category consists of one criterion:  
1) Rapid Scripting Capabilities.  Can the toolset rapidly 
script entity behaviors? 
 
3.3 Performance 
 
The performance category is focused on the actual 
performance of the entity or toolset, and consists of two 
components:  
1) Autonomy. Does the toolset reduce the manpower 
required to monitor entities? 
2) Communication Capability.  Does the toolset support 
more realistic interaction with entities? 
 
4. Benefits 
 
There are numerous anticipated benefits of evaluating 
toolsets using this framework.  First, training fidelity and 
transfer are expected to increase, as rapid scripting 
reduces the time necessary to produce more behavior 
variations than current SAFs provide.  Communication 
capabilities can also enhance realism by allowing the 
trainee to simulate communication with entities (Furness 
& Tyler, 2001).  Next, manpower requirements are 
expected to decrease as the reuse of behavior models in 
various training scenarios and simulations reduces 
scenario generation time.  The production of autonomous 
entities is expected to further reduce manpower costs by 
reducing monitoring requirements.  Costs are further 
reduced by selecting toolsets that have higher TRLs and 
fewer resource requirements.  Finally, reduction in 
scenario generation time and monitoring requirements can 
also alleviate the cognitive strain placed on instructors 
allowing them to focus on other aspects of the training 
scenario, such as performance measurement. 
 
Authors’ Note.  The views expressed herein are those of 
the authors and do not necessarily reflect the official 
position of the organizations with which they are 
affiliated.  
 
5. References 
 
Banks, S., & Stytz, M. (2003).  Progress and prospects 
 for the development of computer-generated 
 actors for military simulation: Part 2 – 
 Reasoning system architectures and human 
 behavior modeling.  Presence, 12(4), 422-436. 
Cox, C., & Fu, D. (2005).  AI for automated combatants 
 in a training application. Proceedings of the 
 second Australasian conference on Interactive 
 entertainment. 
Department of Defense (1998). Department of Defense 
 Modeling and Simulation Glossary, Manual No. 
 5000.59-M., Washington DC. 
Furness, Z., & Tyler, J. (2001). Fully automated 
 simulation forces (FAFs): A grand challenge for 
 military training. Proceedings of the 2001 Fall 
 Simulation Interoperability Workshop. 
Gelenbe, E., Hussain, K., & Kaptan, V. (2005).  
 Simulating autonomous agents in augmented 
 reality.  The Journal of Stems and Software, 74, 
 255–268. 
Whelan, M., Loftus, J., Perme, D., & Baldwin, R. (2002). 
 Reducing training costs through integration of 
 simulations, C4I systems, and expert systems.  
 Proceedings of the 2002 Winter Simulation 
 Conference. 
 
Authors’ Biographies 
 
Jennifer Pagan is a Research Psychology Assistant at 
Kaegan Corporation, Orlando FL.  She has an M.S. 
degree in Industrial/Organizational (I/O) Psychology from 
the University of Central Florida. 
 
Beth F. Wheeler Atkinson is a Research Psychologist at 
NAWCTSD. Her research interests include human-
computer interaction, instructional tools, and distributed 
mission training. She holds a M.A. in General Psychology 
from the University of West Florida. 
 
Melissa M. Walwanis Nelson is a Senior Research 
Psychologist at NAWCTSD. She has a M.S. degree in I/O 
Psychology from the University of Central Florida. Her 
research interests include distributed mission training, 
leadership development, simulator instructional tools, and 
network centric warfare concepts for coalitions. 
Proceedings of the 19th Conference on Behavior Representation in Modeling and Simulation, Charleston, SC, 21 - 24 March 2010
320


Evaluating Behavior Modeling Toolsets

http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=D8E711150F6114C8C7E7A70D65847A47?doi=10.1.1.1046.8647&rep=rep1&type=pdf

Evaluating Behavior Modeling Toolsets

Abstract

Similar works

Full text

Available Versions

CiteSeerX