Collected notes from the Benchmarks and Metrics Workshop

In recent years there has been a proliferation of proposals in the artificial intelligence (AI) literature for integrated agent architectures. Each architecture offers an approach to the general problem of constructing an integrated agent. Unfortunately, the ways in which one architecture might be considered better than another are not always clear. There has been a growing realization that many of the positive and negative aspects of an architecture become apparent only when experimental evaluation is performed and that to progress as a discipline, we must develop rigorous experimental methods. In addition to the intrinsic intellectual interest of experimentation, rigorous performance evaluation of systems is also a crucial practical concern to our research sponsors. DARPA, NASA, and AFOSR (among others) are actively searching for better ways of experimentally evaluating alternative approaches to building intelligent agents. One tool for experimental evaluation involves testing systems on benchmark tasks in order to assess their relative performance. As part of a joint DARPA and NASA funded project, NASA-Ames and Teleos Research are carrying out a research effort to establish a set of benchmark tasks and evaluation metrics by which the performance of agent architectures may be determined. As part of this project, we held a workshop on Benchmarks and Metrics at the NASA Ames Research Center on June 25, 1990. The objective of the workshop was to foster early discussion on this important topic. We did not achieve a consensus, nor did we expect to. Collected here is some of the information that was exchanged at the workshop. Given here is an outline of the workshop, a list of the participants, notes taken on the white-board during open discussions, position papers/notes from some participants, and copies of slides used in the presentations

Evaluating Social Programs at the State and Local Level: The JTPA Evaluation Design Project

This book draws upon the JTPA Evaluation Design Project initiated, developed and directed by the Washington State Employment Security Department. The five essays offer practical, instructive guidance about planning and executing program evaluations.https://research.upjohn.org/up_press/1218/thumbnail.jp

CLiFF Notes: Research In Natural Language Processing at the University of Pennsylvania

The Computational Linguistics Feedback Forum (CLIFF) is a group of students and faculty who gather once a week to discuss the members\u27 current research. As the word feedback suggests, the group\u27s purpose is the sharing of ideas. The group also promotes interdisciplinary contacts between researchers who share an interest in Cognitive Science. There is no single theme describing the research in Natural Language Processing at Penn. There is work done in CCG, Tree adjoining grammars, intonation, statistical methods, plan inference, instruction understanding, incremental interpretation, language acquisition, syntactic parsing, causal reasoning, free word order languages, ... and many other areas. With this in mind, rather than trying to summarize the varied work currently underway here at Penn, we suggest reading the following abstracts to see how the students and faculty themselves describe their work. Their abstracts illustrate the diversity of interests among the researchers, explain the areas of common interest, and describe some very interesting work in Cognitive Science. This report is a collection of abstracts from both faculty and graduate students in Computer Science, Psychology and Linguistics. We pride ourselves on the close working relations between these groups, as we believe that the communication among the different departments and the ongoing inter-departmental research not only improves the quality of our work, but makes much of that work possible

Agents for educational games and simulations

This book consists mainly of revised papers that were presented at the Agents for Educational Games and Simulation (AEGS) workshop held on May 2, 2011, as part of the Autonomous Agents and MultiAgent Systems (AAMAS) conference in Taipei, Taiwan. The 12 full papers presented were carefully reviewed and selected from various submissions. The papers are organized topical sections on middleware applications, dialogues and learning, adaption and convergence, and agent applications

Uncertainty in Economic Growth and Inequality

A step to consilience, starting with a deconstruction of the causality of uncertainty that is embedded in the fundamentals of growth and inequality, following a construction of aggregation laws that disclose the invariance principle across heterogeneous individuals, ending with a reconstruction of metric models that yields deeper structural connections via U.S. GDP and income data

Light Water Sustainability Program: Optimizing Information Automation Using a New Method Based on System-Theoretic Process Analysis

This report describes the interim progress for research supporting the design and optimization of information automation systems for nuclear power plants. Much of the domestic nuclear fleet is currently focused on modernizing technologies and processes, including transitioning toward digitalization in the control room and elsewhere throughout the plant, along with a greater use of automation, artificial intelligence, robotics, and other emerging technologies. While there are significant opportunities to apply these technologies toward greater plant safety, efficiency, and overall cost-effectiveness, optimizing their design and avoiding potential safety and performance risks depends on ensuring that human-performance-related organizational and technical design issues are identified and addressed. This report describes modeling tools and techniques, based on sociotechnical system theory, to support these design goals and their application in the current research effort. The report is intended for senior nuclear energy stakeholders, including regulators, corporate management, and senior plant management. We have developed and employed a method to design an optimized information automation ecosystem (IAE) based on the systems-theoretic constructs underlying sociotechnical systems theory in general and the Systems-Theoretic Accident Modeling and Processes (STAMP) approach in particular. We argue that an IAE can be modeled as an interactive information control system whose behavior can be understood in terms of dynamic control and feedback relationships amongst the system’s technical and organizational components. Up to this point, we have employed a Causal Analysis based on STAMP (CAST) technique to examine a performance- and safety-related incident at an industry partner’s plant that involved the unintentional activation of an emergency diesel generator. This analysis provided insight into the behavior of the plant’s current information control structure within the context of a specific, significant event. Our ongoing analysis is focused on identifying near-term process improvements and longer-term design requirements for an optimized IAE system. The latter analyses will employ a second STAMP-derived technique, System-Theoretic Process Analysis (STPA). STPA is a useful modeling tool for generating and analyzing actual or potential information control structures. Finally, we have begun modeling plantwide organizational relationships and processes. Organizational system modeling will supplement our CAST and STPA findings and provide a basis for mapping out a plantwide information control architecture. CAST analysis findings indicate an important underlying contributor to the incident under investigation, and a significant risk to information automation system performance, was perceived schedule pressure, which exposed weaknesses in interdepartmental coordination between and within responsible plant organizations and challenged the resilience of established plant processes, until a human caused the initiating event. These findings are discussed in terms of their risk to overall system performance and their implications for information automation system resilience and brittleness. We present two preliminary information automation models. The proactive issue resolution model is a test case of an information automation concept with significant near-term potential for application and subsequent reduction in significant plant events. The IAE model is a more general representation of a broader, plantwide information automation system. From our results, we have generated a set of preliminary system-level requirements and safety constraints. These requirements will be further developed over the remainder of our project in collaboration with nuclear industry subject matter experts and specialists in the technical systems under consideration. Additionally, we will continue to pursue the system analyses initiated in the first part of our effort, with a particular emphasis on STPA as the main tool to identify weak or weakening ontrol structures that affect the resilience of organizations and programs. Our intent is to broaden the scope of the analysis from an individual use case to a related set of use cases (e.g., maintenance tasks, compliance tasks) with similar human-system performance challenges. This will enable more generalized findings to refine the Proactive Issue Resolution and IAE models, as well as their system-level requirements and safety constraints. We will use organizational system modeling analyses to supplement STPA findings and model development. We conclude the report with a set of summary recommendations and an initial draft list of system-level requirements and safety constraints for optimized information automation systems