Eliciting informal specifications from scientific modelers for evaluation and debugging

Professional software engineers have an arsenal of techniques such as unit testing and assertions to check their specifications, but these techniques require tools, motivation, experience and training that programmers without professional software engineering training may not have. As a result, professionals in other fields, such as scientific modelers, face greater hurdles in debugging and validating the programs they write. This thesis introduces the concept of "evaluation abstractions" as a framework for tool designers to think about this kind of support. Evaluation abstractions are the patterns of data in program traces and outputs that programmers examine in order to evaluate software behavior. The thesis provides two intellectual contributions aimed at helping tool designers: (1) A theory of evaluation abstraction support (EAST) that describes at a granular scale the factors contributing to a modeler's decision to use or not use an evaluation abstraction support feature; (2) a new user-centered design methodology, Natural Programming Plus (NP+), specialized for the design of interactive languages aimed at experienced users, in a way that allows for validation early in the process. Using EAST and NP+ I built and evaluated an evaluation abstraction support tool for cognitive modelers (psychologists who study human cognition by writing simulations of cognition), with features that (1) elicit and persist a database of a modeler's evaluation abstractions, in a piecemeal, just-in-time fashion as their questions about model behavior arise, and (2) use the modeler's unique set of evaluation abstractions to structure visualizations, listings, and regression tests, as the modeler continues to maintain and develop the project. Using this tool modelers were able to repeatedly answer questions about model behavior that would have been time-consuming and error-prone to check in state-of-the-art cognitive modeling tools. This dissertation includes formative investigation of modelers' evaluation abstractions, iterative development and testing of interaction designs for elicitation and use of evaluation abstractions, a description of a domain-specific language for representing and transforming evaluation abstractions, and two summative studies showing the usability and generalizability of the technique