Observation of Software for Distributed Systems with RCL

Program observation involves formulating a query about the behavior of a program and then observing the program as it executes in order to determine the result of the query. Observation is used in software development to track down bugs and clarify understanding of a program's behavior, and in software testing to ensure that a program behaves as expected for a given input set. RCL is a recursive logic built upon conjunctive global predicates. Computational structures of common paradigms such as butterfly synchronization and distributed consensus can be expressed easily in RCL. A nonintrusive decentralized algorithm for detecting RCL predicates is developed and proven correct