Statistical Symbolic Execution with Informed Sampling

Symbolic execution techniques have been proposed recently for the probabilistic analysis of programs. These techniques seek to quantify the likelihood of reaching program events of interest, e.g., assert violations. They have many promising applications but have scalability issues due to high computational demand. To address this challenge, we propose a statistical symbolic execution technique that performs Monte Carlo sampling of the symbolic program paths and uses the obtained information for Bayesian estimation and hypothesis testing with respect to the probability of reaching the target events. To speed up the convergence of the statistical analysis, we propose Informed Sampling, an iterative symbolic execution that first explores the paths that have high statistical significance, prunes them from the state space and guides the execution towards less likely paths. The technique combines Bayesian estimation with a partial exact analysis for the pruned paths leading to provably improved convergence of the statistical analysis. We have implemented statistical symbolic execution with in- formed sampling in the Symbolic PathFinder tool. We show experimentally that the informed sampling obtains more precise results and converges faster than a purely statistical analysis and may also be more efficient than an exact symbolic analysis. When the latter does not terminate symbolic execution with informed sampling can give meaningful results under the same time and memory limits

A RISK-INFORMED DECISION-MAKING METHODOLOGY TO IMPROVE LIQUID ROCKET ENGINE PROGRAM TRADEOFFS

This work provides a risk-informed decision-making methodology to improve liquid rocket engine program tradeoffs with the conflicting areas of concern affordability, reliability, and initial operational capability (IOC) by taking into account psychological and economic theories in combination with reliability engineering. Technical program risks are associated with the number of predicted failures of the test-analyze-and-fix (TAAF) cycle that is based on the maturity of the engine components. Financial and schedule program risks are associated with the epistemic uncertainty of the models that determine the measures of effectiveness in the three areas of concern. The affordability and IOC models' inputs reflect non-technical and technical factors such as team experience, design scope, technology readiness level, and manufacturing readiness level. The reliability model introduces the Reliability- As-an-Independent-Variable (RAIV) strategy that aggregates fictitious or actual hotfire tests of testing profiles that differ from the actual mission profile to estimate the system reliability. The main RAIV strategy inputs are the physical or functional architecture of the system, the principal test plan strategy, a stated reliability-bycredibility requirement, and the failure mechanisms that define the reliable life of the system components. The results of the RAIV strategy, which are the number of hardware sets and number of hot-fire tests, are used as inputs to the affordability and the IOC models. Satisficing within each tradeoff is attained by maximizing the weighted sum of the normalized areas of concern subject to constraints that are based on the decision-maker's targets and uncertainty about the affordability, reliability, and IOC using genetic algorithms. In the planning stage of an engine program, the decision variables of the genetic algorithm correspond to fictitious hot-fire tests that include TAAF cycle failures. In the program execution stage, the RAIV strategy is used as reliability growth planning, tracking, and projection model. The main contributions of this work are the development of a comprehensible and consistent risk-informed tradeoff framework, the RAIV strategy that links affordability and reliability, a strategy to define an industry or government standard or guideline for liquid rocket engine hot-fire test plans, and an alternative to the U.S. Crow/AMSAA reliability growth model applying the RAIV strategy

How \u3ci\u3eDaubert\u3c/i\u3e and its Progeny Have Failed Criminalistics Evidence and a Few Things the Judiciary Could Do About It.

Part I documents how courts have failed to faithfully apply Daubert’s criteria for scientific validity to this type of evidence. It describes how ambiguities and flaws in the terminology adopted in Daubert combinedwith the opaqueness of forensic-science publications and standards have been exploited to shield some test methods from critical judicial analysis. Simply desisting from these avoidance strategies would be an improvement. Part II notes how part of the U.S. Supreme Court’s opinion in Kumho Tire Co. v. Carmichael has enabled courts to lower the bar for what is presented as scientific evidence by mistakenly maintaining that there is no difference between that evidence and other expert testimony that need not be scientifically validated. It suggests that a version of Rule 702 that explicitly insists on more rigorous validation of evidence that is promoted or understood as being “scientific” would be workable and more clearly compatible with the rule’s common law roots. Part III sketches various meanings of the terms “reliability” and “validity” in science and statistics, on the one hand, and in the rules and opinions on the admissibility of expert evidence, on the other. It discusses the two-part definition of “validity” in the PCAST report and the proposed criteria for demonstrating scientific validity of subjective pattern-matching testimony. It contends that if “validity” means that a procedure (even a highly subjective one) for making measurements and drawing inferences is fit for its intended use, then whether test results that have higher error rates than the ones selected in the report might nevertheless assist fact finders who are also appropriately informed of the evidence’s probative value must be evaluated. Finally, Part IV articulates two distinct approaches to informing judges or jurors of the import of similarities in features: the traditional one in which examiners opine on the truth and falsity of source hypotheses and a more finely grained one in which criminalists report only on the strength of the evidence. It suggests that the rules for admitting scientific evidence need to be flexible enough to accommodate the latter, likelihood-based testimony when it has a satisfactory empirically established basis

Anything for a Cheerio: Brown Capuchins (\u3cem\u3eSapajus [Cebus] apella\u3c/em\u3e) Consistently Coordinate in an Assurance Game for Unequal Payoffs

Unequal outcomes disrupt cooperation in some situations, but this has not been tested in the context of coordination in economic games. To explore this, we tested brown capuchins (Sapajus [Cebus] apella) on a manual version of the Stag Hunt (or Assurance) Game, in which individuals sequentially chose between two options, Stag or Hare, and were rewarded according to their choices and that of their partner. Typically, coordination on Stag results in an equal highest payout, whereas coordinating on Hare results in a guaranteed equal but lower payoff and uncoordinated play results in the lowest payoff when playing Stag. We varied this structure such that one capuchin received double the rewards for the coordinated Stag outcome; thus, it was still both animals\u27 best option, but no longer equally rewarding. Despite the inequality, capuchins coordinated on Stag in 78% of trials, and neither payoff structure nor their partner\u27s choice impacted their decision. Additionally, there was no relationship between self-scratching, a measure of stress in capuchins, and choices. After completing the study, we discovered our reward, cheerios, was sufficiently valuable that in another study, capuchins never refused it, so post hoc we repeated the study using a lower value reward, banana flavored pellets. Capuchins completed only 26% of the pellet trials (compared to 98% with cheerios), constraining our ability to interpret the results, but nonetheless the monkeys showed a decrease in preference for Stag, particularly when they received fewer rewards for the coordinated Stag outcome. These results reinforce capuchins\u27 ability to find coordinated outcomes in the Stag Hunt game, but more work is needed to determine whether the monkeys did not mind the inequality or were unwilling to sacrifice a highly preferred food to rectify it. In either case, researchers should carefully consider the impact of their chosen rewards on subjects\u27 choices

Risk-based regulation of unmanned aircraft systems

The aviation sector is faced with a novel array of new airspace users including Urban Air Mobility (UAM) concepts, personal air mobility vehicles, reusable space launch vehicles, and Unmanned Aircraft Systems (UAS). Focusing on UAS, there is much effort being directed towards the development of safety regulations for this industry. National Aviation Authorities (NAA) have advocated the adoption of a risk-based approach to the development of regulations, whereby regulations are driven by the outcomes of a systematic process to assess and manage identified safety risks. Central to a risk-based approach is the Safety Risk Management Process (SRMP). A review of relevant aviation safety policy, guidance and regulatory material found that aviation safety literature does not adequately address the uncertainty inherent to any SRMP. For example, when measuring risk, only the likelihood and severity are taken into consideration, with uncertainty generally not being mentioned. Where uncertainty is recognised, it is taken into consideration through the use of conservative worst-case assumptions. This can result in the imposition of overly stringent restrictions or worse, regulations that do not adequately mitigate safety risks. Subsequently, providing a more comprehensive treatment of uncertainty in the aviation SRMP is essential to the uptake of a risk-based approach to rule-making. Further, it follows that if assessments of performance can be uncertain, then these uncertainties also need to be accounted for in other NAA regulatory processes such as the regulatory compliance assessment and compliance finding processes. It was found that the current aviation compliance process does not provide an objective means for accounting for uncertainty. As a consequence, compliance assessments can be subjective and inconsistent, with regulators lacking the tools and processes to be able to make objective compliance findings on the basis of compliance risk. A means to enable NAA to account for uncertainty in regulatory compliance processes is needed. The overall aim of this thesis is to improve regulatory outcomes under the new paradigm of risk-based regulation, through providing a conceptual framework for the rational, transparent and systematic treatment of uncertainty in the risk assessment and regulatory decision-making processes. The thesis proposes the application of Bayesian methods and normative decision theory to the aviation safety regulatory process. System Safety Regulations (SSR), commonly referred to as "Part 1309" regulations, for UAS are used as a case study. It is posited that the general theoretical approach proposed in this thesis can improve the objectivity, consistency, and transparency of current aviation regulatory processes. The generalised approaches presented in this thesis enable the adoption of risk-based rulemaking for new aviation sectors and provides the theoretical basis for risk-based compliance; a paradigm shift in how aviation safety regulators approach risk-based regulation

Reasoning About the Reliability of Multi-version, Diverse Real-Time Systems

This paper is concerned with the development of reliable real-time systems for use in high integrity applications. It advocates the use of diverse replicated channels, but does not require the dependencies between the channels to be evaluated. Rather it develops and extends the approach of Little wood and Rush by (for general systems) by investigating a two channel system in which one channel, A, is produced to a high level of reliability (i.e. has a very low failure rate), while the other, B, employs various forms of static analysis to sustain an argument that it is perfect (i.e. it will never miss a deadline). The first channel is fully functional, the second contains a more restricted computational model and contains only the critical computations. Potential dependencies between the channels (and their verification) are evaluated in terms of aleatory and epistemic uncertainty. At the aleatory level the events ''A fails" and ''B is imperfect" are independent. Moreover, unlike the general case, independence at the epistemic level is also proposed for common forms of implementation and analysis for real-time systems and their temporal requirements (deadlines). As a result, a systematic approach is advocated that can be applied in a real engineering context to produce highly reliable real-time systems, and to support numerical claims about the level of reliability achieved

An integrated model for asset reliability, risk and production efficiency management in subsea oil and gas operations

PhD ThesisThe global demand for energy has been predicted to rise by 56% between 2010 and 2040 due to industrialization and population growth. This continuous rise in energy demand has consequently prompted oil and gas firms to shift activities from onshore oil fields to tougher terrains such as shallow, deep, ultra-deep and arctic fields. Operations in these domains often require deployment of unconventional subsea assets and technology. Subsea assets when installed offshore are super-bombarded by marine elements and human factors which increase the risk of failure. Whilst many risk standards, asset integrity and reliability analysis models have been suggested by many previous researchers, there is a gap on the capability of predictive reliability models to simultaneously address the impact of corrosion inducing elements such as temperature, pressure, pH corrosion on material wear-out and failure. There is also a gap in the methodology for evaluation of capital expenditure, human factor risk elements and use of historical data to evaluate risk. This thesis aims to contribute original knowledge to help improve production assurance by developing an integrated model which addresses pump-pipe capital expenditure, asset risk and reliability in subsea systems. The key contributions of this research is the development of a practical model which links four sub-models on reliability analysis, asset capital cost, event risk severity analysis and subsea risk management implementation. Firstly, an accelerated reliability analysis model was developed by incorporating a corrosion covariate stress on Weibull model of OREDA data. This was applied on a subsea compression system to predict failure times. A second methodology was developed by enhancing Hubbert oil production forecast model, and using nodal analysis for asset capital cost analysis of a pump-pipe system and optimal selection of best option based on physical parameters such as pipeline diameter, power needs, pressure drop and velocity of fluid. Thirdly, a risk evaluation method based on the mathematical determinant of historical event magnitude, frequency and influencing factors was developed for estimating the severity of risk in a system. Finally, a survey is conducted on subsea engineers and the results along with the previous models were developed into an integrated assurance model for ensuring asset reliability and risk management in subsea operations. A guide is provided for subsea asset management with due consideration to both technical and operational perspectives. The operational requirements of a subsea system can be measured, analysed and improved using the mix of mathematical, computational, stochastic and logical frameworks recommended in this work

Automatic Processing of High-Rate, High-Density Multibeam Echosounder Data

Multibeam echosounders (MBES) are currently the best way to determine the bathymetry of large regions of the seabed with high accuracy. They are becoming the standard instrument for hydrographic surveying and are also used in geological studies, mineral exploration and scientific investigation of the earth\u27s crustal deformations and life cycle. The significantly increased data density provided by an MBES has significant advantages in accurately delineating the morphology of the seabed, but comes with the attendant disadvantage of having to handle and process a much greater volume of data. Current data processing approaches typically involve (computer aided) human inspection of all data, with time-consuming and subjective assessment of all data points. As data rates increase with each new generation of instrument and required turn-around times decrease, manual approaches become unwieldy and automatic methods of processing essential. We propose a new method for automatically processing MBES data that attempts to address concerns of efficiency, objectivity, robustness and accuracy. The method attributes each sounding with an estimate of vertical and horizontal error, and then uses a model of information propagation to transfer information about the depth from each sounding to its local neighborhood. Embedded in the survey area are estimation nodes that aim to determine the true depth at an absolutely defined location, along with its associated uncertainty. As soon as soundings are made available, the nodes independently assimilate propagated information to form depth hypotheses which are then tracked and updated on-line as more data is gathered. Consequently, we can extract at any time a “current-best” estimate for all nodes, plus co-located uncertainties and other metrics. The method can assimilate data from multiple surveys, multiple instruments or repeated passes of the same instrument in real-time as data is being gathered. The data assimilation scheme is sufficiently robust to deal with typical survey echosounder errors. Robustness is improved by pre-conditioning the data, and allowing the depth model to be incrementally defined. A model monitoring scheme ensures that inconsistent data are maintained as separate but internally consistent depth hypotheses. A disambiguation of these competing hypotheses is only carried out when required by the user. The algorithm has a low memory footprint, runs faster than data can currently be gathered, and is suitable for real-time use. We call this algorithm CUBE (Combined Uncertainty and Bathymetry Estimator). We illustrate CUBE on two data sets gathered in shallow water with different instruments and for different purposes. We show that the algorithm is robust to even gross failure modes, and reliably processes the vast majority of the data. In both cases, we confirm that the estimates made by CUBE are statistically similar to those generated by hand

Auditing Symposium VIII: Proceedings of the 1986 Touche Ross/University of Kansas Symposium on Auditing Problems

Discussant\u27s response to On the economics of product differentiation in auditing / Howard R. Osharow; Unresolved issues in classical audit sample evaluations / Donald R. Nichols, Rajendra P. Srivastava, Bart H. Ward; Discussant\u27s response to Unresolved issues in classical audit sample evaluations / Abraham D. Akresh; Under the spreading chestnut tree, accountants\u27 legal liability -- A historical perspective / Paul J. Ostling; Impact of technological events and trends on audit evidence in the year 2000: Phase I / Gary L. Holstrum, Theodore J. Mock, Robert N. West; Discussant\u27s Response to Impact of technological events and trends on audit evidence in the year 2000: Phase I; Is the second standard of fieldwork necessary / Thomas P. Bintinger; Discussant\u27s response to Is the second standard of fieldwork necessary / Andrew D. Bailey; Interim report on the development of an expert system for the auditor\u27s loan loss evaluation / Kirk P. Kelly, Gary S. Ribar, John J. Willingham; Discussant\u27s response to Interim report on the development of an expert system for the auditor\u27s loan loss evaluation / William F. Messier; Work of the Special Investigations Committee / R. K. (Robert Kuhn) Mautz (1915-2002); Discussant\u27s response to Under the spreading chestnut tree, accountants\u27 legal liability -- A historical perspective / Thomas A. Gavin; Assertion based approach to auditing / Donald A. Leslie; Discussant\u27s response to An assertion-based approach to auditing / William L. Felixhttps://egrove.olemiss.edu/dl_proceedings/1007/thumbnail.jp