Evidence-driven testing and debugging of software systems

Program debugging is the process of testing, exposing, reproducing, diagnosing and fixing software bugs. Many techniques have been proposed to aid developers during software testing and debugging. However, researchers have found that developers hardly use or adopt the proposed techniques in software practice. Evidently, this is because there is a gap between proposed methods and the state of software practice. Most methods fail to address the actual needs of software developers. In this dissertation, we pose the following scientific question: How can we bridge the gap between software practice and the state-of-the-art automated testing and debugging techniques? To address this challenge, we put forward the following thesis: Software testing and debugging should be driven by empirical evidence collected from software practice. In particular, we posit that the feedback from software practice should shape and guide (the automation) of testing and debugging activities. In this thesis, we focus on gathering evidence from software practice by conducting several empirical studies on software testing and debugging activities in the real-world. We then build tools and methods that are well-grounded and driven by the empirical evidence obtained from these experiments. Firstly, we conduct an empirical study on the state of debugging in practice using a survey and a human study. In this study, we ask developers about their debugging needs and observe the tools and strategies employed by developers while testing, diagnosing and repairing real bugs. Secondly, we evaluate the effectiveness of the state-of-the-art automated fault localization (AFL) methods on real bugs and programs. Thirdly, we conducted an experiment to evaluate the causes of invalid inputs in software practice. Lastly, we study how to learn input distributions from real-world sample inputs, using probabilistic grammars. To bridge the gap between software practice and the state of the art in software testing and debugging, we proffer the following empirical results and techniques: (1) We collect evidence on the state of practice in program debugging and indeed, we found that there is a chasm between (available) debugging tools and developer needs. We elicit the actual needs and concerns of developers when testing and diagnosing real faults and provide a benchmark (called DBGBench) to aid the automated evaluation of debugging and repair tools. (2) We provide empirical evidence on the effectiveness of several state-of-the-art AFL techniques (such as statistical debugging formulas and dynamic slicing). Building on the obtained empirical evidence, we provide a hybrid approach that outperforms the state-of-the-art AFL techniques. (3) We evaluate the prevalence and causes of invalid inputs in software practice, and we build on the lessons learned from this experiment to build a general-purpose algorithm (called ddmax) that automatically diagnoses and repairs real-world invalid inputs. (4) We provide a method to learn the distribution of input elements in software practice using probabilistic grammars and we further employ the learned distribution to drive the test generation of inputs that are similar (or dissimilar) to sample inputs found in the wild. In summary, we propose an evidence-driven approach to software testing and debugging, which is based on collecting empirical evidence from software practice to guide and direct software testing and debugging. In our evaluation, we found that our approach is effective in improving the effectiveness of several debugging activities in practice. In particular, using our evidence-driven approach, we elicit the actual debugging needs of developers, improve the effectiveness of several automated fault localization techniques, effectively debug and repair invalid inputs, and generate test inputs that are (dis)similar to real-world inputs. Our proposed methods are built on empirical evidence and they improve over the state-of-the-art techniques in testing and debugging.Software-Debugging bezeichnet das Testen, Aufspüren, Reproduzieren, Diagnostizieren und das Beheben von Fehlern in Programmen. Es wurden bereits viele Debugging-Techniken vorgestellt, die Softwareentwicklern beim Testen und Debuggen unterstützen. Dennoch hat sich in der Forschung gezeigt, dass Entwickler diese Techniken in der Praxis kaum anwenden oder adaptieren. Das könnte daran liegen, dass es einen großen Abstand zwischen den vorgestellten und in der Praxis tatsächlich genutzten Techniken gibt. Die meisten Techniken genügen den Anforderungen der Entwickler nicht. In dieser Dissertation stellen wir die folgende wissenschaftliche Frage: Wie können wir die Kluft zwischen Software-Praxis und den aktuellen wissenschaftlichen Techniken für automatisiertes Testen und Debugging schließen? Um diese Herausforderung anzugehen, stellen wir die folgende These auf: Das Testen und Debuggen von Software sollte von empirischen Daten, die in der Software-Praxis gesammelt wurden, vorangetrieben werden. Genauer gesagt postulieren wir, dass das Feedback aus der Software-Praxis die Automation des Testens und Debuggens formen und bestimmen sollte. In dieser Arbeit fokussieren wir uns auf das Sammeln von Daten aus der Software-Praxis, indem wir einige empirische Studien über das Testen und Debuggen von Software in der echten Welt durchführen. Auf Basis der gesammelten Daten entwickeln wir dann Werkzeuge, die sich auf die Daten der durchgeführten Experimente stützen. Als erstes führen wir eine empirische Studie über den Stand des Debuggens in der Praxis durch, wobei wir eine Umfrage und eine Humanstudie nutzen. In dieser Studie befragen wir Entwickler zu ihren Bedürfnissen, die sie beim Debuggen haben und beobachten die Werkzeuge und Strategien, die sie beim Diagnostizieren, Testen und Aufspüren echter Fehler einsetzen. Als nächstes bewerten wir die Effektivität der aktuellen Automated Fault Localization (AFL)- Methoden zum automatischen Aufspüren von echten Fehlern in echten Programmen. Unser dritter Schritt ist ein Experiment, um die Ursachen von defekten Eingaben in der Software-Praxis zu ermitteln. Zuletzt erforschen wir, wie Häufigkeitsverteilungen von Teileingaben mithilfe einer Grammatik von echten Beispiel-Eingaben aus der Praxis gelernt werden können. Um die Lücke zwischen Software-Praxis und der aktuellen Forschung über Testen und Debuggen von Software zu schließen, bieten wir die folgenden empirischen Ergebnisse und Techniken: (1) Wir sammeln aktuelle Forschungsergebnisse zum Stand des Software-Debuggens und finden in der Tat eine Diskrepanz zwischen (vorhandenen) Debugging-Werkzeugen und dem, was der Entwickler tatsächlich benötigt. Wir sammeln die tatsächlichen Bedürfnisse von Entwicklern beim Testen und Debuggen von Fehlern aus der echten Welt und entwickeln einen Benchmark (DbgBench), um das automatische Evaluieren von Debugging-Werkzeugen zu erleichtern. (2) Wir stellen empirische Daten zur Effektivität einiger aktueller AFL-Techniken vor (z.B. Statistical Debugging-Formeln und Dynamic Slicing). Auf diese Daten aufbauend, stellen wir einen hybriden Algorithmus vor, der die Leistung der aktuellen AFL-Techniken übertrifft. (3) Wir evaluieren die Häufigkeit und Ursachen von ungültigen Eingaben in der Softwarepraxis und stellen einen auf diesen Daten aufbauenden universell einsetzbaren Algorithmus (ddmax) vor, der automatisch defekte Eingaben diagnostiziert und behebt. (4) Wir stellen eine Methode vor, die Verteilung von Schnipseln von Eingaben in der Software-Praxis zu lernen, indem wir Grammatiken mit Wahrscheinlichkeiten nutzen. Die gelernten Verteilungen benutzen wir dann, um den Beispiel-Eingaben ähnliche (oder verschiedene) Eingaben zu erzeugen. Zusammenfassend stellen wir einen auf der Praxis beruhenden Ansatz zum Testen und Debuggen von Software vor, welcher auf empirischen Daten aus der Software-Praxis basiert, um das Testen und Debuggen zu unterstützen. In unserer Evaluierung haben wir festgestellt, dass unser Ansatz effektiv viele Debugging-Disziplinen in der Praxis verbessert. Genauer gesagt finden wir mit unserem Ansatz die genauen Bedürfnisse von Entwicklern, verbessern die Effektivität vieler AFL-Techniken, debuggen und beheben effektiv fehlerhafte Eingaben und generieren Test-Eingaben, die (un)ähnlich zu Eingaben aus der echten Welt sind. Unsere vorgestellten Methoden basieren auf empirischen Daten und verbessern die aktuellen Techniken des Testens und Debuggens

Righting Web Development

The web browser is the most important application runtime today, encompassing all types of applications on practically every Internet-connected device. Browsers power complete office suites, media players, games, and augmented and virtual reality experiences, and they integrate with cameras, microphones, GPSes, and other sensors available on computing devices. Many apparently native mobile and desktop applications are secretly hybrid apps that contain a mix of native and browser code. History has shown that when new devices, sensors, and experiences appear on the market, the browser will evolve to support them. Despite the browser\u27s importance, developing web applications is exceedingly difficult. Web browsers organically evolved from a document viewer into a ubiquitous program runtime. The browser\u27s scripting language for web designers, JavaScript, has grown into the only universally supported programming language in the browser. Unfortunately, JavaScript is notoriously difficult to write and debug. The browser\u27s high-level and event-driven I/O interfaces make it easy to add simple interactions to webpages, but these same interfaces lead to nondeterministic bugs and performance issues in larger applications. These bugs are challenging for developers to reason about and fix. This dissertation revisits web development and provides developers with a complete set of development tools with full support for the browser environment. McFly is the first time-traveling debugger for the browser, and lets developers debug web applications and their visual state during time-travel; components of this work shipped in Microsoft\u27s ChakraCore JavaScript engine. BLeak is the first system for automatically debugging memory leaks in web applications, and provides developers with a ranked list of memory leaks along with the source code responsible for them. BCause constructs a causal graph of a web application\u27s events, which helps developers understand their code\u27s behavior. Doppio lets developers run code written in conventional languages in the browser, and Browsix brings Unix into the browser to enable unmodified programs expecting a Unix-like environment to run directly in the browser. Together, these five systems form a solid foundation for web development

New techniques for functional testing of microprocessor based systems

Electronic devices may be affected by failures, for example due to physical defects. These defects may be introduced during the manufacturing process, as well as during the normal operating life of the device due to aging. How to detect all these defects is not a trivial task, especially in complex systems such as processor cores. Nevertheless, safety-critical applications do not tolerate failures, this is the reason why testing such devices is needed so to guarantee a correct behavior at any time. Moreover, testing is a key parameter for assessing the quality of a manufactured product. Consolidated testing techniques are based on special Design for Testability (DfT) features added in the original design to facilitate test effectiveness. Design, integration, and usage of the available DfT for testing purposes are fully supported by commercial EDA tools, hence approaches based on DfT are the standard solutions adopted by silicon vendors for testing their devices. Tests exploiting the available DfT such as scan-chains manipulate the internal state of the system, differently to the normal functional mode, passing through unreachable configurations. Alternative solutions that do not violate such functional mode are defined as functional tests. In microprocessor based systems, functional testing techniques include software-based self-test (SBST), i.e., a piece of software (referred to as test program) which is uploaded in the system available memory and executed, with the purpose of exciting a specific part of the system and observing the effects of possible defects affecting it. SBST has been widely-studies by the research community for years, but its adoption by the industry is quite recent. My research activities have been mainly focused on the industrial perspective of SBST. The problem of providing an effective development flow and guidelines for integrating SBST in the available operating systems have been tackled and results have been provided on microprocessor based systems for the automotive domain. Remarkably, new algorithms have been also introduced with respect to state-of-the-art approaches, which can be systematically implemented to enrich SBST suites of test programs for modern microprocessor based systems. The proposed development flow and algorithms are being currently employed in real electronic control units for automotive products. Moreover, a special hardware infrastructure purposely embedded in modern devices for interconnecting the numerous on-board instruments has been interest of my research as well. This solution is known as reconfigurable scan networks (RSNs) and its practical adoption is growing fast as new standards have been created. Test and diagnosis methodologies have been proposed targeting specific RSN features, aimed at checking whether the reconfigurability of such networks has not been corrupted by defects and, in this case, at identifying the defective elements of the network. The contribution of my work in this field has also been included in the first suite of public-domain benchmark networks

Practical Run-time Checking via Unobtrusive Property Caching

The use of annotations, referred to as assertions or contracts, to describe program properties for which run-time tests are to be generated, has become frequent in dynamic programing languages. However, the frameworks proposed to support such run-time testing generally incur high time and/or space overheads over standard program execution. We present an approach for reducing this overhead that is based on the use of memoization to cache intermediate results of check evaluation, avoiding repeated checking of previously verified properties. Compared to approaches that reduce checking frequency, our proposal has the advantage of being exhaustive (i.e., all tests are checked at all points) while still being much more efficient than standard run-time checking. Compared to the limited previous work on memoization, it performs the task without requiring modifications to data structure representation or checking code. While the approach is general and system-independent, we present it for concreteness in the context of the Ciao run-time checking framework, which allows us to provide an operational semantics with checks and caching. We also report on a prototype implementation and provide some experimental results that support that using a relatively small cache leads to significant decreases in run-time checking overhead.Comment: 30 pages, 1 table, 170 figures; added appendix with plots; To appear in Theory and Practice of Logic Programming (TPLP), Proceedings of ICLP 201

Record and Transplay: Partial Checkpointing for Replay Debugging

Software bugs that occur in production are often difficult to reproduce in the lab due to subtle differences in the application environment and nondeterminism. Toward addressing this problem, we present Transplay, a system that captures application software bugs as they occur in production and deterministically reproduces them in a completely different environment, potentially running a different operating system, where the application, its binaries and other support data do not exist. Transplay introduces partial checkpointing, a new mechanism that provides two key properties. It efficiently captures the minimal state necessary to reexecute just the last few moments of the application before it encountered a failure. The recorded state, which typically consists of a few megabytes of data, is used to replay the application without requiring the specific application binaries or the original execution environment. Transplay integrates with existing debuggers to provide facilities such as breakpoints and single-stepping to allow the user to examine the contents of variables and other program state at each source line of the application's replayed execution. We have implemented a Transplay prototype that can record unmodified Linux applications and replay them on different versions of Linux as well as Windows. Experiments with server applications such as the Apache web server show that Transplay can be used in production with modest recording overhead

Doctor of Philosophy

dissertationA modern software system is a composition of parts that are themselves highly complex: operating systems, middleware, libraries, servers, and so on. In principle, compositionality of interfaces means that we can understand any given module independently of the internal workings of other parts. In practice, however, abstractions are leaky, and with every generation, modern software systems grow in complexity. Traditional ways of understanding failures, explaining anomalous executions, and analyzing performance are reaching their limits in the face of emergent behavior, unrepeatability, cross-component execution, software aging, and adversarial changes to the system at run time. Deterministic systems analysis has a potential to change the way we analyze and debug software systems. Recorded once, the execution of the system becomes an independent artifact, which can be analyzed offline. The availability of the complete system state, the guaranteed behavior of re-execution, and the absence of limitations on the run-time complexity of analysis collectively enable the deep, iterative, and automatic exploration of the dynamic properties of the system. This work creates a foundation for making deterministic replay a ubiquitous system analysis tool. It defines design and engineering principles for building fast and practical replay machines capable of capturing complete execution of the entire operating system with an overhead of several percents, on a realistic workload, and with minimal installation costs. To enable an intuitive interface of constructing replay analysis tools, this work implements a powerful virtual machine introspection layer that enables an analysis algorithm to be programmed against the state of the recorded system through familiar terms of source-level variable and type names. To support performance analysis, the replay engine provides a faithful performance model of the original execution during replay

A Survey of Algorithmic Debugging

"© ACM, 2017. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in ACM Computing Surveys, {50, 4, 2017} https://dl.acm.org/doi/10.1145/3106740"[EN] Algorithmic debugging is a technique proposed in 1982 by E. Y. Shapiro in the context of logic programming. This survey shows how the initial ideas have been developed to become a widespread debugging schema ftting many diferent programming paradigms and with applications out of the program debugging feld. We describe the general framework and the main issues related to the implementations in diferent programming paradigms and discuss several proposed improvements and optimizations. We also review the main algorithmic debugger tools that have been implemented so far and compare their features. From this comparison, we elaborate a summary of desirable characteristics that should be considered when implementing future algorithmic debuggers.This work has been partially supported by the EU (FEDER) and the Spanish Ministerio de Economia y Competitividad under grant TIN2013-44742-C4-1-R, TIN2016-76843-C4-1-R, StrongSoft (TIN2012-39391-C04-04), and TRACES (TIN2015-67522-C3-3-R) by the Generalitat Valenciana under grant PROMETEO-II/2015/013 (SmartLogic) and by the Comunidad de Madrid project N-Greens Software-CM (S2013/ICE-2731).Caballero, R.; Riesco, A.; Silva, J. (2017). A Survey of Algorithmic Debugging. ACM Computing Surveys. 50(4):1-35. https://doi.org/10.1145/3106740S135504Abramson, D., Foster, I., Michalakes, J., & Sosič, R. (1996). Relative debugging. Communications of the ACM, 39(11), 69-77. doi:10.1145/240455.240475K. R. Apt H. A. Blair and A. Walker. 1988. Towards a theory of declarative knowledge. In Foundations of Deductive Databases and Logic Programming J. Minker (Ed.). Morgan Kaufmann Publishers Inc. San Francisco CA 89--148. 10.1016/B978-0-934613-40-8.50006-3 K. R. Apt H. A. Blair and A. Walker. 1988. Towards a theory of declarative knowledge. In Foundations of Deductive Databases and Logic Programming J. Minker (Ed.). Morgan Kaufmann Publishers Inc. San Francisco CA 89--148. 10.1016/B978-0-934613-40-8.50006-3Arora, T., Ramakrishnan, R., Roth, W. G., Seshadri, P., & Srivastava, D. (1993). Explaining program execution in deductive systems. Lecture Notes in Computer Science, 101-119. doi:10.1007/3-540-57530-8_7E. Av-Ron. 1984. Top-Down Diagnosis of Prolog Programs. Ph.D. Dissertation. Weizmann Institute. E. Av-Ron. 1984. Top-Down Diagnosis of Prolog Programs. Ph.D. Dissertation. Weizmann Institute.A. Beaulieu. 2005. Learning SQL. O’Reilly Farnham UK. A. Beaulieu. 2005. Learning SQL. O’Reilly Farnham UK.D. Binks. 1995. Declarative Debugging in Gödel. Ph.D. Dissertation. University of Bristol. D. Binks. 1995. Declarative Debugging in Gödel. Ph.D. Dissertation. University of Bristol.B. Braßel and H. Siegel. 2008. Debugging Lazy Functional Programs by Asking the Oracle. Springer-Verlag Berlin 183--200. DOI:http://dx.doi.org/10.1007/978-3-540-85373-2_11 10.1007/978-3-540-85373-2_11 B. Braßel and H. Siegel. 2008. Debugging Lazy Functional Programs by Asking the Oracle. Springer-Verlag Berlin 183--200. DOI:http://dx.doi.org/10.1007/978-3-540-85373-2_11 10.1007/978-3-540-85373-2_11Caballero, R. (2005). A declarative debugger of incorrect answers for constraint functional-logic programs. Proceedings of the 2005 ACM SIGPLAN workshop on Curry and functional logic programming - WCFLP ’05. doi:10.1145/1085099.1085102Caballero, R., García-Ruiz, Y., & Sáenz-Pérez, F. (2012). Declarative Debugging of Wrong and Missing Answers for SQL Views. Lecture Notes in Computer Science, 73-87. doi:10.1007/978-3-642-29822-6_9Caballero, R., García-Ruiz, Y., & Sáenz-Pérez, F. (2015). Debugging of wrong and missing answers for datalog programs with constraint handling rules. Proceedings of the 17th International Symposium on Principles and Practice of Declarative Programming - PPDP ’15. doi:10.1145/2790449.2790522Caballero, R., Martin-Martin, E., Riesco, A., & Tamarit, S. (2015). A zoom-declarative debugger for sequential Erlang programs. Science of Computer Programming, 110, 104-118. doi:10.1016/j.scico.2015.06.011Caballero, R., & Rodríguez-Artalejo, M. (2002). A Declarative Debugging System for Lazy Functional Logic Programs. Electronic Notes in Theoretical Computer Science, 64, 113-175. doi:10.1016/s1571-0661(04)80349-9Ceri, S., Gottlob, G., & Tanca, L. (1989). What you always wanted to know about Datalog (and never dared to ask). IEEE Transactions on Knowledge and Data Engineering, 1(1), 146-166. doi:10.1109/69.43410Chen, M., Mao, S., & Liu, Y. (2014). Big Data: A Survey. Mobile Networks and Applications, 19(2), 171-209. doi:10.1007/s11036-013-0489-0Chitil, O., & Davie, T. (2008). Comprehending finite maps for algorithmic debugging of higher-order functional programs. Proceedings of the 10th international ACM SIGPLAN symposium on Principles and practice of declarative programming - PPDP ’08. doi:10.1145/1389449.1389475Chitil, O., Faddegon, M., & Runciman, C. (2016). A Lightweight Hat. Proceedings of the 28th Symposium on the Implementation and Application of Functional Programming Languages - IFL 2016. doi:10.1145/3064899.3064904O. Chitil C. Runciman and M. Wallace. 2001. Freja Hat and Hood—A Comparative Evaluation of Three Systems for Tracing and Debugging Lazy Functional Programs. Springer Berlin 176--193. O. Chitil C. Runciman and M. Wallace. 2001. Freja Hat and Hood—A Comparative Evaluation of Three Systems for Tracing and Debugging Lazy Functional Programs. Springer Berlin 176--193.O. Chitil C. Runciman and Malcolm Wallace. 2003. Transforming Haskell for Tracing. Springer-Verlag Berlin 165--181. DOI:http://dx.doi.org/10.1007/3-540-44854-3_11 10.1007/3-540-44854-3_11 O. Chitil C. Runciman and Malcolm Wallace. 2003. Transforming Haskell for Tracing. Springer-Verlag Berlin 165--181. DOI:http://dx.doi.org/10.1007/3-540-44854-3_11 10.1007/3-540-44854-3_11Minh Ngoc Dinh, Abramson, D., & Chao Jin. (2014). Scalable Relative Debugging. IEEE Transactions on Parallel and Distributed Systems, 25(3), 740-749. doi:10.1109/tpds.2013.86Faddegon, M., & Chitil, O. (2015). Algorithmic debugging of real-world haskell programs: deriving dependencies from the cost centre stack. ACM SIGPLAN Notices, 50(6), 33-42. doi:10.1145/2813885.2737985Faddegon, M., & Chitil, O. (2016). Lightweight computation tree tracing for lazy functional languages. Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation - PLDI 2016. doi:10.1145/2908080.2908104Ferrand, G. (1987). Error diagnosis in logic programming an adaptation of E.Y. Shapiro’s method. The Journal of Logic Programming, 4(3), 177-198. doi:10.1016/0743-1066(87)90001-xFritzson, P., Shahmehri, N., Kamkar, M., & Gyimothy, T. (1992). Generalized algorithmic debugging and testing. ACM Letters on Programming Languages and Systems, 1(4), 303-322. doi:10.1145/161494.161498Fromherz, M. P. J. (s. f.). Towards declarative debugging of concurrent constraint programs. Lecture Notes in Computer Science, 88-100. doi:10.1007/bfb0019403Harman, M., & Hierons, R. (2001). An overview of program slicing. Software Focus, 2(3), 85-92. doi:10.1002/swf.41F. Henderson T. Conway Z. Somogyi D. Jeffery P. Schachte S. Taylor C. Speirs T. Dowd R. Becket M. Brown and P. Wang. 2014. The Mercury Language Reference Manual (Version 14.01.1). The University of Melbourne. F. Henderson T. Conway Z. Somogyi D. Jeffery P. Schachte S. Taylor C. Speirs T. Dowd R. Becket M. Brown and P. Wang. 2014. The Mercury Language Reference Manual (Version 14.01.1). The University of Melbourne.C. Hermanns and H. Kuchen. 2013. Hybrid Debugging of Java Programs. Springer-Verlag Berlin 91--107. DOI:http://dx.doi.org/10.1007/978-3-642-36177-7_6 10.1007/978-3-642-36177-7_6 C. Hermanns and H. Kuchen. 2013. Hybrid Debugging of Java Programs. Springer-Verlag Berlin 91--107. DOI:http://dx.doi.org/10.1007/978-3-642-36177-7_6 10.1007/978-3-642-36177-7_6Hirunkitti, V., & Hogger, C. J. (s. f.). A generalised query minimisation for program debugging. Lecture Notes in Computer Science, 153-170. doi:10.1007/bfb0019407Hughes, J. (2010). Software Testing with QuickCheck. Lecture Notes in Computer Science, 183-223. doi:10.1007/978-3-642-17685-2_6G. Hutton. 2016. Programming in Haskell. Cambridge University Press Cambridge UK. G. Hutton. 2016. Programming in Haskell. Cambridge University Press Cambridge UK.Insa, D., & Silva, J. (2010). An algorithmic debugger for Java. 2010 IEEE International Conference on Software Maintenance. doi:10.1109/icsm.2010.5609661Insa, D., & Silva, J. (2011). Optimal Divide and Query. Lecture Notes in Computer Science, 224-238. doi:10.1007/978-3-642-24769-9_17Insa, D., & Silva, J. (2011). An optimal strategy for algorithmic debugging. 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011). doi:10.1109/ase.2011.6100055D. Insa and J. Silva. 2011c. Scaling Up Algorithmic Debugging with Virtual Execution Trees. Springer-Verlag Berlin 149--163. DOI:http://dx.doi.org/10.1007/978-3-642-20551-4_10 10.1007/978-3-642-20551-4_10 D. Insa and J. Silva. 2011c. Scaling Up Algorithmic Debugging with Virtual Execution Trees. Springer-Verlag Berlin 149--163. DOI:http://dx.doi.org/10.1007/978-3-642-20551-4_10 10.1007/978-3-642-20551-4_10D. Insa and J. Silva. 2015a. Automatic transformation of iterative loops into recursive methods. Information 8 Software Technology 58 (2015) 95--109. DOI:http://dx.doi.org/10.1016/j.infsof.2014.10.001 10.1016/j.infsof.2014.10.001 D. Insa and J. Silva. 2015a. Automatic transformation of iterative loops into recursive methods. Information 8 Software Technology 58 (2015) 95--109. DOI:http://dx.doi.org/10.1016/j.infsof.2014.10.001 10.1016/j.infsof.2014.10.001Insa, D., & Silva, J. (2015). A Generalized Model for Algorithmic Debugging. Lecture Notes in Computer Science, 261-276. doi:10.1007/978-3-319-27436-2_16Insa, D., Silva, J., & Riesco, A. (2013). Speeding Up Algorithmic Debugging Using Balanced Execution Trees. Lecture Notes in Computer Science, 133-151. doi:10.1007/978-3-642-38916-0_8Insa, D., Silva, J., & Tomás, C. (2013). Enhancing Declarative Debugging with Loop Expansion and Tree Compression. Lecture Notes in Computer Science, 71-88. doi:10.1007/978-3-642-38197-3_6K. Jensen and N. Wirth. 1974. PASCAL User Manual and Report. Springer-Verlag Berlin. 10.1007/978-3-662-21554-8 K. Jensen and N. Wirth. 1974. PASCAL User Manual and Report. Springer-Verlag Berlin. 10.1007/978-3-662-21554-8Jia, Y., & Harman, M. (2011). An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering, 37(5), 649-678. doi:10.1109/tse.2010.62Kamkar, M., Shahmehri, N., & Fritzson, P. (s. f.). Bug localization by algorithmic debugging and program slicing. Lecture Notes in Computer Science, 60-74. doi:10.1007/bfb0024176S. Köhler B. Ludäscher and Y. Smaragdakis. 2012. Declarative Datalog Debugging for Mere Mortals. Springer-Verlag Berlin 111--122. S. Köhler B. Ludäscher and Y. Smaragdakis. 2012. Declarative Datalog Debugging for Mere Mortals. Springer-Verlag Berlin 111--122.Kouh, H.-J., & Yoo, W.-H. (2003). The Efficient Debugging System for Locating Logical Errors in Java Programs. Lecture Notes in Computer Science, 684-693. doi:10.1007/3-540-44839-x_72Benzmüller, C., & Miller, D. (2014). Automation of Higher-Order Logic. Handbook of the History of Logic, 215-254. doi:10.1016/b978-0-444-51624-4.50005-8Kowalski, R., & Kuehner, D. (1971). Linear resolution with selection function. Artificial Intelligence, 2(3-4), 227-260. doi:10.1016/0004-3702(71)90012-9K. Kuchcinski W. Drabent and J. Maluszynski. 1993. Automatic Diagnosis of VLSI Digital Circuits Using Algorithmic Debugging. Springer-Verlag Berlin 350--367. DOI:http://dx.doi.org/10.1007/BFb0019419 10.1007/BFb0019419 K. Kuchcinski W. Drabent and J. Maluszynski. 1993. Automatic Diagnosis of VLSI Digital Circuits Using Algorithmic Debugging. Springer-Verlag Berlin 350--367. DOI:http://dx.doi.org/10.1007/BFb0019419 10.1007/BFb0019419S. Liang. 1999. Java Native Interface: Programmer’s Guide and Reference (1st ed.). Addison-Wesley Longman Publishing Co. Inc. Boston MA. S. Liang. 1999. Java Native Interface: Programmer’s Guide and Reference (1st ed.). Addison-Wesley Longman Publishing Co. Inc. Boston MA.Lloyd, J. W. (1987). Declarative error diagnosis. New Generation Computing, 5(2), 133-154. doi:10.1007/bf03037396J. W. Lloyd. 1987b. Foundations of Logic Programming (2nd ed.). Springer-Verlag Berlin. 10.1007/978-3-642-83189-8 J. W. Lloyd. 1987b. Foundations of Logic Programming (2nd ed.). Springer-Verlag Berlin. 10.1007/978-3-642-83189-8W. Lux. 2006. Münster Curry User’s guide (Release 0.9.10 of May 10 2006). Retrieved from http://danae.uni-muenster.de/∼lux/curry/user.pdf. W. Lux. 2006. Münster Curry User’s guide (Release 0.9.10 of May 10 2006). Retrieved from http://danae.uni-muenster.de/∼lux/curry/user.pdf.Lux, W. (2008). Declarative Debugging Meets the World. Electronic Notes in Theoretical Computer Science, 216, 65-77. doi:10.1016/j.entcs.2008.06.034I. MacLarty. 2005. Practical Declarative Debugging of Mercury Programs. Ph.D. Dissertation. Department of Computer Science and Software Engineering The University of Melbourne. I. MacLarty. 2005. Practical Declarative Debugging of Mercury Programs. Ph.D. Dissertation. Department of Computer Science and Software Engineering The University of Melbourne.Naganuma, J., Ogura, T., & Hoshino, T. (s. f.). High-level design validation using algorithmic debugging. Proceedings of European Design and Test Conference EDAC-ETC-EUROASIC. doi:10.1109/edtc.1994.326833Naish, L. (1992). Declarative diagnosis of missing answers. New Generation Computing, 10(3), 255-285. doi:10.1007/bf03037939H. Nilsson. 1998. Declarative Debugging for Lazy Functional Languages. Ph.D. Dissertation. Linköping Sweden. H. Nilsson. 1998. Declarative Debugging for Lazy Functional Languages. Ph.D. Dissertation. Linköping Sweden.NILSSON, H. (2001). How to look busy while being as lazy as ever: the Implementation of a lazy functional debugger. Journal of Functional Programming, 11(6), 629-671. doi:10.1017/s095679680100418xNilsson, H., & Fritzson, P. (s. f.). Algorithmic debugging for lazy functional languages. Lecture Notes in Computer Science, 385-399. doi:10.1007/3-540-55844-6_149Nilsson, H., & Fritzson, P. (1994). Algorithmic debugging for lazy functional languages. Journal of Functional Programming, 4(3), 337-369. doi:10.1017/s095679680000109xNilsson, H., & Sparud, J. (1997). Automated Software Engineering, 4(2), 121-150. doi:10.1023/a:1008681016679Ostrand, T. J., & Balcer, M. J. (1988). The category-partition method for specifying and generating fuctional tests. Communications of the ACM, 31(6), 676-686. doi:10.1145/62959.62964Pereira, L. M. (1986). Rational debugging in logic programming. Third International Conference on Logic Programming, 203-210. doi:10.1007/3-540-16492-8_76B. Pope. 2006. A Declarative Debugger for Haskell. Ph.D. Dissertation. The University of Melbourne Australia. B. Pope. 2006. A Declarative Debugger for Haskell. Ph.D. Dissertation. The University of Melbourne Australia.Ramakrishnan, R., & Ullman, J. D. (1995). A survey of deductive database systems. The Journal of Logic Programming, 23(2), 125-149. doi:10.1016/0743-1066(94)00039-9Riesco, A., Verdejo, A., Martí-Oliet, N., & Caballero, R. (2012). Declarative debugging of rewriting logic specifications. The Journal of Logic and Algebraic Programming, 81(7-8), 851-897. doi:10.1016/j.jlap.2011.06.004DeRose, L., Gontarek, A., Vose, A., Moench, R., Abramson, D., Dinh, M. N., & Jin, C. (2015). Relative debugging for a highly parallel hybrid computer system. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC ’15. doi:10.1145/2807591.2807605Runeson, P. (2006). A survey of unit testing practices. IEEE Software, 23(4), 22-29. doi:10.1109/ms.2006.91Russo, F., & Sancassani, M. (1992). A declarative debugging environment for DATALOG. Lecture Notes in Computer Science, 433-441. doi:10.1007/3-540-55460-2_32E. Y. Shapiro. 1982a. Algorithmic Program Debugging. MIT Press Cambridge MA. E. Y. Shapiro. 1982a. Algorithmic Program Debugging. MIT Press Cambridge MA.Shapiro, E. Y. (1982). Algorithmic program diagnosis. Proceedings of the 9th ACM SIGPLAN-SIGACT symposium on Principles of programming languages - POPL ’82. doi:10.1145/582153.582185Shmueli, O., & Tsur, S. (1991). Logical diagnosis ofLDL programs. New Generation Computing, 9(3-4), 277-303. doi:10.1007/bf03037166Silva, J. (s. f.). A Comparative Study of Algorithmic Debugging Strategies. Lecture Notes in Computer Science, 143-159. doi:10.1007/978-3-540-71410-1_11Silva, J. (2011). A survey on algorithmic debugging strategies. Advances in Engineering Software, 42(11), 976-991. doi:10.1016/j.advengsoft.2011.05.024Silva, J., & Chitil, O. (2006). Combining algorithmic debugging and program slicing. Proceedings of the 8th ACM SIGPLAN symposium on Principles and practice of declarative programming - PPDP ’06. doi:10.1145/1140335.1140355J. A. Silva E. R. Faria R. C. Barros E. R. Hruschka A. C. P. L. F. de Carvalho and J. Gama. 2013. Data stream clustering: A survey. Comput. Surv. 46 1 Article 13 (July 2013) 31 pages.DOI:http://dx.doi.org/10.1145/2522968.2522981 10.1145/2522968.2522981 J. A. Silva E. R. Faria R. C. Barros E. R. Hruschka A. C. P. L. F. de Carvalho and J. Gama. 2013. Data stream clustering: A survey. Comput. Surv. 46 1 Article 13 (July 2013) 31 pages.DOI:http://dx.doi.org/10.1145/2522968.2522981 10.1145/2522968.2522981SOSIČ, R., & ABRAMSON, D. (1997). Guard: A Relative Debugger. Software: Practice and Experience, 27(2), 185-206. doi:10.1002/(sici)1097-024x(199702)27:23.0.co;2-dL. Sterling and E. Shapiro. 1986. The Art of Prolog: Advanced Programming Techniques. The MIT Press Cambridge MA. L. Sterling and E. Shapiro. 1986. The Art of Prolog: Advanced Programming Techniques. The MIT Press Cambridge MA.P. Kambam Sugavanam. 2013. Debugging Framework for Attribute Grammars. Ph.D. Dissertation. University of Minnesota. P. Kambam Sugavanam. 2013. Debugging Framework for Attribute Grammars. Ph.D. Dissertation. University of Minnesota.Tamarit, S., Riesco, A., Martin-Martin, E., & Caballero, R. (2016). Debugging Meets Testing in Erlang. Lecture Notes in Computer Science, 171-180. doi:10.1007/978-3-319-41135-4_10A. Tessier and G. Ferrand. 2000. Declarative diagnosis in the CLP scheme. In Analysis and Visualization Tools for Constraint Programming: Constraint Debugging Pierre Deransart Manuel V. Hermenegildo and Jan Maluszynski (Eds.). Springer-Verlag Berlin 151--174. 10.1007/10722311_6 A. Tessier and G. Ferrand. 2000. Declarative diagnosis in the CLP scheme. In Analysis and Visualization Tools for Constraint Programming: Constraint Debugging Pierre Deransart Manuel V. Hermenegildo and Jan Maluszynski (Eds.). Springer-Verlag Berlin 151--174. 10.1007/10722311_6Zinn, C. (2013). Algorithmic Debugging for Intelligent Tutoring: How to Use Multiple Models and Improve Diagnosis. Lecture Notes in Computer Science, 272-283. doi:10.1007/978-3-642-40942-4_24Zinn, C. (2014). Algorithmic Debugging and Literate Programming to Generate Feedback in Intelligent Tutoring Systems. KI 2014: Advances in Artificial Intelligence, 37-48. doi:10.1007/978-3-319-11206-0_

Optimization Techniques for Algorithmic Debugging

[EN] Nowadays, undetected programming bugs produce a waste of billions of dollars per year to private and public companies and institutions. In spite of this, no significant advances in the debugging area that help developers along the software development process have been achieved yet. In fact, the same debugging techniques that were used 20 years ago are still being used now. Along the time, some alternatives have appeared, but there still is a long way for them to be useful enough to get into the software development process. One of them is algorithmic debugging, which abstracts the information the user has to investigate to debug the program, allowing them to focus on what is happening instead of how it is happening. This abstraction comes at a price: the granularity level of the bugs that can be detected allows for isolating wrongly implemented functions, but which part of them contains the bug cannot be found out yet. This thesis focusses on improving algorithmic debugging in many aspects. Concretely, the main aims of this thesis are to reduce the time the user needs to detect a programming bug as well as to provide the user with more detailed information about where the bug is located. To achieve these goals, some techniques have been developed to start the debugging sessions as soon as possible, to reduce the number of questions the user is going to be asked about, and to augment the granularity level of those bugs that algorithmic debugging can detect, allowing the debugger in this way to keep looking for bugs even inside functions. As a result of this thesis, three completely new techniques have been defined, an already existent technique has been improved, and two new algorithmic debugging search strategies have been defined that improve the already existent ones. Besides these theoretical results, a fully functional algorithmic debugger has been implemented that contains and supports all these techniques and strategies. This debugger is written in Java, and it debugs Java code. The election of this language is justified because it is currently one of the most widely extended and used languages. Also because it contains an interesting combination of unsolved challenges for algorithmic debugging. To further increase its usability, the debugger has been later adapted as an Eclipse plugin, so it could be used by a wider number of users. These two debuggers are publicly available, so any interested person can access them and continue with the research if they wish so.[ES] Hoy en día, los errores no detectados de programación suponen un gasto de miles de millones al año para las empresas e instituciones públicas y privadas. A pesar de esto, no ha habido ningún avance significativo en el área de la depuración que ayude a los desarrolladores durante la fase de desarrollo de software. De hecho, las mismas técnicas de depuración que se utilizaban hace 20 años se siguen utilizando ahora. A lo largo del tiempo, han surgido algunas alternativas, pero todavía queda un largo camino para que estas sean lo suficientemente útiles como para abrirse camino en el proceso de desarrollo de software. Una de ellas es la depuración algorítmica, la cual abstrae la información que el programador debe investigar para depurar el programa, permitiéndole de este modo centrarse en el qué está ocurriendo en vez de en el cómo. Esta abstracción tiene un coste: el nivel de granularidad de los errores que pueden detectarse nos permite como máximo aislar funciones mal implementadas, pero no averiguar qué parte de estas contiene el error. Esta tesis se centra en mejorar la depuración algorítmica en muchos aspectos. Concretamente, los principales objetivos de esta tesis son reducir el tiempo que el usuario necesita para detectar un error de programación así como proporcionar información más detallada de dónde se encuentra el error. Para conseguir estos objetivos, se han desarrollado técnicas para iniciar las sesiones de depuración lo antes posible, reducir el número de preguntas que se le van a realizar al usuario, y aumentar el nivel de granularidad de los errores que la depuración algorítmica puede detectar, permitiendo así seguir buscando el error incluso dentro de las funciones. Como resultado de esta tesis, se han definido tres técnicas completamente nuevas, se ha mejorado una técnica ya existente, y se han definido dos nuevas estrategias de depuración algorítmica que mejoran las previamente existentes. Además de los resultados teóricos, también se ha desarrollado un depurador algorítmico completamente funcional que contiene y respalda todas estas técnicas y estrategias. Este depurador está escrito en Java y depura código Java. La elección de este lenguaje se justifica debido a que es uno de los lenguajes más ampliamente extendidos y usados actualmente. También debido a que contiene una combinación interesante de retos todavía sin resolver para la depuración algorítmica. Para aumentar todavía más su usabilidad, el depurador ha sido posteriormente adaptado como un plugin de Eclipse, de tal manera que pudiese ser usado por un número más amplio de usuarios. Estos dos depuradores están públicamente disponibles para que cualquier persona interesada pueda acceder a ellos y continuar con la investigación si así lo deseara.[CA] Hui en dia, els errors no detectats de programació suposen una despesa de milers de milions a l'any per a les empreses i institucions públiques i privades. Tot i això, no hi ha hagut cap avanç significatiu en l'àrea de la depuració que ajude als desenvolupadors durant la fase de desenvolupament de programari. De fet, les mateixes tècniques de depuració que s'utilitzaven fa 20 anys es continuen utilitzant ara. Al llarg del temps, han sorgit algunes alternatives, però encara queda un llarg camí perquè estes siguen prou útils com per a obrir-se camí en el procés de desenvolupament de programari. Una d'elles és la depuració algorítmica, la qual abstrau la informació que el programador ha d'investigar per a depurar el programa, permetent-li d'esta manera centrar-se en el què està ocorrent en compte de en el com. Esta abstracció té un cost: el nivell de granularitat dels errors que poden detectar-se ens permet com a màxim aïllar funcions mal implementades, però no esbrinar quina part d'estes conté l'error. Esta tesi es centra a millorar la depuració algorítmica en molts aspectes. Concretament, els principals objectius d'esta tesi són reduir el temps que l'usuari necessita per a detectar un error de programació així com proporcionar informació més detallada d'on es troba l'error. Per a aconseguir estos objectius, s'han desenvolupat tècniques per a iniciar les sessions de depuració com més prompte millor, reduir el nombre de preguntes que se li formularan a l'usuari, i augmentar el nivell de granularitat dels errors que la depuració algorítmica pot detectar, permetent així continuar buscant l'error inclús dins de les funcions. Com resultat d'esta tesi, s'han definit tres tècniques completament noves, s'ha millorat una tècnica ja existent, i s'han definit dos noves estratègies de depuració algorítmica que milloren les prèviament existents. A més dels resultats teòrics, també s'ha desenvolupat un depurador algorítmic completament funcional que conté i protegix totes estes tècniques i estratègies. Este depurador està escrit en Java i depura codi Java. L'elecció d'este llenguatge es justifica pel fet que és un dels llenguatges més àmpliament estesos i usats actualment. També pel fet que conté una combinació interessant de reptes encara sense resoldre per a la depuració algorítmica. Per a augmentar encara més la seua usabilitat, el depurador ha sigut posteriorment adaptat com un plugin d'Eclipse, de tal manera que poguera ser usat per un nombre més ampli d'usuaris. Estos dos depuradors estan públicament disponibles perquè qualsevol persona interessada puga accedir a ells i continuar amb la investigació si així ho desitjara.Insa Cabrera, D. (2016). Optimization Techniques for Algorithmic Debugging [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/68506TESISPremios Extraordinarios de tesis doctorale

Innovative Techniques for Testing and Diagnosing SoCs

We rely upon the continued functioning of many electronic devices for our everyday welfare, usually embedding integrated circuits that are becoming even cheaper and smaller with improved features. Nowadays, microelectronics can integrate a working computer with CPU, memories, and even GPUs on a single die, namely System-On-Chip (SoC). SoCs are also employed on automotive safety-critical applications, but need to be tested thoroughly to comply with reliability standards, in particular the ISO26262 functional safety for road vehicles. The goal of this PhD. thesis is to improve SoC reliability by proposing innovative techniques for testing and diagnosing its internal modules: CPUs, memories, peripherals, and GPUs. The proposed approaches in the sequence appearing in this thesis are described as follows: 1. Embedded Memory Diagnosis: Memories are dense and complex circuits which are susceptible to design and manufacturing errors. Hence, it is important to understand the fault occurrence in the memory array. In practice, the logical and physical array representation differs due to an optimized design which adds enhancements to the device, namely scrambling. This part proposes an accurate memory diagnosis by showing the efforts of a software tool able to analyze test results, unscramble the memory array, map failing syndromes to cell locations, elaborate cumulative analysis, and elaborate a final fault model hypothesis. Several SRAM memory failing syndromes were analyzed as case studies gathered on an industrial automotive 32-bit SoC developed by STMicroelectronics. The tool displayed defects virtually, and results were confirmed by real photos taken from a microscope. 2. Functional Test Pattern Generation: The key for a successful test is the pattern applied to the device. They can be structural or functional; the former usually benefits from embedded test modules targeting manufacturing errors and is only effective before shipping the component to the client. The latter, on the other hand, can be applied during mission minimally impacting on performance but is penalized due to high generation time. However, functional test patterns may benefit for having different goals in functional mission mode. Part III of this PhD thesis proposes three different functional test pattern generation methods for CPU cores embedded in SoCs, targeting different test purposes, described as follows: a. Functional Stress Patterns: Are suitable for optimizing functional stress during I Operational-life Tests and Burn-in Screening for an optimal device reliability characterization b. Functional Power Hungry Patterns: Are suitable for determining functional peak power for strictly limiting the power of structural patterns during manufacturing tests, thus reducing premature device over-kill while delivering high test coverage c. Software-Based Self-Test Patterns: Combines the potentiality of structural patterns with functional ones, allowing its execution periodically during mission. In addition, an external hardware communicating with a devised SBST was proposed. It helps increasing in 3% the fault coverage by testing critical Hardly Functionally Testable Faults not covered by conventional SBST patterns. An automatic functional test pattern generation exploiting an evolutionary algorithm maximizing metrics related to stress, power, and fault coverage was employed in the above-mentioned approaches to quickly generate the desired patterns. The approaches were evaluated on two industrial cases developed by STMicroelectronics; 8051-based and a 32-bit Power Architecture SoCs. Results show that generation time was reduced upto 75% in comparison to older methodologies while increasing significantly the desired metrics. 3. Fault Injection in GPGPU: Fault injection mechanisms in semiconductor devices are suitable for generating structural patterns, testing and activating mitigation techniques, and validating robust hardware and software applications. GPGPUs are known for fast parallel computation used in high performance computing and advanced driver assistance where reliability is the key point. Moreover, GPGPU manufacturers do not provide design description code due to content secrecy. Therefore, commercial fault injectors using the GPGPU model is unfeasible, making radiation tests the only resource available, but are costly. In the last part of this thesis, we propose a software implemented fault injector able to inject bit-flip in memory elements of a real GPGPU. It exploits a software debugger tool and combines the C-CUDA grammar to wisely determine fault spots and apply bit-flip operations in program variables. The goal is to validate robust parallel algorithms by studying fault propagation or activating redundancy mechanisms they possibly embed. The effectiveness of the tool was evaluated on two robust applications: redundant parallel matrix multiplication and floating point Fast Fourier Transform