Statistical Assertions for Validating Patterns and Finding Bugs in Quantum Programs

In support of the growing interest in quantum computing experimentation, programmers need new tools to write quantum algorithms as program code. Compared to debugging classical programs, debugging quantum programs is difficult because programmers have limited ability to probe the internal states of quantum programs; those states are difficult to interpret even when observations exist; and programmers do not yet have guidelines for what to check for when building quantum programs. In this work, we present quantum program assertions based on statistical tests on classical observations. These allow programmers to decide if a quantum program state matches its expected value in one of classical, superposition, or entangled types of states. We extend an existing quantum programming language with the ability to specify quantum assertions, which our tool then checks in a quantum program simulator. We use these assertions to debug three benchmark quantum programs in factoring, search, and chemistry. We share what types of bugs are possible, and lay out a strategy for using quantum programming patterns to place assertions and prevent bugs.Comment: In The 46th Annual International Symposium on Computer Architecture (ISCA '19). arXiv admin note: text overlap with arXiv:1811.0544

Quantum Symbolic Execution

With advances in quantum computing, researchers can now write and run many quantum programs. However, there is still a lack of effective methods for debugging quantum programs. In this paper, quantum symbolic execution (QSE) is proposed to generate test cases, which helps to finding bugs in quantum programs. The main idea of quantum symbolic execution is to find the suitable test cases from all possible ones (i.e. test case space). It is different from the way of classical symbol execution, which gets test cases by calculating instead of searching. QSE utilizes quantum superposition and parallelism to store the test case space with only a few qubits. According to the conditional statements in the debugged program, the test case space is continuously divided into subsets, subsubsets and so on. Elements in the same subset are suitable test cases that can test the corresponding branch in the code to be tested. QSE not only provides a possible way to debug quantum programs, but also avoids the difficult problem of solving constraints in classical symbolic execution

Identifying Flakiness in Quantum Programs

In recent years, software engineers have explored ways to assist quantum software programmers. Our goal in this paper is to continue this exploration and see if quantum software programmers deal with some problems plaguing classical programs. Specifically, we examine whether intermittently failing tests, i.e., flaky tests, affect quantum software development. To explore flakiness, we conduct a preliminary analysis of 14 quantum software repositories. Then, we identify flaky tests and categorize their causes and methods of fixing them. We find flaky tests in 12 out of 14 quantum software repositories. In these 12 repositories, the lower boundary of the percentage of issues related to flaky tests ranges between 0.26% and 1.85% per repository. We identify 46 distinct flaky test reports with 8 groups of causes and 7 common solutions. Further, we notice that quantum programmers are not using some of the recent flaky test countermeasures developed by software engineers. This work may interest practitioners, as it provides useful insight into the resolution of flaky tests in quantum programs. Researchers may also find the paper helpful as it offers quantitative data on flaky tests in quantum software and points to new research opportunities.Comment: 7 pages, 16 listings, 2 tables, accepted at ESEM 2023: The 17th ACM/IEEE International Symposium on Empirical Software Engineering and Measuremen

Testing Multi-Subroutine Quantum Programs: From Unit Testing to Integration Testing

Quantum computing has emerged as a promising field with the potential to revolutionize various domains by harnessing the principles of quantum mechanics. As quantum hardware and algorithms continue to advance, the development of high-quality quantum software has become crucial. However, testing quantum programs poses unique challenges due to the distinctive characteristics of quantum systems and the complexity of multi-subroutine programs. In this paper, we address the specific testing requirements of multi-subroutine quantum programs. We begin by investigating critical properties through a survey of existing quantum libraries, providing insights into the challenges associated with testing these programs. Building upon this understanding, we present a systematic testing process tailored to the intricacies of quantum programming. The process covers unit testing and integration testing, with a focus on aspects such as IO analysis, quantum relation checking, structural testing, behavior testing, and test case generation. We also introduce novel testing principles and criteria to guide the testing process. To evaluate our proposed approach, we conduct comprehensive testing on typical quantum subroutines, including diverse mutations and randomized inputs. The analysis of failures provides valuable insights into the effectiveness of our testing methodology. Additionally, we present case studies on representative multi-subroutine quantum programs, demonstrating the practical application and effectiveness of our proposed testing processes, principles, and criteria.Comment: 53 page

Noise-Aware Quantum Software Testing

Quantum Computing (QC) promises computational speedup over classic computing for solving some complex problems. However, noise exists in current and near-term quantum computers. Quantum software testing (for gaining confidence in quantum software's correctness) is inevitably impacted by noise, to the extent that it is impossible to know if a test case failed due to noise or real faults. Existing testing techniques test quantum programs without considering noise, i.e., by executing tests on ideal quantum computer simulators. Consequently, they are not directly applicable to testing quantum software on real QC hardware or noisy simulators. To this end, we propose a noise-aware approach (named QOIN) to alleviate the noise effect on test results of quantum programs. QOIN employs machine learning techniques (e.g., transfer learning) to learn the noise effect of a quantum computer and filter it from a quantum program's outputs. Such filtered outputs are then used as the input to perform test case assessments (determining the passing or failing of a test case execution against a test oracle). We evaluated QOIN on IBM's 23 noise models with nine real-world quantum programs and 1000 artificial quantum programs. We also generated faulty versions of these programs to check if a failing test case execution can be determined under noise. Results show that QOIN can reduce the noise effect by more than $80\%$. To check QOIN's effectiveness for quantum software testing, we used an existing test oracle for quantum software testing. The results showed that the F1-score of the test oracle was improved on average by $82\%$ for six real-world programs and by $75\%$ for 800 artificial programs, demonstrating that QOIN can effectively learn noise patterns and enable noise-aware quantum software testing

The Quantum Frontier of Software Engineering: A Systematic Mapping Study

Context. Quantum computing is becoming a reality, and quantum software engineering (QSE) is emerging as a new discipline to enable developers to design and develop quantum programs. Objective. This paper presents a systematic mapping study of the current state of QSE research, aiming to identify the most investigated topics, the types and number of studies, the main reported results, and the most studied quantum computing tools/frameworks. Additionally, the study aims to explore the research community's interest in QSE, how it has evolved, and any prior contributions to the discipline before its formal introduction through the Talavera Manifesto. Method. We searched for relevant articles in several databases and applied inclusion and exclusion criteria to select the most relevant studies. After evaluating the quality of the selected resources, we extracted relevant data from the primary studies and analyzed them. Results. We found that QSE research has primarily focused on software testing, with little attention given to other topics, such as software engineering management. The most commonly studied technology for techniques and tools is Qiskit, although, in most studies, either multiple or none specific technologies were employed. The researchers most interested in QSE are interconnected through direct collaborations, and several strong collaboration clusters have been identified. Most articles in QSE have been published in non-thematic venues, with a preference for conferences. Conclusions. The study's implications are providing a centralized source of information for researchers and practitioners in the field, facilitating knowledge transfer, and contributing to the advancement and growth of QSE