P4Testgen: An Extensible Test Oracle For P4

We present P4Testgen, a test oracle for the P4-16 language that supports automatic generation of packet tests for any P4-programmable device. Given a P4 program and sufficient time, P4Testgen generates tests that cover every reachable statement in the input program. Each generated test consists of an input packet, control-plane configuration, and output packet(s), and can be executed in software or on hardware. Unlike prior work, P4Testgen is open source and extensible, making it a general resource for the community. P4Testgen not only covers the full P4-16 language specification, it also supports modeling the semantics of an entire packet-processing pipeline, including target-specific behaviors-i.e., whole-program semantics. Handling aspects of packet processing that lie outside of the official specification is critical for supporting real-world targets (e.g., switches, NICs, end host stacks). In addition, P4Testgen uses taint tracking and concolic execution to model complex externs (e.g., checksums and hash functions) that have been omitted by other tools, and ensures the generated tests are correct and deterministic. We have instantiated P4Testgen to build test oracles for the V1model, eBPF, and the Tofino (TNA and T2NA) architectures; each of these extensions only required effort commensurate with the complexity of the target. We validated the tests generated by P4Testgen by running them across the entire P4C program test suite as well as the Tofino programs supplied with Intel's P4 Studio. In just a few months using the tool, we discovered and confirmed 25 bugs in the mature, production toolchains for BMv2 and Tofino, and are conducting ongoing investigations into further faults uncovered by P4Testgen

SIGCOMM 2023 Artifact for P4Testgen: An Extensible Test Oracle For P4

<p>This is the artifact associated with the paper "P4Testgen: An Extensible Test Oracle For P4". Abstract:</p><blockquote><p>We present P4Testgen, a test oracle for the P4-16 language. P4Testgen supports automatic test generation for any P4 target and is designed to be extensible to many P4 targets. It models the complete semantics of the target's packet-processing pipeline including the P4 language, architectures and externs, and target-specific extensions. To handle non-deterministic behaviors and complex externs (e.g., checksums and hash functions), P4Testgen uses taint tracking and concolic execution. It also provides path selection strategies that reduce the number of tests required to achieve full coverage."We have instantiated P4Testgen for the V1model, eBPF, PNA, and Tofino P4 architectures. Each extension required effort commensurate with the complexity of the target. We validated the tests generated by P4Testgen by running them across the entire P4C test suite as well as the programs supplied with the Tofino P4 Studio. Using the tool, we have also confirmed 25 bugs in mature, production toolchains for BMv2 and Tofino.</p></blockquote><p>The zip file contains the paper, source code, coverage data, and a docker image with an executable. The P4Testgen code can be found in <i>p4c/backends/p4tools/modules/testgen</i>.</p><p>You can load the docker image with</p><blockquote><p>docker load < p4testgen_docker_image_aec_sigcomm23.tar.gz</p></blockquote><p>and run with</p><blockquote><p>docker run --rm -it -v.:/p4c --entrypoint bash p4lang/p4c:p4testgen_aec_sigcomm23</p></blockquote><p>A p4testgen binary will be available inside the image, which you can run on the p4 programs contained in <i>p4c/testdata/p4_16_samples</i>.</p><p>For further instructions on how to build, please see https://p4.org/projects/p4testgen or <i>p4c/backends/p4tools/modules/testgen/README.md</i> inside the archive.</p><h3>Reproducing artifacts</h3><p>These instructions assume that <i>p4testgen_aec_sigcomm23.zip</i> has been extracted and we are operating inside the extracted folder.</p><h4>Table 7 and Figure 8</h4><p>Run <i>p4c/backends/p4tools/modules/testgen/benchmarks/plots.py</i>. The numbers are sourced from a manual evaluation process. <i>pins_middleblock_stripped.p4</i> and <i>pins_middleblock_mod.p4</i> are part of the supplied artifacts and can be found in the p4 folder. The commands to reproduce are as follows:</p><p>"None":</p><blockquote><p>time p4testgen --target bmv2 --arch v1model --test-backend PTF --max-tests 0 --out-dir results --seed 1 --track-coverage STATEMENTS --packet-size-range 0:12000 p4/pins_middleblock_stripped.p4</p></blockquote><p>"1500B pkt":</p><blockquote><p>time p4testgen --target bmv2 --arch v1model --test-backend PTF --max-tests 0 --out-dir results --seed 1 --track-coverage STATEMENTS --packet-size-range 12000:12000 p4/pins_middleblock_mod.p4</p></blockquote><p>"P4-constraints":</p><blockquote><p>time p4testgen --target bmv2 --arch v1model --test-backend PTF --max-tests 0 --out-dir results --seed 1 --track-coverage STATEMENTS --packet-size-range 0:12000 p4/pins_middleblock_mod.p4</p></blockquote><p>"P4-constraints + 1500B pkt":</p><blockquote><p>time p4testgen --target bmv2 --arch v1model --test-backend PTF --max-tests 0 --out-dir results --seed 1 --track-coverage STATEMENTS --packet-size-range 12000:12000 p4/pins_middleblock_mod.p4</p></blockquote><p>"P4-constraints + 1500B IPv4 pkt":</p><blockquote><p>time p4testgen --target bmv2 --arch v1model --test-backend PTF --max-tests 0 --out-dir results --seed 1 --track-coverage STATEMENTS --packet-size-range 12000:12000 -DTESTGEN_ASSUME_IP p4/pins_middleblock_mod.p4</p></blockquote><p>"P4-constraints + 1500B IPv4-TCP pkt":</p><blockquote><p>time p4testgen --target bmv2 --arch v1model --test-backend PTF --max-tests 0 --out-dir results  --seed 1 --track-coverage STATEMENTS --packet-size-range 12000:12000 -DTESTGEN_ASSUME_IP -DTESTGEN_ASSUME_TCP p4/pins_middleblock_mod.p4</p></blockquote><p>Replace the data array in line 49 of plots.py with the results gathered from these runs and run the script p4c/backends/p4tools/modules/testgen/benchmarks/plots.py.</p><h4>Table 6 and Figure 6/7</h4><p>Figure 7 can only be reproduced with access to a Tofino binary and the corresponding <i>tna_simple_switch</i> program. Please reach out to us to see whether we can share a binary. We shared our measurement data as part of the provided <i>coverage_data.zip</i>. To produce Figure 7 run <i>p4c/backends/p4tools/modules/testgen/benchmarks/plots.py -i coverage_data -o results</i></p><p>Other numbers in table 6 can be reproduced with the commands</p><blockquote><p>p4c/backends/p4tools/modules/testgen/benchmarks/test<i>coverage.py -b /usr/local/bin/p4testgen -o benchmark/pins_middleblock</i> --test-backend PTF -i 10 -p p4c/testdata/p4_16_samples/pins/pins_middleblock.p4</p></blockquote><blockquote><p>p4c/backends/p4tools/modules/testgen/benchmarks/test_coverage.py -b /usr/local/bin/p4testgen -o benchmark/up4 --test-backend PTF -i 10 -p p4c/testdata/p4_16_samples/omec/up4.p4</p></blockquote><blockquote><p>p4c/backends/p4tools/modules/testgen/benchmarks/test_coverage.py -b /usr/local/bin/p4testgen -o benchmark/dash-pipeline-pna-dpdk -i 10 -p p4c/testdata/p4_16_samples/dash/dash-pipeline-pna-dpdk.p4 --test-mode DPDK</p></blockquote><p>Figure 6 can be derived from the data collected in the measurements. Alternatively, run</p><blockquote><p>time p4testgen --target bmv2 --arch v1model --test-backend PTF --max-tests 10000 --out-dir results --stop-metric MAX_NODE_COVERAGE --track-coverage STATEMENTS p4c/testdata/p4_16_samples/pins/pins_middleblock.p4 --seed 1 --print-performance-report</p></blockquote><p>Run <i>p4c/backends/p4tools/modules/testgen/benchmarks/plots.py -i coverage_data -o plots</i></p><h4>Table 2</h4><p>Data in Table 2 is sourced by running P4Testgen on the respective program, e.g.,</p><blockquote><p>time p4testgen --target bmv2 --arch v1model --test-backend PTF --max-tests 0 --out-dir results p4c/testdata/p4_16_samples/pins/pins_middleblock.p4 --seed 1</p></blockquote><blockquote><p>time p4testgen --target bmv2 --arch v1model --test-backend PTF --max-tests 0 --out-dir results p4c/testdata/p4_16_samples/omec/up4.p4 --seed 1</p></blockquote><blockquote><p>time p4testgen --target dpdk --arch pna --test-backend PTF --max-tests 0 --out-dir results p4c/testdata/p4_16_samples/dash/dash-pipeline-pna-dpdk.p4 --seed 1</p></blockquote><h4>Bugs</h4><p>Open-source bugs are linked in Table 8. Many of them are fixed and may not be reproducible anymore. Please reach out to us if you would like to reproduce these bugs. We have also marked them as failing tests <a href="https://github.com/p4lang/p4c/blob/main/backends/p4tools/modules/testgen/targets/bmv2/test/BMV2PTFXfail.cmake#L5">here</a>.</p&gt