Software vulnerabilities can have serious consequences, which is why many
techniques have been proposed to defend against them. Among these,
vulnerability detection techniques are a major area of focus. However, there is
a lack of a comprehensive approach for benchmarking these proposed techniques.
In this paper, we present the first survey that comprehensively investigates
and summarizes the current state of software vulnerability detection
benchmarking. We review the current literature on benchmarking vulnerability
detection, including benchmarking approaches in technique-proposing papers and
empirical studies. We also separately discuss the benchmarking approaches for
traditional and deep learning-based vulnerability detection techniques. Our
survey analyzes the challenges of benchmarking software vulnerability detection
techniques and the difficulties involved. We summarize the challenges of
benchmarking software vulnerability detection techniques and describe possible
solutions for addressing these challenges