Given enough eyeballs, all bugs are shallow? Revisiting Eric Raymond with bug bounty programs

Bug bounty programs offer a modern way for organizations to crowdsource their software security, and for security researchers to be fairly rewarded for the vulnerabilities they find. However, little is known on the incentives set by bug bounty programs – how they drive engagement and new bug discoveries. This article provides an empirical investigation of the strategic interactions among the managers and participants of bug bounty programs, as well as the intermediation by bug bounty platforms. We find that for a given bug bounty program, each security researcher can only expect to discover a bounded number of bugs. This result offers a validation step to a theory brought forth early on by Brady et al. This theory proposes that each security researcher inspecting a piece of software offers a unique environment of skills and mindset, which is amenable to the discovery of bugs that others may not be able to uncover. Bug bounty programs indeed benefit from the engagement of large crowds of researchers. Conversely, security researchers benefit greatly from searching for bugs in multiple bug bounty programs. However, we find that following a strong front-loading effect, newly launched programs attract researchers at the expense of older programs: the probability of finding bugs decays as ∼1/t0.4∼1/t0.4 after the launch of a program, even though bugs found later yield on average higher rewards. Our results lead us to formulate three recommendations for organizing bug bounty programs and platforms: (i) organize enrollment, mobility, and renewal of security researchers across bounty programs, (ii) highlight and organize programs for front-loading, and (iii) organize fluid market transactions to reduce uncertainty and thus reduce incentives for security researchers to sell on the black market

Given enough eyeballs, all bugs are shallow? Revisiting Eric Raymond with bug bounty programs

Bug bounty programs offer a modern way for organizations to crowdsource their software security, and for security researchers to be fairly rewarded for the vulnerabilities they find. However, little is known on the incentives set by bug bounty programs – how they drive engagement and new bug discoveries. This article provides an empirical investigation of the strategic interactions among the managers and participants of bug bounty programs, as well as the intermediation by bug bounty platforms. We find that for a given bug bounty program, each security researcher can only expect to discover a bounded number of bugs. This result offers a validation step to a theory brought forth early on by Brady et al. This theory proposes that each security researcher inspecting a piece of software offers a unique environment of skills and mindset, which is amenable to the discovery of bugs that others may not be able to uncover. Bug bounty programs indeed benefit from the engagement of large crowds of researchers. Conversely, security researchers benefit greatly from searching for bugs in multiple bug bounty programs. However, we find that following a strong front-loading effect, newly launched programs attract researchers at the expense of older programs: the probability of finding bugs decays as ∼1/t0.4∼1/t0.4 after the launch of a program, even though bugs found later yield on average higher rewards. Our results lead us to formulate three recommendations for organizing bug bounty programs and platforms: (i) organize enrollment, mobility, and renewal of security researchers across bounty programs, (ii) highlight and organize programs for front-loading, and (iii) organize fluid market transactions to reduce uncertainty and thus reduce incentives for security researchers to sell on the black market

Friendly Hackers to the Rescue: How Organizations Perceive Crowdsourced Vulnerability Discovery

Over the past years, crowdsourcing has increasingly been used for the discovery of vulnerabilities in software. While some organizations have extensively used crowdsourced vulnerability discovery, other organizations have been very hesitant in embracing this method. In this paper, we report the results of a qualitative study that reveals organizational concerns and fears in relation to crowdsourced vulnerability discovery. The study is based on 36 key informant interviews with various organizations. The study reveals a set of pre-adoption fears (i.e., lacking managerial expertise, low quality submissions, distrust in security professionals, cost escalation, lack of motivation of security professionals) as well as the post-adoption issues actually experienced. The study also identifies countermeasures that adopting organizations have used to mitigate fears and minimize issues. Implications for research and practice are discussed

The Use of Bug Bounty Programs for Software Reliability Improvement

As the number of security breaches caused by third-party applications significantly increased, digital platforms are launching BBPs to help improve software reliability. BBPs bring benefits to the platform and vendors, meanwhile impose additional costs; and may change the vendors’ reliability investment incentive. We build a model to examine strategic decisions of launching and participating in a BBP for the platform and third-party vendor, respectively. We find that the platform’s (vendor’s) launching (participation) decisions depend on two key factors: the expected loss due to security breaches and the vendor’s investment efficiency. The incentive of using BBP, for the platform and vendor, sometimes is inconsistent. Only when the potential loss is high and investment efficiency is low, BBP would be the equilibrium outcome. We find using the BBP is not always socially optimal. Under certain conditions, it reduces the overall software reliability, makes the platform less reliable, and hurts end users

Decentralized Attack Search and the Design of Bug Bounty Schemes

Systems and blockchains often have security vulnerabilities and can be attacked by adversaries, with potentially significant negative consequences. Therefore, infrastructure providers increasingly rely on bug bounty programs, where external individuals probe the system and report any vulnerabilities (bugs) in exchange for rewards (bounty). We develop a simple contest model of bug bounty. A group of individuals of arbitrary size is invited to undertake a costly search for bugs. The individuals differ with regard to their abilities, which we capture by different costs to achieve a certain probability to find bugs if any exist. Costs are private information. We study equilibria of the contest and characterize the optimal design of bug bounty schemes. In particular, the designer can vary the size of the group of individuals invited to search, add a paid expert, insert an artificial bug with some probability, and pay multiple prizes