However, all three bug bounty platforms downplayed this risk, and none of the three acknowledged any instance where a bug bounty program led to GDPR breach notification. "I don't have data on how often bounty program activity triggered a breach notification in 2019," HackerOne's Rice says. "It's quite rare, so I'd guess a handful, if there were any at all."
Bugcrowd pointed out that 90% of its bug bounty programs are invite-only private programs, meaning NDA required. "All researchers participating in private bug bounty programs," they write, "must be pre-vetted through our internal vetting process. In order to be invited to private programs, researchers must prove their abilities and trustworthiness via public programs."
BugCrowd declined to elaborate on what the process of pre-vetting researchers actually looks like. So, this reporter signed up for an account and had immediate access to all public programs without any additional steps.
To be eligible for private programs, BugCrowd invites researchers to fill out this Google form, which asks researchers for their LinkedIn, GitHub and Twitter accounts, "Name or Pseudonym", plus "any relevant certifications or qualifications that validate your testing credentials." The form also asks, "Would you be willing to be background checked?" noting that, "Specific customers require researchers to be background checked in order to participate in their private programs." The form warns that "if you don’t provide detailed information you will not be considered for private program invites."
SynAck, which runs only private bug bounty programs, vets all its independent researchers (the "SynAck Red Team"), including a criminal background check, and accepts only around 20% of applicants, a SynAck representative tells CSO.
What this really highlights is that companies can augment but not replace a traditional pen testing engagement with bug bounties. Further, only a small portion of bug reporting can be outsourced, and convoluted attempts to do so might backfire.
Some things your security team has to do in house.
NDAs are not ISO compliant
Good security people are scarce, and at a time of zero unemployment among information security professionals, it can be tempting to outsource anything you can. But a VDP is not something you can fully outsource.
HackerOne's marketing strongly implies that its platform can help companies outsource ISO 29147 and ISO 30111, the standards that define best practices on receiving security bug reports, fixing those bugs, and publishing advisories. However, the sole author of both of those standards, Moussouris, says this is flat out not possible.
ISO 29147 standardizes how to receive security bug reports from an outside reporter for the first time and how to disseminate security advisories to the public.
ISO 30111 documents internal digestion of bug reports and remediation within an affected software maker. ISO provided CSO with a review copy of both standards, and the language is unambiguous.
These standards make clear that private bug bounty NDAs are not ISO compliant. "When non-disclosure is a required term or condition of reporting bugs via a bug bounty platform, that fundamentally breaks the process of vulnerability disclosure as outlined in ISO 29147," Moussouris says. "The purpose of the standard is to allow for incoming vulnerability reports and [her emphasis] release of guidance to affected parties."
ISO 29147 lists four major goals, including "providing users with sufficient information to evaluate risk due to vulnerabilities," and lists eight different reasons why publishing security advisories is a standardized requirement, including "informing public policy decisions" and "transparency and accountability." Further, 29147 says that public disclosure makes us all more secure in the long term. "The theory supporting vulnerability disclosure holds that the short-term risk caused by public disclosure is outweighed by longer-term benefits from fixed vulnerabilities, better informed defenders, and systemic defensive improvements."
Contrast this with HackerOne's "5 critical components of a vulnerability disclosure policy" that lets organizations muzzle security researchers. "Set non-binding expectations for how reports will be evaluated," the company recommends. "This section can include the duration between submission and response, confirmation of vulnerability, follow-on communications, expectation of recognition, and if or when finders have permission to publicly disclose their findings." [emphasis ours]
Rice disputes the accusation that HackerOne is unable to offer ISO compliance, saying that "H1 Response [their VDP offering] allows organizations to comply with all guidance from 29147 and 30111 in its default configuration." That's like saying Monster Sugar O'Cereal is a part of a balanced breakfast. Technically true, but it doesn't add much nutrition.
"For bug bounty companies to claim they help at all with ISO 30111 shows they don't actually understand these standards or how to comply with them. It's false advertising at best, and outright lies at worst," Moussouris says. "None of what the bug bounty platforms provide has anything to do with this part of the process [ISO 30111]. They can only help with a small part of ISO 29147, which is intake and initial triage."
Triage. Welcome to a world of pain.
HackerOne "weaponized triage"
Running a VDP typically results in a trickle of reports, as only good-faith researchers expecting zero payout will contact you. Start paying a bug bounty, though, and every wannabe script kiddie looking for a quick payout will flood your inbox with garbage reports.
"When people run their own bug bounty programs, they quickly live to regret it," Latacora’s Van Houtven says. "They get thousands of reports, mostly bad reports. There's a massive incentive [for low-skilled bug finders] to spam everyone with complete bullshit."
That's not an argument to outsource bug bounties to one of the platforms, but rather to question whether your organization needs a bug bounty at all. A tsunami of bad bug reports is a problem that the bug bounty platforms both create and solve, Van Houtven says.
"HackerOne has weaponized this. They make a ton of money off triage. The problem is HackerOne has a terrible perverse incentive. They want to preserve the status quo of people submitting bullshit scanner findings," Van Houtven says. "HackerOne's job is to make bug bounties as bad as possible for everyone because they make more money that way. I think HackerOne should not exist. Their business model is misery."
Rice denies this charge. "HackerOne has every incentive to make triage a pleasant experience for hackers, our customers and our staff," he says. "We are constantly working to improve the triage process for everyone."
Responsible use of bug bounties
A company with an existing VDP and a mature bug triage and remediation team can and, in some cases, should augment their existing, robust red team with a bug bounty. But the bug bounty is the cherry on top of the cybersecurity sundae. It's nice to have, looks pretty, and adds a bit of flavor oomph.
There are layers: The foundation is a published VDP, at CompanyX.com/security, with a security@ email address, a PGP key (that works), and clear safe harbor language that permits researcher disclosure. The second is to employ a proper penetration testing consultancy and, if organizational size and budget warrants it, an in-house red team. Once those planks of due diligence have been laid, a bug bounty can help find issues everyone else has missed.
Mature organizations can and should run their own VDP in house. If they are ready for an avalanche of dubious bug reports, they might optionally choose to run a bug bounty.
To the extent that bug bounty platforms need to exist, they are a modest value-add. "I told them before I left," Moussouris says, "If you guys can continue to reduce friction between researchers and vendors, that's a good thing. If you try to sell control, you're in the wrong space."