Several years ago, crowdsourced quantitative investment firm Quantopian implemented its first bug bounty program after they knew a pen-testing firm couldn’t have caught everything. The bounty structure was loose and participants weren’t actively recruited or invited. Fast forward to today, CISO Jonathan Kamens claims that their bug bounty program is the foundation of Quantopian’s application security practice, which helps protect its over 230,000 registered members. So much so, they are continuing to evolve their program to keep hackers engaged. This morning, Quantopian announced they are temporarily increasing their bounties 1.5x their usual bounty structure and they’ve announced new additions to the site ripe for testing. We connected with Jonathan to dig into what makes the program successful and what it’s been like working with hackers over the years. Here’s a peek at what we learned:
Q: Introduce yourself and Quantopian. Tell us what you do and why cybersecurity is so important to your business.
A: Quantopian is a crowd-source quantitative investment firm. We inspire talented people from around the world to write investment algorithms. Headquartered in Boston, one of the fintech capitals of the world, we provide capital, education, data, a research environment, and a development platform to algorithm authors. We offer license agreements for algorithms that fit our investment strategy, and the licensed authors are paid based on their strategy’s performance. In addition, we run a daily contest where authors can win real money for high-performing algorithms.
I joined Quantopian six and a half years ago, less than a year after the company’s founding. Originally, I was Vice President of Operations, and our information security was one of my many responsibilities. Eventually, it became clear to us that we needed someone dedicated full-time to security, and I transitioned into the role of CISO. I’ve worked in infosec on and off throughout my 30-year career. Being Quantopian’s CISO has been my first opportunity to focus on it full-time.
The 230,000+ registered members of our community rely on us to keep their algorithms and data safe. Our business literally depends on us earning and keeping the trust of our members. We don’t read our members’ algorithms’ source code, and it’s critical for us to make sure no one else does either.
Q: Why did Quantopian decide to start a bug bounty program in the first place? What have been some results of your program to date?
A: Several years ago, we contracted with a well-regarded cybersecurity consulting firm to do pen-testing and a white-box security audit of our entire platform. They identified some issues with our platform, which we addressed, including one substantive vulnerability in the sandbox in which we execute our members’ algorithms. At the time, we thought they had done a good, thorough job evaluating the security of our platform.
Subsequent to that, we published a /security page on our site, explaining our approach to security, inviting hackers to test our site, and offering bounties for vulnerabilities that were disclosed responsibly to us. Although that was the extent of our bug-bounty “program” — we weren’t really promoting it anywhere and we were pretty vague about our bounty amounts — we started receiving reports from hackers of vulnerabilities which the consulting firm should have found but didn’t. We decided we needed to try something different.
One of our product managers, Abhijeet Kalyan, had used HackerOne in the past, so he suggested we give it a try. At that time HackerOne had a free tier where the only cost was the bounties paid out including fees, so we literally had nothing to lose: if HackerOne hackers didn’t find any legitimate vulnerabilities, we wouldn’t have to pay anything.
But the talented hackers on HackerOne did find vulnerabilities. Some of them were in new parts of the application that the consulting firm didn’t test, but we found that, once again, hackers were finding things that the expensive consultants should have found but didn’t. At this point, we were convinced that HackerOne’s model was superior to paying consultants for pen-testing, and we’ve become even more convinced of that as our HackerOne bounty program has matured.
Q: Tell us more about the bug bounty promotion you’re running this month.
A: If we’re doing cybersecurity right, we expect the number of valid reports coming into our bug bounty program to decrease from their initial peak: the hackers participating in our program have found and reported all of the “easy” issues, and we’ve fixed them. That is, indeed, what we’ve been seeing. There are two things we can focus on to re-energize the hackers participating in our program and increase the flow of valid reports: increase our bounties to encourage hackers to spend time looking for harder-to-find vulnerabilities; and let the hackers know about recent changes to the site so they can focus their testing on those, since that’s where new vulnerabilities are most likely to be introduced.
The promotion we announced this morning attacks on both of those fronts: we’re temporarily increasing our bounties to 1.5x, and we’ve enumerated three significant recent changes to the site — custom datasets, an upgrade to our Jupyter notebook environment, and mailing list opt-in/out — with which we hope our hackers will engage productively.
Moving forward, we hope to do these promotions regularly to keep hackers engaged in our program and ensure that our new features get the testing they need.
Q: What’s in scope? What findings are most interesting to your team?
A: It’s probably more interesting to ask what’s not in scope; we’ve collected on our policy page a list of the types of reports we haven’t found useful in the past. Outside of those, we try to be liberal in what we accept and pay bounties for. For example, a hacker recently reported an outdated library with a known vulnerability on our site. Although he wasn’t able to figure out how to exploit the vulnerability, we had known we needed to update it but hadn’t done it properly, so we paid out a bounty on that report.
Q: How does the bug bounty program impact Quantopian’s larger cybersecurity strategy?
A: Information security at a company like Quantopian cuts across many different areas and functions. Our bug bounty program is the foundation of our application security practice.
Q: How does the bug bounty program supplement the work of the existing team?
A: We rely on a large, ever-growing corpus of automated tests to protect against regressions, including security regressions, making it into our code base. Furthermore, we educate our developers in secure coding practices on an ongoing basis, we do security reviews for many projects before they are released, and we have a robust code review practice in which nearly everything that goes into our code is looked at by at least two people – or three people for changes that are security-sensitive. However, despite all these efforts, we know that bugs are going to slip through and some of those are going to impact security. That’s where our bug bounty program comes in.
Q: What has it been like working with hackers thus far? What has been one of your favorite hacker interactions to-date?
A: My two favorite interactions with hackers both actually happened at HackerOne’s [email protected] Conference in San Francisco last year. First, I was amused to realize that two of the three hackers participating in the panel discussion of top HackerOne hackers had submitted vulnerabilities to Quantopian’s bug bounty program.
Even better than that, however, was when someone walked up to me at the networking hour at the end of the conference and said, “Hi. I’m sorry to interrupt. I’m Jon Bottarini. We’ve never met in person, but I needed to introduce myself to you because you changed my life.” He then explained that two years ago when he was fresh out of college, the first-ever report he submitted to a bug bounty program was submitted to Quantopian’s, and (I’m quoting him here) “it was garbage.” He said I kindly and patiently explained to him why his report was garbage and then paid him $100 anyway. And then he said the way I responded to him in that report set him on the path to where he is now, working as a technical program manager at HackerOne.
I haven’t always succeeded at being as patient and kind as I would like with the hackers in our program. However, every time I find myself becoming frustrated with a hacker interaction, I think about Jon and the impact I had on him, and my patience is restored.
Q: What advice would you give hackers participating in your program?
A: Read our program policy! We’ve invested a lot of time and effort ensuring that it’s clear and comprehensive. The best way to ensure that you don’t waste your time or ours is to read and understand our policy.
If you have any questions about our program, please don’t hesitate to reach out and ask. You can always reach us at [email protected] We also have a Slack channel where you can chat with us and other hackers in our program. Email us if you’d like an invite.
If you’re interested in learning more about Quantopian’s program or want to get hacking, visit https://hackerone.com/quantopian.