At a recent Dagstuhl seminar on web application security, I spoke with Jasvir Nagra (one of the people at Google working on Caja) about the best way to run a public security contest. The goal of the contest would be to encourage people to find as many bugs as possible, with additional rewards for more severe security vulnerabilities. As a starting point, we discussed the ongoing Google Native Client (NaCl) security contest. For this contest, the top five bug-finders over a ten-week period receive recognition and cash prizes of $8192, $4096, $2048, $1024, and $1024. While this model has a few strong points, it also is problematic in the way that it incentivizes security research. This article discusses how to more effectively design a security contest.

Before getting into the problems with the NaCl security contest, let us first take a look at what it does right. One thing I like about it is that there is a distinct “winner,” as well as second through fifth place, which are also meaningful awards. This highlights a key point of security contests: it is about the public recognition, not the money. $8,192 is a drop in the bucket for someone who is talented enough to win a major security contest. However, the recognition from winning fosters not only a sense of accomplishment, but could propel the individual into a very well-paid job or consulting position. Unless the prizes were upped by an order of magnitude (say $100,000 instead of $8,192), the amount of cash probably would not make much of a difference. And also, why should Google pay more if it can achieve the same result by posting a picture and bio of the winner on its blog?

Now onto the problems with the NaCl contest… First, it is possible for someone to find a bug and receive no prize or recognition. As a security researcher, spending time to search for bugs that may or may not exist is already an extremely high-risk endeavor; months of work could yield nothing. This prize structure only increases this risk. Any security competition should recognize and reward all participants who discover new vulnerabilities. Perhaps a $512 prize along with a list of all successful bug-finders would do the trick.

Presumably, the goal of security testing is to continuously discover bugs throughout the life of a product, not just find them once. The NaCl contest completely fails in this regard. Only those researchers who are available during the 10-week period of the contest will participate, and they will stop doing so after the contest is over. Ideally, anyone should feel compelled to jump in at any time, and spend any amount of effort finding bugs. Also, security evaluation is still important for more mature products! One possible way of achieve these goals is to let the contest run indefinitely, but announce winners every month and every year. This way, you would still have the incentive of someone being the “winner” (the yearly winner might be even more renowned than a 10-week winner), but researchers would continue to search for bugs.

Many security researchers have an affiliation with an institution, such as university, large company, or consulting group. These institutions have very little incentive to encourage their employees to work on finding security vulnerabilities for someone else (in this case, Google). Google could change this by soliciting an affiliation from each participant, and then giving a yearly award to the institution whose members collectively discovered the most and best bugs. This award could actually be a sizeable grant ($100,000) for the institution to conduct future security research. Furthermore, this award could encompass security contests for multiple products, such as Caja and NaCl. Rewarding institutions for successful security research would encourage their members to put even more effort into security contests.

The last way that one might improve participation in a security contest is to give extra recognition to the winners. For example, you could display the winner’s picture, bio, and an anecdote about how he or she discovered the bug(s). This type of promotion (you could call it bragging) really motivates people to put in extra effort to win a security competition. At the end of the day, the more likely a person or institution thinks they are to gain recognition from the vendor, the more likely they are to participate in a security contest.

This entry was posted on Friday, April 17th, 2009 at 2:13 pm and is filed under Vulnerabilities, Economics, Articles, Penetration Testing. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.