Going, Going … Gone! Auctioning IT Security Bugs

One day in 1947 the Mark computer at the Harvard Computer Laboratory failed, and the cause was traced back to a bug (specifically a moth) found in one of its electromechanical relays. It was already common for engineers to attribute inexplicable defects to “bugs in the system”, but the Mark team had now found real evidence of one of these mythical bugs. The bug was taped into the Mark log book entry for posterity, and soon all computer defects were referred to as bugs. While the Mark story has passed into computer folklore, the bugs themselves have not.

Most bugs today occur in software, and part of the reason for this is sheer size. Windows XP has 40 million lines of code, which if printed out on A4 paper would stretch for over 148 kilometres when laying the pages end-to-end. If you wanted to check Windows Vista, you would have a further 37 kilometres of reading, and a further 180 kilometres for the Mac operating system. When software becomes this large and complex it is not surprising that some bugs are introduced.

Security bugs, also referred to as vulnerabilities, potentially permit an attacker to bypass security mechanisms and gain unauthorized access to system resources. Security bugs normally do not interfere with the operation of software and typically only become an issue when a bug is found and further exploited in an attack. The Zotob worm that infected many high profile companies in late 2006 exploited a vulnerability that had been undetected since the release of Windows 2000. In May 2008 a 25-year old bug in the BSD file system was discovered and fixed.

How are vulnerabilities found? Well software vendors uncover many bugs themselves, but a significant number of vulnerabilities are also found by third parties such as security software specialists, researchers, independent individuals and groups, and of course, hackers. Software vendors encourage these bug hunters to report their findings back to them first before going public with the vulnerability, so that an appropriate patch can be prepared to limit any potential negative consequences. In the computer security industry this is called “responsible disclosure”. For bug hunters that abide by this principle, the software vendors are normally willing to give kudos for finding vulnerabilities, and in some cases, also provide a nominal financial reward.

Over time bug hunters have started to become more interested in the financial reward part rather than the kudos. For example, in August last year a lone security researcher was asking €20,000 for a vulnerability he had found in a Nokia platform. Was this a good deal? What monetary value do you place on finding such a vulnerability? Software vendors want to produce patches for vulnerabilities in their products as soon as possible, while hackers want to produce code exploiting a known vulnerability as soon as possible. Considering these alternatives, how much should the bug hunter be paid for his or her services? Are low financial rewards from software companies driving bug hunters into the arms of hackers? Perhaps a market could efficiently create a fair price.

Charlie Miller certainly thinks so. The ex-NSA employee described his experiences in selling vulnerabilities in an interesting paper, and you can read a quick interview at Security Focus. Miller found it hard to set a fair price and had to rely significantly on his personal contacts to negotiated a sale – contacts that are likely not to be available to most security researchers.

In 2007 a Swiss company called WSLabi introduced an online marketplace where security bugs can be advertised and auctioned off to the highest bidder. This latest experiment in security economics appears to have lasted about 18 months and the WSLabi site is now unavailable. The marketplace started running in July 2007, and in the first two months 150 vulnerabilities were submitted and 160,000 people visited the site. Of those sales that have been publicly disclosed, the final bids have been between 100€ to 5100€. An example list of advertised vulnerabilities on the marketplace is shown below.

The WSLabi marketplace was controversial since many security people believe that hackers, or more recently, organized digital crime, should not have early access to vulnerabilities. In 2005 eBay accepted a critical vulnerability in Excel as an item to be auctioned, but management quickly shut down the auction saying that the sale would violates the site's policy against encouraging illegal activity. "Our intention is that the marketplace facility on WSLabi will enable security researchers to get a fair price for their findings and ensure that they will no longer be forced to give them away for free or sell them to cyber-criminals," said Herman Zampariolo, head of the auction site. In another interview Zampariolo explains that "The number of new vulnerabilities found in developed code could, according to IBM, be as high as 140,000 per year. The marketplace facility on WSLabi will enable security researchers to get a fair price for their findings and ensure that they will no longer be forced to give them away for free or sell them to cyber-criminals" .

Unfortunately the marketplace experiment was unexpectedly cut short. In April 2008 the WSLabi CEO Roberto Preatoni blogged about his regrettable arrest and the damage this had on the trust of WSLabi as a broker, notwithstanding being Swiss! Six months later in a public interview Preatoni remarked that "It [the marketplace] didn't work very well … it was too far ahead of its time". Apparently security researchers recognised the value of having an auction site like WSLabi, but very few buyers proved willing to use the site. Buyers seemed to value their privacy more than they value a fair price. Preatoni is now directing his company’s efforts to using zero-day threat information for the development of unified threat hardware modules – which you can read about on the last WSLabi blog post from October 2008.

The Half-life of Vulnerabilities is still 30 Days

Wolfgang Kandek, CTO of Qualys, recently gave an update on the Laws of Vulnerabilities research that Qualys initiated in 2004. Based on scanning 3 million IP addresses, and considering 2 million vulnerabilities, the initial results found that the half-life of unpatched vulnerabilities was 30 days. That is, the observed rate of patching halved the number of open vulnerabilities each month.

Kandek repeated this exercise on a grander scale in 2008, scanning 80 million IP addresses for over 870 million vulnerabilities, including 72 million that were critical. The data confirmed that the vulnerability half-life was 29.5 days, essentially unchanged from the initial finding 4 years before. This was an average taken over several 5 industry sectors, where the service sector had the lowest half-life at 21 days and the manufacturing sector had the highest at 51 days. The health sector weighed in at 38 days. Topping the list of the chronically under-patched were MS Office, Windows 2003 SP2, the Sun Java Plugin and Adobe Acrobat.

While the average half-life has remained essentially constant over the last 4 years, Kandek notes that the time from discovery to exploiting a vulnerability is going down. Qualys is aware of 56 zero-day exploits, and the availability of exploits is now measured in single digit days. Even though the half-life measure suggests that a given set of vulnerabilities will rapidly become “extinct”, in practice their threat lives on indefinitely since most vulnerabilities are never fully patched. Further, this patching rate is offset by a 60% replacement rate by new vulnerabilities.

Kandek concludes that

“Security is getting more difficult with attackers becoming extremely sophisticated and the window of exploitation shrinking to days for most critical vulnerabilities … Our goal with this research is to help organizations across different industries understand the broader trends, the potential for damage and the priority of vulnerabilities, so they can make more effective and more immediate decisions to protect their networks. With research like that outlined in the Laws of Vulnerabilities 2.0, we can provide the industry with a statistical look at threat trends in real-time."

Also, take a look at some recent advice from Tenable Security on how to read vulnerability reports, which will help you interpret Kandek's charts.

Randomness Tests for Packed Malware

In my post Risk Factors for AV Scanning, I discussed a recent patent filing by Kaspersky related to reducing the amount of malware scanning based on various file properties. One of those properties was whether the file is packed or not, since packed files are to be treated more suspiciously than plain unpacked files.

The impact of packed malware is discussed more thoroughly by Tzi-cker Chiueh of Symantec Research Labs in a 2007 presentation on the difficulties in detecting malware, where he devotes 5 slides to the "Packer Problem". Packing adds another dimension to the production of malware signatures which must now account for potentially ten or more layers of packing that obscure the real code that needs to be scanned. Symantec knows of over 1000 different packing programs all of which need to be recognised and stripped off to apply signature scanning.

But is it possible to distinguish between suspicious and actual packed malware using statistical properties of the files without unpacking?

Three authors (Tim Erbinger, Li Sun & Serdar Boztas) have done some interesting research into developing specific randomness tests for detecting packed malware. As it turns out, malware does exhibit a definite randomness signal when the randomness measure accounts for local patterns and structure.

The paper begins by observing that common randomness measures, such as the entropy or other statistical tests, compute a global measure of randomness over all available data and tend to obscure (local) sections of code that may be highly random or highly structured. The figure below shows the distribution of bytes in the program calc.exe before and after packing with UPX. The distribution becomes more uniform after packing, but still far from uniform.

The authors proposed several randomness tests that preserve locality (structure or lack thereof), based on constructing a Huffman tree at the byte level of the data (packed file). The Huffman tree algorithm computes codes for each observed byte where more frequent bytes are assigned shorter codes. If the data is random, causing the byte frequencies to be similar, then the Huffman tree codes will be roughly of similar length. Structured data on the other hand will produce codes of quite differing lengths.

The authors then sample the data at various bytes positions, collecting the corresponding Huffman codes, and then normalise the code lengths between 0 and 1 (where a higher value means more random). The byte sampling strategies proposed are (1) sample a fixed number of bytes equally spaced in the data (2) slide a window of fixed size across all the data. Below we see the sliding window test applied to the basename binary from UnxUtils before and after being packed with FGS 2.0.

UnxUtils has 116 binaries ranging in size from a few KB to about 200 KB, and the authors performed a collection of experiments to validate their local randomness tests using 6 well-known packers.
The authors observe that there is a definite characteristic signal of packed files. They go onto to further propose additional tests that can can be used to discriminate between packers.

Returning to Tzi-cker Chiueh and his packer problem, he suggests to workaround packing by Just-in-Time scanning approach which tries to scan a file just before it runs in its unpacked form, and also to consider whitelisting. Perhaps this new local randomness test from Erbinger, Sun & Boztas can help him out.

Related Posts

• Risk Factors for AV Scanning
• The Positive Trust Model and Whitelist

The Positive Trust Model and Whitelisting

I recently came across the presentation The Positive Trust Model and Whitelists, by Wyatt Starnes of Signacert, a company that specialises in whitelisting solutions (get the video here). I thought the presentation made some good points, worth repeating here, and extending with other opinions (of which there are many).

Whitelisting, like virtualisation, is a mainframe concept largely forgotten in the era of personal computing, recently rediscovered in our modern IT environment. There are various forms of whitelisting for security purposes, for example in the context of combating email fraud and SPAM, but here we will be concerned with application whitelisting - a method to ensure that only approved applications and their associated executables are permitted to run on a given machine.

John W. Thompson, CEO of Symantec, from his 2008 RSA keynote, supported whitelisting in the face of growing malware diversity (quoted by Starnes)

From where I sit, a few things are very, very, clear. If the growth of malicious software continues to outpace the growth of legitimate software, techniques like whitelisting, where we identify and allow only the good stuff to come in, will become much, much, more critical.

This is a telling statement from the CEO of a company whose cash cow is desktop AV software, the epitome of blacklisting technology. Thompson, and other companies whose business models are firmly based on blacklisting, now agree that whitelisting as a malware defence is an idea whose time has come.

Malware is Increasing

Blacklisting is not really about maintaining a list of prohibited software, but rather maintaining a database of malware signatures to evaluate the state of software through scanning. Software is blacklisted when it is identified to have characteristics identical or similar to known malware. And this is the key point - known malware. The successful and timely identification of malware depends on the rapid identification, production and distribution of updates to signature databases.

Over the last year an inflection point was reached where malware crossed over as being produced in greater quantities than legitimate software. We are heading to the same state of affairs in email where SPAM dominates the number of legitimate messages. Starnes depicted this situation as follows

The slide heading for the graphic above is Chase the Infinite or Confirm the Finite? The question asks whether it is a better IT defensive strategy to attempt to screen a wide and increasing variety of malware, or focus on maintaining the current integrity of the known components of your IT system.

A presentation from Martin Fréchette of Symantec Labs, given at RAID 2007, provides more background. First he has a more detailed graph on the number of new threats, which are essentially increasing exponentially.

By the end of 2008 there were approximately 1,000,000 known examples of malware, over 2/3 of which had been produced in 2008. That is, 2008 saw more malware produced than all previous years combined. While this sounds alarming, Fréchette notes that part of the reason known malware has been increasing rapidly is due to better detection methods, in particular honeypots and malware sensor networks.

But malware is also increasing due to a change in strategy of the malware industry. Fréchette observes a shift from a mass distribution of a small number of threats to micro distribution of millions of distinct threats, more recently referred to as targeted attacks. Symantec has observed single days where 10,000 new virus strains have been produced, mainly through a technique known as server-side polymorphism, which can automatically regenerate malware strains.

Fréchette notes that the micro distribution strategy is greatly reducing the effectiveness of classic malware signature detection. Even just a few years ago a single signature could be expected to protect 10,000 users whereas today that expectation has dropped to less than 20 users. That is, malware attacks are so specific that signatures serve only to protect small groups of users. Thus signatures must be produced in vast numbers to protect the broader user community.

The Twilight of Blacklisting

The AV blacklisting industry has reached a point of diminishing returns - the marginal value of producing additional signatures is minimal, but the underlying model can offer no more advice than to simply keep doing exactly that. The AV signature cycle of detect-produce-distribute is being overwhelmed, and the effectiveness of AV solutions (that is, the fraction of known malware that is detectable) is decreasing. Equivalently, the false positive rate is increasing, and consumers are getting less protection than they expect.

There is a significant burden on networks to distribute signatures and also on platforms to perform scanning. Scanning each and every file is neither feasible nor effective. In October last year I posted some remarks on a new patent granted to Kaspersky for a risk-based approach to AV scanning which described criteria (risk factors) for reducing the amount of file scanning. In July last year Robert Vamosi of CNET reported that Norton will also follow a risk-based approach in 2009 with their products, creating a trust index that will be used to judge how often files are scanned. However when I posed the question Would you support less malware scanning to improve user performance? over at LinkedIn the resounding answer was No.

Blacklisting only has a future as a primary security defense if we can actually find ways to do less of it and still retain a low false positive rate. But this sounds like squaring the circle.

Charge of the White Brigade

Enter Whitelisting. Rather than attempting to determine if an arbitrary file (executable) is malicious based on signatures or other criteria, whitelisting creates approved copies of software and simply checks whether the current copy of a binary is the same as its approved copy. Software that is not on the approved list are blocked from running, period. Starnes represents the whitelist production process as follows

There is a significant reliance on hashing and signatures for trust, where signature here means a PKI signature, not a blacklist signature. Unauthorized change in a file is detected (with high probability) by a change in its associated hash which will in turn cause the signature verification step to fail.

Notice that the hash-sign paradigm of trust here detects changes in software, and does not indicate the absence of security vulnerabilities in software. The point of whitelisting is not to prevent insecure software from being unintentionally loaded onto desktops through an authorized software distribution process. It strives to prevent software (whether secure or no) from being loaded on your desktop in an unauthorized manner. Whitelisting makes sure that the assumed good software stays good, and keeps out the unknown and potentially malicious, software. In essence whitelisting is about maintaining a known software state, and implementing authorized change from one known state to another.

Whitelisting therefore requires a repository of trusted software, which Starnes refers to as a collection of platinum images (you can pick your favourite precious metal or stone).

So while blacklists require a signature database and other contextual information for assessing potential malware, whitelisting also requires a repository for proper functioning. The difference is that whitelisting mainly performs comparisons between software to be executed and its respective repository image (a simple check), while the blacklisting database is used to scan and assess the security of the software in question (a more difficult operation).

The size of the whitelist repository grows as a function of the software base supported, while the blacklist database grows in proportion to the amount of known malware - which we have just seen is increasing at an unchecked rate. Chase the infinite or confirm the finite?

While the whitelist model is compelling, in practice it requires significant work to deploy. Assessing and creating the initial list of approved software is a significant task, and would be greatly assisted by existing CMDB implementation efforts in a company. Also, the success of whitelisting is wedded to its tight integration into the software patching and upgrading process. The repository will be in constant flux since software itself is in a constant state of (approved and required) change.

The Way Forward

Not unexpectedly, there is quite a bit of hype around whitelisting, mainly concerning its magical powers to cure security conundrums such as zero-day attacks and endless patching cycles for example. But it will do neither of these two things. Whitelisting does not prevent developers from coding buffer overflows into applications, nor does it prevent malicious attacks from exploiting them. Developers will still require security coding education, and identified vulnerabilities will still require patching.

Nonetheless, the major vendors agree that we will require blacklisting in some form, but whitelisting may become the new leading actor. Bit9 (a whitelist provider) and Kaspersky (a blacklist provider) have teamed up to provide a hybrid consumer solution, where software is scanned only when it is not present on a whitelist. This is not quite whitelisting as intended but it represents the first step in a gradual integration, and more importantly, a way to preserve the blacklisting revenue model. One way or another, whitelists will be coming to a desktop near you soon.

You can find the research used to produce this post as a FreeMind mindmap rendered into Flash here.

Related Posts

• Risk Factors for AV Scanning