Investigating False Positives and Other Security Anomalies Part 1

Find vulnerabilities on the computers on your network; apply a patch, and all done, right? Well, maybe, and maybe not. Part of any good security program includes using a variety of tools to assess the risks in your environment. Specifically, I am talking about the periodic vulnerability assessments that are performed on the desktop and server computers in your network. Let’s assume you are an all Windows shop for the moment; On the most fundamental level, you get this risk assessment done for you every time you visit the Windows Updates site on the Internet. The Windows Update site uses a scanning engine to determine what is installed on your computer, what the most current patching levels are, and whether or not your computer has those patches. Same with your antivirus software – you are given the latest updates based on the most currently known threats and whether or not you currently have the definitions for those threats (in this case viruses).

Unfortunately, Windows Updates and your antivirus software aren’t the final and definitive answer about the status of your computer’s security level. The same is true for any other single security assessment tool. In a large enterprise environment, it is often necessary to use a variety of tools to assess whether or not you are protecting your systems. The information from one can often be used to validate or refute the information from the others. In other words, having multiple tools gives you a system of “checks and balances” in determining the total picture. Having multiple tools is also a good way to aid in investigations concerning the validity of security assessment information.

The purpose of this article is to give some definitions of a few types of vulnerability assessment tools, discuss definitions of types of vulnerability indications, and discuss situations where investigations may reveal a different story than what was originally thought to exist. It is important to understand that vulnerability scanning and other assessments aren’t necessarily straight forward or cut and dried insofar as the information they can tell you about what vulnerabilities exist. You often have to rely on your skills as an investigator to uncover the real story and figure out how to truly remediate the situation.

First some definitions: It is useful to understand the types of vulnerability situations that may be incurred at any given time. The true existence of a vulnerable item or configuration must be known in order to remediate or mitigate the vulnerability. It is also important to understand the different types of tools used to obtain vulnerability information. To get an idea what I mean by that, take a look at the following definitions:

Patch Management System: A system, usually centrally managed, that is used to assess patch statuses on end systems, determine which patches are applicable, and which patches need to be applied based on patching levels of the target system. Such a system can then be used to deploy the Patches To the end nodes, and return a follow-on assessment of whether or not the patch successfully applied. Windows/Microsoft Updates and Microsoft Baseline Security Analyzer (MBSA) are fundamental examples of this. Large enterprise environments may choose to use Microsoft WSUS or SMS, or third party tools such Shavlik, Ecora, or PatchLink. These types of systems usually look at what patches are needed based on what operating system and software are installed.

Vulnerability Scanner: These types of tools are a bit different in scope and purpose that patch management systems. Whereas patch management systems look for needed patches based on what is installed and operating on the system, a vulnerability scanner usually looks deeper for the existence of certain files and registry entries, whether or not the files and settings are actually used. Scanners can also look for other vulnerable configurations, such as too many admin users on the box, passwords that don’t expire, and similar items. Vulnerability scanners typically have scanning profiles that are based on CVE data and other vulnerability definitions. These profile definitions usually tell the scanner to look for the existence of certain file versions and date stamps that are known to be vulnerable. It doesn’t matter whether or not these files are actually in use, or are just left over files from an upgrade. If they exist, the computer is vulnerable. The saying often heard in the security community: “If the bits are on the box, you MUST remediate.”

True Positive: This means that a vulnerability item has been found, and it is correctly identified as existing on the computer. This is what is commonly referred to as a “known/known” situation.

True Negative: One or more vulnerabilities that were being tested for were not found on the target machine, and they were correctly ruled out from existing on the machine. Again – this is a situation of “known/known” data.

False Positive: Vulnerability was detected on the target, but investigation reveals that the vulnerability was incorrectly identified. The danger here is that this may be ignored in the future, even if the vulnerability detection later reveals a true positive situation. The other danger is that something is an evident false positive, but investigation reveals that another condition exists which caused the false positive to occur. Is this, then, really a false positive? More on that later.

False Negative: A vulnerability condition exists on the target machine, but the vulnerability assessment tools failed to identify it as being present. This is the most dangerous of all situations – your nodes are vulnerable, but you don’t even know it.

For the reasons alluded to in the definitions above, it is necessary to use a combination of these tools to get a true picture of an end node’s vulnerability status. Multiple tools that agree on vulnerability information can leave you with a pretty good level of confidence about whether or not vulnerabilities exist. If the tools don’t agree, however, then that should immediately cause you to launch an investigation to determine the true status. Unfortunately, the false negative situation may still exist. As mentioned before, this is truly the most dangerous situation where your vulnerability status is concerned, because you really “don’t know what you don’t know.” And your tools may simply all be in agreement that a vulnerability does not exist when it really does. Fortunately, this particular scenario in which all tools agree on a false negative is very rare.

False Positive? For an example of how a seemingly “false positive” situation exists that is worthy of further investigation, consider the following scenarios:

Scenario 1: A computer on the centralized patch management system shows that it is completely patched and up to date. Even a visit to the Windows Updates site reveals that the computer is up to date – no critical patches are offered. A later scan with a vulnerability scanner reveals that a vulnerability item exists on the computer. An in depth investigation includes using multiple tools to verify the vulnerability – the patching tools once again all say that the computer is patched, the vulnerability scanners again all say that it is vulnerable. Further investigation leads to a review of the session logs generated when the vulnerability scanner performed its scan. The logs reveal that a particular DLL file exists in the \Common Files\Office10 folder of the computer. You immediately say “Wait a minute!” because you know that the computer is now running MS Office 2003, which uses the Office11 folder for its files. The real story is that the computer was upgraded from MS Office XP to MS Office 2003, and the vulnerable DLL file is just a leftover. Here it is a seemingly false positive, but really and truly, the bits are on the box – the box is vulnerable. “If the bits are on the box, you must remediate.”

In the case above, testing revealed that simply renaming the file removed the vulnerability. This is done, by the way, in case doing so breaks an application, and roll-back is needed to troubleshoot. If it turns out that this file is indeed needed by some other program, then a non-vulnerable version can be obtained from the vendor and the vulnerability will be resolved. Exploits often target a file of a specific name, and if the file is not found (even if it is renamed) then the exploit won’t be effective.

Scenario 2: Your patch management system says that your computers are all the way patched, good to go (hint: all the way patched for the current software installation). Your vulnerability scanner then says that the computers are vulnerable for several MS Office patches. You double check, and sure enough, the patch management system says that those patches are applied. A visit to Windows Updates offers you no critical patches. So you dig out your MBSA tool and do a few scans. MBSA even says that those particular patches are applied. But wait a minute: You look further into the MBSA scan data and it reveals that a service pack for MS Office has not been applied. You apply the service pack. You perform another vulnerability scan – the office patches are still needed. Now you go to the Windows Updates site and your centralized patch management system, and wouldn’t you know it? Those patches called out by the vulnerability scanner are now needed. You apply the patches and your are now clean.

How did this happen? Remember: I gave you the hint in the beginning of the scenario that the computers were patched for the software that was currently installed. Installing that MS Office service pack significantly changed that installation and the old patches that were installed only applied to a computer that did not have the latest service pack. There were newer versions of those same patches that applied to the latest service pack. You applied these new patches, and then all of your assessment tools show that you are now clean and fully up to date.

Wrapping it All Up:

Vulnerability assessments are a vital part of your security program, and often involve using multiple tools to be effective. One tool will act as a system of “checks and balances” against each of the others. It is easy to get lulled into a sense of false security if only relying on one tool – it is even possible that all of your tools won’t find all of the problems. Remediation is not always straight forward either. Applying the patches doesn’t fix everything, and sometimes your system is vulnerable for things that can’t be patched. In some cases, changing a configuration by simply applying a service pack drastically changes the picture.

It is often necessary to rely on investigative skills and go in directions that are seemingly irrelevant.But by relying on multiple tools, performing sound investigations, and keeping up with due diligence, it is possible to minimize risk on your network and keep the threats somewhat in check.Remember: You will never eliminate risk completely.You can only hope to minimize it.But by having valid data from a variety of sources, you can make prudent risk assessments and ensure that your environment is as secure as possible.

On to Part 2

Additional Resources:

Microsoft Baseline Security Analyzer: http://www.microsoft.com/technet/security/tools/mbsahome.mspx

Vulnerability Scanners Explained:
http://www.windowsitpro.com/Article/ArticleID/43888/43888.html

Free Vulnerability Scanning Tools:
http://netsecurity.about.com/od/vulnerabilityscanners/Free_Vulnerability_Scanning_Software.htm

Retina Single Audit Scanners:
http://www.eeye.com/html/resources/downloads/audits/NetApi.html

Foundstone Free Scanning Tools:
http://www.foundstone.com/

The Dirty Dozen: 12 Ways to Kill False Positives:
http://www.bcs.org/server.php?show=ConWebDoc.9384