Why Patch Management is a Moving Target

Whether you use a centralized patch management system for your organization or rely on less sophisticated measures such as manual patching, you will often find that patch management is a constantly moving target. Patch management is a fundamental security task, but yet it seems to be one of the hardest in which to achieve a consistently high security “score.” And patching is only part of the issue. Reporting metrics that allow you to see tangible results are not always easy to obtain.

The data required for many measures of Cyber-security health or “scorecards” is often more readily pulled from a centralized system, and often organizations wish to report how many nodes have 100% of their security patches installed. This includes possibly a very large number of devices, depending on the size of your organization, including servers, desktops, routers, and switches. It also makes sense to focus efforts only on patches that are 30 days old and older, and that have not been superseded or replaced. The reason for this is because you are still testing the newest patches, and if you are in a very large organization have not yet had time to fully deploy all the new patches. Additionally, the older the patch, the greater the risk if that hole is still not closed. Plus, it doesn’t make sense to include patches that have been superseded by newer patches, as your scoring metrics would give erroneous results if those patches were included.

The purpose of this article is to address an example of an organizational patch status improvement effort and illustrate findings of an experiment to improve these patch statuses. This project was specifically aimed at the Microsoft specific security patches for the Windows XP Professional operating system, and all references to patches in this article will be geared toward those patches. This report will discuss the findings of that project, and describe ways that other programs can use to improve their own patching statuses. The results and recommendations are geared toward a fairly large organization. Your mileage may vary.

The Project:

I and my team recently performed an analysis of patching statuses to determine how to improve patch statuses for our entire organization. We belong to a fairly large organization with many business units, each business unit having their own IT staffs that manage their computers. One of the goals of this project was to look at ways to improve patching statuses and to document specifics concerning any anomalies that were found. By pushing out specific types of patches, and analyzing the results of those patch deployments, I was able to put together some strategies to help others with their own patching efforts. The analysis of patching efforts was performed as follows:

1) Concentrate on the "low hanging fruit" by focusing on the Microsoft critical security patches that are greater than 30 days old, and with the highest incidences of "Not Patched" statuses. This was done in phases:

Phase 1: Push out the outstanding Microsoft Office Service Packs plus Windows Defender signature files. These were chosen because they represented the largest number of un-patched vulnerabilities in the environment, as seen in the image below:

Phase 2: Push out the new Office patches offered as a result of Phase 1 completion above. This is important because applying a service pack will usually result in the computer needing additional patches that apply only to the new service pack level on the machine.

Phase 3: Push out the top 5 "not patched" patches resulting from phases 1 and 2 above.

Phase 4: Push out any remaining patches as appropriate.

2) Identify patches that are deploying successfully, but are not showing as "Patched" in our patch management system. This will include verifying that the patch is applied by using Microsoft Security Baseline Analyzer (MBSA) and Windows/Microsoft Updates.

3) Compile a list of patches that are having deployment detection signature problems and submit to the patch management system engineering for assistance with detection signatures.

4) Identify computers on the patch management system that are not checking in to get their patches. This will include looking at deployment reports to see which computers have not checked in no more than 24 hours after the deployment has been sent.

Findings:

As shown in the image below, patching statuses tend to fluctuate dramatically from day to day. This can be caused by machines falling in and out of patch status due to new patches being released to replace older versions of the same patch. For example, the Windows Defender DAT files are released approximately every three days. Rather than releasing a new patch each time, our patch management system simply replaces the existing patch with a new revision of the same patch. As the new revision is released, the computers fall out of patch status because they have the older DAT files. As the IT staffs push out the DAT files, the patching statuses go back up.

Some more specific examples of patching issues include:

Patches That Change Frequently:

The Microsoft Windows Defender DAT Files: These definition files are released by Microsoft approximately every three days. Since they are categorized as Critical-01 patches, they cause the patch statuses to fluctuate significantly every time they are released, and then again when they are subsequently deployed. This patch was the single largest reason why patch statuses greatly fluctuated from day to day.

Patches That Cause Other Patches to be Applicable:

Service Packs: Once installed, these patches tend to make the computer detect as needing additional patches. Some patches may only apply to a newer service pack level, and were thus not applicable to the machine until the latest service pack was installed.

Patches With Deployment Issues:

Microsoft Office Patches: These patches in particular were found to have a number of difficulties when deployed. In some cases, the patch is deployed, and completes successfully according to the patch management server’s deployment status. Even though the patch deployed successfully, the patch did not apply because it produced an error message that it could not find the Office installation files. In other cases the patch fails, for the same reason as stated above. Ensuring that the installation files have not been removed manually, or through the Disk Cleanup procedure typically resolves this issue. In some cases, it was necessary to uninstall and reinstall Microsoft Office, again ensuring that the Office Installation files are not removed.

Microsoft .NET Framework 1.1 SP1: This specific patch typically fails when being deployed. The reason for failure was found to have been on computers that also have the .NET Framework 1.1 Hotfix (kb928366) installed. The resolution is to go to Add/Remove Programs and remove this hotfix, deploy the .NET Framework 1.1 SP1 patch. The computer will then likely show up as needing the MS07-040 patch. Deploy MS07-040 if needed.

Patches With Detection Issues:

MS08-018 for Microsoft Project: This patch is not supposed to apply to versions of Project 2003 that have service pack 3 applied, but our patch management system incorrectly identifies the computers with Office 2003, SP3 as needing it. This is still an open issue with engineering and will hopefully be resolved soon.

Patch Management System Housekeeping Issues:

If you are using a centralized patch management system, and you are using the various reporting features to obtain your patch statuses, then it is important to take a look at housekeeping. One important thing I found in my testing was that simply deleting stale accounts out of the patch management system increased patch statuses.
The below image is an example of how much difference in patching status can be achieved just by doing housekeeping and nothing else. The patch status for the month of April was taken at the end of the month, and the patch status being shown for May was taken at the beginning of the month after clearing out all the dead computer accounts:

The result was that patching statuses for every business unit (BU) except one improved, with an overall improvement going from 42% to 58% just by clearing out dead computer accounts.

Recommendations for Improving Patch Statuses:

If you are a large organization, use a centralized patch management system. The ability to gather data on the whole organization is vital to enabling you to keep track of gaps in patching efforts.

Make sure that your centralized patch management system is being properly maintained, in terms of housekeeping. Get those stale computer accounts out of there.
Start small. Break your patching efforts into pieces, and go for the “low hanging fruit’ first. Look for the patches where the most computers need them, and start there. If you have a lot of these in your environment, break them up into groups and deploy them over several deployments if needed.

Test, test, test! If you are trying to bring an entire organization up from a dismal patching status, don’t try to push them all at once, and be sure to perform testing to make sure to discover if any patches break anything.

When pushing out service packs or roll-ups, be aware that installation of a patch of this type will often result in additional patches being applicable that were not applicable previously because of the new configuration.

Monitor patch deployments and subsequent detection results. In cases where patches deploy successfully but detect as still not patched, check to see what error messages are occurring during the deployment. In the case of Microsoft Office patches erring out, for example, ensure that the Office installation files have not been inadvertently removed from the computer.
Develop a patching routine and communicate this with your end users. Get them used to the fact that you will usually be pushing out patches the same night of the month (if they are in your central offices) and to leave their computers on that night. For remote users that receive their patches through your centralized patching system, make sure they are aware that patches will be coming to them on a certain day and give them instructions for how to properly receive the patch:

Example:

When coming in through the corporate VPN to replicate email or other databases, ensure they leave the computer on long enough to receive patches on the day you deploy them.

Other Follow-up Action:

Remember: Patch management is not something that you do once to get caught up then forget about. You have to treat patching as a constantly moving target, and always follow-up on patching efforts. Get into the habit of always keeping an eye on patch statuses and results of patch deployments.

Determine if an application is the mandated or authorized solution to be used. Sometimes you find that you are chasing patches for products that are no longer in use or maybe even not even authorized on your systems. Why patch a product that isn’t even needed? Removing it is more secure and less time consuming than patching it.

Continue to monitor patching efforts and publish lists of those patches which remain as the most likely to be causing degraded patch status.

Assist IT staffs with troubleshooting computer detection, discovery, and patch assessment issues that may exist. It could be that the patch assessments on a certain machine are out of date and not even accurate.

Monitor patch management and security discussion forums such as the patchmanagement.org listserv. If a particular patch is causing breakages or deployment issues, this is where you will find out about it the quickest.

Wrapping It All Up:

Getting a handle on patching statuses can be a real challenge for a large and geographically dispersed organization. A centralized patch management can greatly assist your efforts, particularly if you are in a large organization. Break your patching effort up into phases, and go for the “low hanging fruit” to get caught up. Be sure to continuously monitor deployments and patching statuses, and address issues where the deployments are not starting as they should, or the patch is not detecting as it should.