What is patching?
Keeping your software up to date is one of the most simple and effective steps to take to ensure your environment stays secure.
A common tactic for attackers is to exploit known vulnerabilities in software that an organisation has not yet updated.
Software owners often release patches to reduce the impact of vulnerabilities or remove them. Applying these patches in a timely manner remains the number one critical control for organisations to keep their IT systems secure, and to protect themselves from being breached.
Modern IT environments are complex, with interconnected components and software. Attackers take advantage of the interconnectedness to join multiple vulnerabilities together to get into business systems. To combat this, patching becomes critical to reduce the number of issues and vulnerabilities that could be used against you.
It’s important to include internet-of-things (IoT) devices as part of your patch management strategy. This includes any smart devices that have support for internet connectivity. These devices are part of your environment and network and can increase the attack surface for your organisation.
How to implement patching across your organisation
Each environment is unique and can be complex. There are several things to think about when attempting to implement consistent patching across an organisation.
The sections below point out the different steps you should be taking when it comes to implementing patching.
Understand your environment
Understand what software and firmware you have in your environment, so that you know what needs to be patched. This allows you to know which patches are applicable to you and what the impact would be if a vulnerability is exploited. As a company grows, it gets harder to track what is within the environment.
Configuration management databases (CMDB)
For larger organisations, a CMDB can be helpful and used to give you a granular view of the components in your environment – from the hardware and devices all the way down to the software versions they are running.
CMDB are often populated by hand – which means they are often out of date or simply have incorrect information. Having automated reporting to ensure that the CMDB is always up to date is very valuable.
Having a central repository makes it easier for this data to feed into other processes, like patching or incident response.
Even without a CMDB, an organisation can still benefit from using tools that scan and collect data about the environment. Software discovery and inventory tools, such as software inventory in Windows System Centre Configuration Manager (SCCM), help you understand what software is running in your environment.
Scanning and discovery tools
Vulnerability scanning tools, such as Nessus or OpenVAS, help you identify which vulnerabilities your environment is currently exposed to. It also provides patch references for those that have been resolved by the vendor.
Scanning and discovery tools can only help when a device or hardware is connected to your network, and reachable by the scanner. The software status of segmented networks, or devices that are regularly taken off the network such as laptops and other mobile devices, can be hard to capture.
If staff can use their own personal devices in your environment (bring-your-own-device), you may be able to discover these devices, but you will have little to no control over the software patches that are applied.
Mobile device management (MDM) tools give you more control over these untrusted devices by enforcing security policies and patch levels. If you don’t use an MDM tool, have a clear agreement and policy with your staff around patching their devices. If staff don’t patch, be prepared to ring-fence these devices on a separate untrusted network until they are patched.
Legacy software
Hardware and devices may be in a position where they can’t be patched due to hardware limitations. For example:
- a Samsung Galaxy S III that can’t support the latest versions of the Android operating system, or
- a legacy application that needs to run on a Windows 2003 server.
The legacy software on these devices and hardware won’t be able to be patched and pose a bigger risk to the organisation.
Know when updates are released
Vendors typically send notifications out to their mailing lists when patches are released. Some vendors have clear patch release days – like Microsoft’s Patch Tuesday. Once you have an idea of the software you are running in your environment, make sure you have a clear and central way to receive patch notifications.
Send these notifications to a central location to review and filter. This will prevent urgent or important notifications from being lost in the inbox.
Keep track of the vendors who have regular patch release dates and consider these release dates as part of your patching process.
Vendors may release emergency patches outside the regular release date so it’s a good idea to sign up to their mailing lists and notifications.
For an additional step, you can subscribe to data feeds for vulnerabilities. The National Institute of Standards and Technology (NIST) in the United States provide multiple data feeds, including an RSS feed. These provide updates on recently announced public vulnerabilities. The solution to a vulnerability may include an immediate release of a patch, so it can help to keep track of recent vulnerabilities to stay across emergency patches.
If possible, integrate this information with your CMDB, so you can automatically see which systems need to be updated.
Have a standard process for applying patches
You should have a clear process for applying patches – sometimes patches will come with no warning and will need to be applied as soon as possible. Having a clear set of steps for each standard patch release will make it easier when the emergency patches are released.
Some things you may put into the process are:
- understanding what the patch is for,
- setting a time window to deploy the patch,
- back-ups and when to do rollbacks,
- testing the patch,
- notifying end users,
- verifying the patch is complete, and
- what to do when there’s an emergency patch.
For more detail on what to include at each step see our standard patching process below.
You can configure some software to automatically apply updates when patches are released. We recommend you apply automatic updates where possible. You need to balance the risk between applying a patch that may contain a bug against the risk of including those patches into your increasing patching cycle.
Automating the deployment and testing of patches can help you save time and resources – and win you favours when it comes to discussing the risk of patching to the business. Consider using tools like WSUS or Ansible to automate downloading, applying patches, and running of scripts to test that patches were applied correctly.
Where possible, design and build your systems so that you can patch without outages. By ensuring that patching and rebooting one component will not cause a system outage, you enable your staff to apply updates during business hours, increasing the likelihood of patches being applied. If something goes wrong, the right people are awake and on hand to resolve the issue.
If you don’t perform your patching process regularly and your software starts falling multiple cycles out of date, the risk becomes higher. For example, you may try to make your software current again by installing lots of updates.
Patches aren’t just security fixes – they can also include changes to features and to the actual software experience itself. Because of this, updating several patches at once could have unintended impacts to end users or software performance.
Applying more updates at the same time also increases the duration of the change window. You may find that some updates require previous updates, and you need to perform multiple rounds of updates to get completely up to date.
Highlight the risk of delaying patching
Despite patching being critical, it is often not done. There could be many reasons why, such as:
- there is not enough time,
- there are not enough resources, or
- there are too many other critical processes or issues being handled.
Our recommendation is that all software should always be kept up to date, and organisations should only consider delaying patching in exceptional circumstances.
If your organisation is considering delaying patching a vulnerability, communicate with the business and risk owners so they
understand:
- which system or components are directly or indirectly impacted – including the systems that the vulnerable system communicates with,
- the risk the vulnerability carries – including what an attacker could achieve and what the likelihood of carrying out that attack is,
- the impact to the organisation if the vulnerability is exploited – including downtime or loss of intellectual property or customer data, and
- the organisation’s ability to detect if the vulnerability has been exploited – or other controls that may prevent it from happening.
Based off this understanding, work with the risk owners to determine if the risk is worth delaying a patch. Make sure decision makers have all the information they need to make the best decision.
Keep track of all the patch exceptions the organisation makes. One decision may not have a high impact, however when looking at the aggregate of all the previous decisions, the risk or impact may be higher. Attackers are often able to exploit unpatched components and chain previous, older vulnerabilities to carry out an attack. It is important for the business to look at the whole instead of one patch on its own.
Explaining the risk can be difficult. Try explaining the different scenarios that could happen if a patch is not applied. If there are multiple patches missing, explain how unpatched vulnerabilities could be chained together and introduce higher risks. For example, a cross–site scripting (XSS) vulnerability on a webserver may not be considered high risk on its own. However, this could be chained with other vulnerabilities, like open redirect vulnerabilities, to raise the risk higher.
When discussing the need for patching, sometimes people focus on preferring features to fixes. Try to explain that patches are necessary to get the ability to try new features.
Security fixes can critically change the way software is used, and future features will be built on top of that new model. By falling back and not applying security fixes, this could prevent those new features from being used. It also prevents your IT team from focusing their time and resources on forward thinking tasks as they may have to spend more time monitoring the environment and monitoring those security holes.
Create a standard patching process
A standard patching process means you have standard steps to follow. They also help when you need to deploy an emergency patch.
The following points should be considered when creating a standard patching process.
Understanding the patch
First, know what the patch is for. Read the patch notes and details from the vendor and understand:
- what software or firmware is in scope,
- what the dependencies are for deploying the patch,
- what issues are resolved by the patch, and
- what the known issues are with deploying the patch. Any known issues with no workarounds could affect your business operations. That risk will need to be weighed up against the risk of not applying the patch.
Getting the patch files
When you get the patch files, confirm that the files come from an appropriate source. Before downloading files from a vendor’s website, check the TLS/SSL certificate details and verify that the owner of the domain is the owner you expect it to be. Phishing attacks use spoofed websites or emails to convince users to download and run malicious files. Don’t download your patch files from sources that you don’t share an encrypted channel with – like a website served over HTTP or a file share over FTP.
Often patches are made available through mirrors, or servers that have copies of patch files that may be geographically closer to your server (which can help with download speeds). You should only use mirrors that have signed patch files, so that you know it has from the right and trusted source, or at a minimum a checksum hash.
After you download the file, run a checksum (like a hash check using a hash function from the SHA-2 family) over the file. A vendor should provide you with their hash. Compare the two to make sure you have the right file. If the checksums are not the same, there could be a problem with the integrity of the file. You should contact the vendor to check.
Download your patch files to a central location within your environment (after you have validated the source). For example, use Microsoft’s Windows Server Update Service (WSUS) to download all your Windows software (including operating systems, web browsers, and other applications). This will save you time in your patch change window and help you later when you move to automate the process.
Patch window and timing
Finding the right time to apply updates is a balancing act. You need to find the right balance of ensuring minimal impact to service (consider how you can patch without disrupting service at all), while also minimising disruption to your staff, and their personal time.
Setting a patching window during a low traffic period will reduce the impact to end users. These windows should be long enough to allow the patching to finish.
Consider the following things when setting the patching window.
- Allow time for a rollback in case the patching rollout fails later in the process.
- In case it is required, hold these windows at a time that will allow the component to be rebooted without massive disruption.
- If your company operates in multiple time zones and the low traffic periods are too short, you can consider multiple patching windows and deploying patches in batches.
Backup and rollback
Any time you are making changes to a system; it is important to have a rollback plan. This may involve taking backups before applying patches. Sometimes even after testing a patch successfully, there may be some bugs that are discovered later that disrupt your organisation.
Having a backup to roll back to and having a clear way to make the call to roll back to a backup are important. The decision to roll back should consider:
- what the issue is,
- if the vendor is aware of the issue,
- when the vendor plans to have a fix, and
- if the impact is too high to the organisation to wait for that fix to be available.
Testing
First deploy a patch to a test environment or to a single instance before it is deployed to your entire environment.
Test to make sure that the critical functions the software provides, still operates the same way it did before. This could be:
- patching a gateway and testing to make sure that traffic still flows correctly, or
- patching a business application and testing to make sure the key features still work and produce the same outcomes.
You can’t always test everything – it is usually a best effort attempt.
Vendors put their patches through their own code development and testing process before they are released. When assessing the level of testing that you need to perform before applying a patch to a live environment, consider the amount of testing that vendors perform before releasing patches to the public and the level of risk posed by the software if left unpatched with a known vulnerability.
Notification to end users
Patching should be a regular process, and most end users don’t need to know when it happens. To reduce notification fatigue, you should only push notifications to end users when emergency patches are applied outside of the regular patch window or when there is an action required on their part (ex: a system reboot).
To encourage a culture of patching, it would be beneficial to have information and guidelines available to end users so they can learn more about:
- how the process works,
- how the impact to end users has been considered (with areas like patch windows and test scope), and
- what vulnerabilities were recently patched.
Verification
After the patch has been deployed across the environment, it is important to verify that patching is complete.
You could have received the notification that the patch file was pushed to a server, and the file was executed – however the server could have failed to reboot, and the patch may not be in effect just yet.
If the patch you deployed was widespread throughout the environment, you can perform this check by using active scanning tools to see if the vulnerabilities have been resolved or if the software has an updated version number.
Emergency
When a patch is released with no warning and outside of a normal vendor release cycle – that usually means it is critical.
You should be prepared to review these type of emergency patches and determine if it is applicable to your environment. If the answer is yes, you should take the patching process that you have now and make changes to it that cuts down the time to deploy and adds to end user notification.
Depending on the severity, you may decide to:
- limit testing and instead notify end users of the severity of the issue, and
- thank them for understanding that their use of the software may be interrupted while patches are rolled out.
The steps above are not just unique to patches – these are typically steps that are followed for other types of changes too. When deciding on a patch process, you can always use your change management process as a foundation – just make sure to include the steps that are somewhat unique to patching (like validating file checksums).
How to measure success
A successful patching control looks different for different types of organisations. They all should have the same end goal and outcomes.
- You have a complete view of the software in your environment. Each day, you know what new software is added and what the current patch levels are.
- All your systems kept up to date with the latest patches.
- You have a comprehensive patch management strategy that covers all software on your systems. It will include:
- identification,
- prioritisation,
- scheduling,
- testing, and
- handling.
- You have a mostly automated patch process which includes:
- automation of the notification,
- identification,
- download,
- verification,
- packaging,
- backup,
- testing, and
- deployment of patches.
- Actions that require assessing risk or making judgement calls remain manual.
- All delayed or cancelled patches are re-evaluated with each patch cycle to assess the risk of not patching vs risk of patching.
- You have supporting documentation and information available to your organisation’s staff to inform them of the strategy and importance of patching.
- Your organisation understands the risk associated with the environment’s current patch levels and works to mitigate the risk.
Key takeaways
- Every IT environment is different and unique. There is no ‘one-size fits all’ when it comes to patching. This is why it is critical to first understand what you have in your environment, and how your organisation operates before you can decide how you deploy your patches.
- Perform regular checks of your IT environment to see what software versions you have. The scan may find software that was missed from previous patching. This could be because some systems were skipped or missed during a change window, or automated patching is not working as intended.
- A common reason for not patching is that there is not enough time or resources. Automating the patching and testing process can help reduce the workload and increase your security.
- Another reason for not patching is to avoid disruption to service. When designing and building systems, design them to be more maintainable, which is both good business practice, and good security practice.
- When components in your environment reach their end–of–life, where the vendor will no longer support it, patches may no longer be released. You will need to consider how you will look after these legacy components and systems.