Patch, Mitigate or Retire: How I Triage a Critical Vulnerability

What one alert turns into

A serious vulnerability gets announced and, within an hour, everyone is asking the same question in different words. Security wants to know which applications are affected. Engineering managers want to know how much of the sprint it will take. Leadership wants a date. Developers start digging through dependency trees, and the infrastructure team starts checking servers and container images.

Sometimes the answer is that nothing we run is affected. More often it’s somewhere unexpected, in a library pulled in by another library that nobody has looked at since the service was built.

The fix in the advisory is usually small: upgrade to a patched version. The work around it is what takes weeks. Somebody has to find every system that includes the component, confirm whether the vulnerable code is actually used, work out which version fixes it and whether the application can take that version, test the upgrade, coordinate deployments across teams with different release schedules, and then prove the fix landed everywhere. In an organization with dozens or hundreds of applications, that is a coordination problem much more than a coding one.

A single vulnerability alert turns into weeks of work when nobody knows exactly what is running, who owns it and how hard it is to upgrade. The patch is often a version bump. Finding every affected system, judging the real exposure and shipping the fix safely is the expensive part.

Most alerts aren’t zero-days

“Zero-day” gets used for almost any frightening new CVE, and the distinction matters for the response. NIST’s glossary defines a zero-day attack as one that “exploits a previously unknown hardware, firmware, or software vulnerability.” With a true zero-day there is no patch yet, so the only tools are mitigation and detection. Most of the alerts teams scramble over are newly disclosed vulnerabilities that arrive with a fix. That is a race to apply it before attackers use it, which is a different problem with different tools.

Log4Shell is the example most engineers remember. The flaw was in Log4j, a Java logging library, and it allowed unauthenticated remote code execution. Jen Easterly, then CISA’s director, called it “one of the most serious I’ve seen in my entire career, if not the most serious.” What made it so much work was how deeply Log4j was buried. Plenty of teams had never added it themselves. It came in through a framework, a vendor product or a tool, so the first question for many organizations wasn’t how to patch it but whether they ran it at all.

The PHP years taught the same lesson

Before most of my work moved to Django, Rails and React, I worked on PHP and LAMP stacks, when PHP was how a large share of the web got built. It made dynamic sites cheap and quick to ship on shared hosting, and it picked up a reputation for security problems along the way: SQL injection, remote file inclusion, unsafe uploads and cross-site scripting.

A lot of that came from application code and defaults rather than flaws in the language itself. Queries were built by concatenating user input. Files were included from paths a visitor could influence. The register_globals setting turned request parameters into variables, which made it easy to overwrite a variable the code trusted. According to the PHP manual, its default changed to off in PHP 4.2.0, and it was removed entirely in 5.4.0. When an advisory came out, the useful response was rarely just upgrading PHP. It meant reviewing the applications running on it.

PHP isn’t only history either. W3Techs currently reports PHP on roughly 70% of the websites whose server-side language it can identify, and about 8% of those still run PHP 5, a version the PHP project stopped patching long ago. Every one of those sites works fine until an alert shows up that it can’t take a fix for.

Exposure decides the urgency

Finding a vulnerable version is the start of the analysis. A CVSS score tells you how bad the flaw could be in general. It doesn’t tell you whether the vulnerable component is deployed, whether the vulnerable feature is turned on, or whether anyone outside the building can reach it. An internet-facing service with an unauthenticated exploit is a different situation from an internal tool behind a VPN that only five people use. The internal tool still needs fixing, but on a different clock.

Two public sources do a lot of the sorting. CISA’s Known Exploited Vulnerabilities catalog lists vulnerabilities confirmed to be exploited in real attacks. If a CVE is on it, the safe assumption is that attacks have already started. FIRST’s EPSS estimates the probability that a vulnerability will be exploited in the next 30 days, which helps separate the many critical scores from the few that attackers actually go after.

Compensating controls are where my early work still shows up. I started out writing Cisco router access control lists and supporting a managed firewall and VPN service in a Security Operations Center, and I wrote about that in security at every layer. A firewall rule doesn’t fix vulnerable code, and a vulnerable application behind a firewall isn’t automatically safe. But a rule that keeps the vulnerable endpoint off the internet can be the difference between an emergency deploy tonight and a tested upgrade next week.

Prioritize a vulnerability by exposure and exploitation, not the severity score alone: whether it is in CISA’s Known Exploited Vulnerabilities catalog, how likely EPSS says exploitation is, whether the vulnerable component and code path are actually in use, whether the system is reachable from the internet, whether exploitation needs a login and what controls already sit in front of it.

Patch, mitigate or retire

Once I know the exposure, there are three ways to respond, and the right one depends on the system as much as on the vulnerability.

Patching is the default, and for a maintained application with a decent deployment pipeline it should be routine. The trouble starts when the patch isn’t a patch. A common pattern goes like this: the fixed version of a library needs a newer major version of the framework, the framework needs a newer runtime, and the newer runtime breaks two other dependencies. A security fix becomes a framework upgrade, with regression testing for the whole application. That is the point where technical debt turns into a security problem, because years of deferred upgrades all come due on the attacker’s schedule instead of yours.

Mitigating means reducing exposure without changing the vulnerable code: disabling the affected feature, restricting network access, adding a WAF rule or a configuration change from the advisory. It buys time and it is often the right first move while a proper fix is tested. It doesn’t remove the vulnerability, so it needs an end date.

Retiring the system is the option teams forget to put on the table. If an application is already being replaced, spending weeks upgrading it can mean building something that gets thrown away. CISA’s guidance for federal agencies covers this case directly: for a known exploited vulnerability, agencies should remove the product from their networks if it is end of life or can’t otherwise be updated. Outside government, the same logic holds. Sometimes the safest fix is to finally shut down the system that should have been replaced already, and move its last users onto the new one sooner than planned.

What I won’t accept is “it’s being shut down soon” as the whole plan. Shutdown dates slip. A system that is three months from retirement has a way of still running a year later, and the vulnerability doesn’t care about the roadmap. If retirement is the answer, the system gets mitigated in the meantime and the shutdown gets a real date with an owner.

Writing down an accepted risk

When the decision is to delay a fix, I want it written down, short enough that people actually read it:

• Which systems are affected and what they expose
• Why we aren’t patching now: the cost, the regression risk or the planned retirement
• What mitigation is in place, and how we know it works
• Who accepted the risk. That should be someone who owns the business consequence, not only the engineer who found it
• The date it expires, and what would end it early, such as the vulnerability showing up in the KEV catalog or a working public exploit

There is a big difference between accepting a known risk on purpose and not knowing the risk exists. The record is what turns the second into the first. On public-sector systems this isn’t optional, because an unpatched dependency affects the system’s authorization to operate, and I wrote about carrying that kind of obligation through a replacement in modernizing public-sector applications.

Making the next alert smaller

The organizations that handle these alerts well have usually done the boring work beforehand. They know what they run. Each application has an owner and a dependency inventory or SBOM generated from the build, so “are we affected?” is a search, not an investigation. End-of-life dates for runtimes and frameworks are on a calendar before they arrive. And the deployment pipeline can ship a one-line dependency change to production the same day, because small changes and frequent deploys are already normal. That last one is the same habit I push for in everyday delivery, and it pays off most on the day nobody planned for.

Major vulnerabilities also tend to get things done that should have been done anyway. Old dependencies get upgraded, unused services get turned off, firewall rules get reviewed, and the application everyone meant to retire finally gets retired. Those are good outcomes, and I would rather get them from routine maintenance than from an emergency. I don’t think every application needs the latest version of everything. I do think every application needs an owner, a known end-of-life date and a realistic plan to either maintain it or retire it.

Short answers