Training 20+ TAC Engineers on a Security Service I Was Still Learning

The first training course I ever built was for a service I had only just learned myself. I was one of the first three people supporting a new managed security service in a carrier’s Security Operations Center (SOC). As the service grew, the Technical Assistance Center (TAC) had to be able to support it without us. I helped train more than twenty TAC engineers to do that, and afterwards I supported their escalations.

What was the SOC supporting?

The carrier sold managed firewalls and IPsec VPNs on NetScreen hardware to its business customers. Supporting the service meant taking change requests, maintaining security policy on each customer’s behalf, and troubleshooting connections that crossed networks we didn’t run. At first, three of us did all of it, without a runbook.

I had no title for that work and I wasn’t running the SOC. I had never done security operations before. Most of what I knew about the service I had learned in the months before, from the documentation, from the devices in front of us and from customer tickets. I wrote about the technology and those first months in I Was Working on Network Security Long Before DevSecOps Was a Thing. This post is about what came after: handing the service to the people who would support it next.

The hard part was preparing the training

Knowing how to troubleshoot the service didn’t mean I knew how to teach it. I had never prepared a training course. I had a week to fill and a PowerPoint deck to build from nothing, and the questions I kept coming back to were basic ones. What goes first? How much can one slide hold before nobody reads it? How do I explain something I mostly do by feel?

The closest thing I had to course material was what the three of us had written down as we learned. Notes like that are written by people who already understand the service, for people who already understand it. A class needs everything the authors have stopped noticing: why a rule exists, what a healthy tunnel looks like, what to check first and when to stop and escalate. Each of those had to become a step someone else could follow on a call without me.

A week with the TAC

I flew to the East Coast for a week of training with a group of more than twenty TAC engineers. They were the people who would answer the phone when a customer’s VPN went down at three in the morning, with nobody from the SOC on the line.

What they needed most was a way to troubleshoot the service. That meant how traffic moves between zones and policies, what network address translation does to an address, why both ends of an IPsec tunnel have to agree before it comes up, where the session table and logs show what really happened, and the order to rule things out in. A firewall gets blamed in nearly every ticket, so the method had to start with the whole path from client to server.

Their questions found the gaps quickly, in my understanding as much as in our notes. Teaching the service made me learn it again, properly this time.

What did training the TAC teach me?

Be patient with people who don’t have the skill set or the fundamentals yet. A room of engineers rarely starts from the same place, and the people missing the basics are the ones the training most needs to reach.

Some engineers in the room had the networking background to follow a firewall policy the first time through. Others didn’t have those fundamentals yet, and a policy between zones makes little sense until you know how a packet is routed and what an address translation changes. Going faster would have suited the first group and left the second holding slides they couldn’t use on a real call.

Patience meant going back to how the network was supposed to work, as many times as it took, in a week that was already full. Usually the gap isn’t the trainee’s fault. The material assumed knowledge it never taught.

Taking the escalations

After the training, the TAC took the customer calls and troubleshot them first. The cases they couldn’t close came to me as escalations.

An escalation tells you something about the training as well as the customer. When the same kind of problem keeps coming back, either the material didn’t cover it or the fundamentals underneath it are still missing. Either way, the fix is more teaching, and fixing the ticket alone leaves the next one on its way.

That’s still how I think about mentoring engineers. A support organization that depends on the two or three people who were there at the beginning is a single point of failure. The aim is a team that handles the problem without you, and the escalations show how close it is.

Short answers