Recently “Derek A.”, an infosec manager at a Philadelphia hospital, posted a provocative post on LinkedIn with the headline: 🛑STOP DOING PHISHING SIMULATIONS ON YOUR USERS!🛑.
I love controversial opinions… mostly because they spur conversations, help us deepen our understanding, and grow our first principle thinking around a subject. This one doesn’t disappoint.
Rather than hop on the “this is bad advice” or “I totally agree” bandwagon with personal anecdotes, this feels like an opportunity to share real data on the subject. While I do have opinions on how best to run these programs (and lots of experience to back it up), I think a more interesting take is to present hard data and research on the billions of data points we’ve reviewed here at Elevate Security (with a bit of my personal 2 cents at the end).
In this post we’ll cover the following:
- Do phishing simulations work?
- Do they help prevent real-world attacks?
- Is there a better approach?
We’ll keep this phishing focused, but Elevate has data across many more dimensions.
Do Phishing Simulations Work? Yes, No, Maybe…
As a backdrop, Cyentia Institute recently released a research report which mined the billions of datasets we’ve pulled into our Elevate Security Platform. We have visibility into phishing simulations, email security tooling, web gateways, identity solutions, data loss prevention tools, etc. If an employee is doing something, good or bad, we can see that. This is tremendously powerful, because our customers enjoy insights into workforce decisions and can begin to see if there’s correlation across their tooling. This allows them to see whether efforts such as phishing simulations, training or other work is having the impact they’d expect, such as whether it’s reducing employee mistakes, decreasing business risks such as ransomware, and more.
To answer the question, “do they work?” we legitimately need to align on, “for what purpose?” and “to what end?” Without knowing the organizational goal, it’s all too easy to have significantly polar opposite opinions on their efficacy.
Let me share some data, regardless of the answer to those questions. First, phishing simulations do help to reduce click through rates of phishing simulations and do help improve reporting.
But, there are two super important caveats to this:
- Running phishing simulations helps reduce click through rates of future phishing simulations (more to come on this…)
- Phishing simulations help increase reporting to your security team
- After about a year of simulations, results stabilize (we see about a 5% click through rate ongoing)
The last point is important. Clearly we’re not seeing much improvement at all after the first year. Anyone expecting more from their phishing simulation program is going to have difficulty making huge leaps forward beyond a status quo.
The smaller the organization or department, the more likely the organization is going to have data all over the map. This isn’t that surprising, as smaller companies will generally have less mature security programs. Results do tend to stabilize as company and department size grow.
However, we do have to realize that no matter how much we invest in phishing simulations, we should set our expectations appropriately. While we can drive individual click through or compromise rates down in the short run, at the end of the day, some people will always click, most employees will eventually click, and at an organizational level someone will always click (Figure 1). Yes, that’s 100% of the time!
Do Phishing Simulations Help Prevent Real World Attacks?
This is the most interesting question. You may have noticed how I worded an important point above:
- Running phishing simulations helps reduce click through rates of future phishing simulations
That doesn’t conclude ANYTHING about true phishing attacks in the wild. We’re not running phishing simulations just to help people improve on phishing simulations. We’re running them with the expectation that this translates to reducing clicks and compromises in the real-world.
Our early research on the topic has shown that reducing click through rates on phishing simulations does NOT in fact correlate back to reducing click through rates on real world phishing attacks.
This could mean a number of things (forgive me, I’ll hypothesize a bit here because the data isn’t abundantly clear):
- We’re training employees on how to detect phishing simulation emails (each platform has a few giveaway signs)
- We’re not sufficiently matching the sophistication of real world attackers
- We’ve done a poor job of understanding employee context in our controls protecting employees
I want to dive into that last point much more, because it’s so important to unpack.
Is There A Better Approach?
The root of the challenge in answering whether phishing simulations work or not is that we don’t have a strong enough context and understanding of individual users to truly protect them. For example, could you answer the following questions:
- Which of your users are most at risk of having their accounts compromised, downloading ransomware, losing sensitive data, etc.?
- Have you deployed any controls proactively to better protect those at risk?
- Are you measuring the efficacy of the program you have and the controls, technologies and training you’ve rolled out?
The answer for most folks on all of these questions is, “No”. If we can’t answer these questions, we lose insight into the two original questions I asked about whether phishing simulations work?
To what purpose, and to what end?
We can’t lose sight of the fact that our ultimate goal is to reduce the likelihood and impact of these types of attacks in the real world, in this particular case social engineering. With that, the only way to truly move the needle is to start with measuring and understanding that risk. Real world attacks in particular, because we’ve seen that phishing simulation results do not correlate well to real life.
Once we have an understanding of that risk, we need to do something about it to help truly reduce that risk. It’s helpful thinking about it from these perspectives:
- Employee & Executive Feedback – real-time feedback to employees and executives on the impact of their security decisions and their effect on overall organizational risk
- Decision Support – leveraging employee risk visibility to support internal staff decisions and automating decisions that impact your workforce risk
- Tailored Controls – proactively automating “just right” controls for your employees based on their real-world risk
Taking this approach moves from a reactive and disconnected approach to a proactive and connected security budget.
I invite you to explore our solution to see how Elevate Security moves beyond the limitations of phishing simulations to truly reduce an organization’s human attack surface.
