Introducing the Phishing Difficulty Calculator: How Hard Are Your Phishing Tests?

Masha Sedova

Published on 13 June 2018

You roll out a brand-new phishing test to your users, the results start to come in…and they’re worse than the last test. What happened? You’ve educated them about phishing, regularly alert them of new attacks, yet it seems they’re still not getting it…or are they?

Phishing tests can serve two purposes- education on the latest phishing scams or measuring the effectiveness of your security education program. Before kicking off a test it’s important to understand which of these two reasons is the primary reason for the test. Why? If you are running the test to measure the effectiveness of your security training the score is only useful if you’re measuring like-to-like tests.

As security practitioners, we like to roll out new and different phishing tests to our users reflecting the latest attacks, which is great, but when it comes to evaluating test scores over time, the data may be less than useful if you don’t know how hard or easy each one is. Comparing a hard test to an easy test won’t tell you much about the effectiveness of training. For example, you roll out an easy test and everyone passes with flying colors with a near 0% click-through rate. ? Then, the next month you roll out a hard test and everyone fails. ? How did your security training get worse and your employees more gullible in one month?!

You see, we’ve been trained to focus only on the score, but that’s just half the picture. We first need to know how hard our tests are and then we can make better sense of how the click-through rates reflect that. To improve how you measure phishing scores, you need to do three things:

  • Define what an easy, medium and hard test looks like
  • Measure test scores over time, comparing only like-to-like tests
  • Improve testing over time based on the score data

Having rolled out hundreds of phishing tests throughout our careers and needing a better way to standardize the difficulty of these tests and how we measure them, we created a dynamic Phishing Difficulty Calculatorthat we’ve been using successfully in-house for years. We use this technique to phish employees before and after they complete Hacker’s Mind to demonstrate its effectiveness. We even take it one step further with A/B testing and send the follow-up phishing test to employees who haven’t gone through the training to compare results with alumni.

Today, we’re excited to open up the calculator for everyone to use — and it’s free!

What Is the Phishing Difficulty Calculator?

The calculator incorporates twelve elements of a phishing test that determine how difficult it is to pass. By indicating in the calculator how easy, medium, or hard each element of your own test is, you can find out its difficulty score.

phishing calculator example

For example, if your test has easy grammatical errors (obvious to spot), a hard layout (it’s well-designed), and medium urgency (sent during the workday but no set deadline to respond), you would end up with a difficulty score of around 57%.

Knowing this score, you can determine if the test is difficult enough based on your training and can compare it to a test with a similar difficulty level. You can play around with this formula to see how making certain elements of your test easier or harder changes the difficulty level so that you can find the right balance for the purpose of your training and testing.

How It Works

Using the Phishing Difficulty Calculator is easy. Simply mark in column G the difficulty level of each of the twelve elements of your test and watch the grade at the bottom of the column update in real-time.

phishing calculator grading example

To determine how the difficulty criteria for each element of your test, look at the descriptions in columns B, C, and D.

Example of phishing calculator

You’ll see the calculator evaluates elements including:

  • Personalization (correct name and title)
  • Layout (well-designed or not)
  • Day and time of the test (sent during the weekday, on a weekend, or late at night)
  • Urgency to respond (a set deadline or not)
  • Spelling or grammar errors
  • And more

There is also an Importance score in column E, which weighs each element differently based on how important it is for a user to spot. For example, if the test includes a person’s correct name and title in the email, that’s much more important for a user to detect than if there is a single misspelling. The more accurate a mock phishing email is, the trickier it will be for a user to spot it, requiring them to truly exhibit security awareness in order to spot the phish.

You can customize the importance score based on what’s most important to your organization.

Find Out Your Difficulty Score

Curious to know how difficult your phishing tests are? Create your copy of the Phishing Difficulty Calculator to find out! Then, as you roll out new phishing tests, you can quickly calculate their difficulty level, better compare test scores and truly measure progress over time.

To begin using this test for yourself, simply click File > Make a Copy to save it to your own Google Drive and begin filling in column G and customizing the importance score yourself.

Try the Free Phishing Difficulty Calculator Today!