In Chapter 9, I discussed the importance of having team members who possessed both technical and interpersonal skills. I made this claim because those on your team who are able to walk the halls of your organization spreading the good news of InfoSec while also participating in technical meetings are the individuals on your team who really move your program forward. Once you have your people in place, this next step is about measuring your progress toward a particular goal or objective. This chapter covers the value of metrics, which ones to focus on, and how to use them to improve security for the company.
Why Measure?
If you choose to build your InfoSec program following the seven steps I’ve laid out, measuring your program’s progress along the way will greatly help your cause. Measurement is the tool used to convince management of the progress you’re making year after year.
You’ll use your metrics to highlight to management that their investment in InfoSec is paying off. The front office speaks the language of money. Therefore, you should show in numbers the ROI from InfoSec.
One of the best ways I’ve discovered to show this ROI is to illustrate that the company is more secure this year than last year. The way to do that is by capturing a couple of key metrics. Without the hard data, leadership is just taking your word for it.
The tough part will be convincing them you’re measuring the right items and that these items reflect a mature, self-defending organization. This is your job. The use of metrics shows leadership that the InfoSec program is moving in the right direction (and that you were a wise choice to lead the team).
Understanding What to Measure
So now the question is, what to measure? You have a lot of metrics to choose from. but which ones matter to leadership, especially early in your tenure? If you’ve ever visited Securitymetrics.org, you’ve seen the hundreds of security metrics available. They’re all good metrics. But which ones really matter?
Before deciding what to measure, let’s identify the objective we want to achieve, and then determine how best to measure our progress toward that objective. Based on the previous six steps, our objective is to influence the culture toward individual responsibility for security over the information assets under their control in an effort to establish the neighborhood watch. So which metrics matter when you’re building a self-defending organization? Which metrics capture staff’s ability to partner with you and perform their role to defend the company’s information assets? Those are the objectives we want to measure progress toward.
To measure progress in creating a self-defending organization, two metrics surface. They are closely related to each other. I’ve used these 2 metrics for the better part of the last 20 years, and they resonate well with company leadership because they are simple:
· Metric 1: Can staff recognize a policy violation when it occurs, and do they know how to report it?
· Metric 2: Can staff identify a phishing email, and do they know how to report it?
These two metrics reflect the overall awareness level among staff members, and their ability to report policy violations or suspected security incidents when they happen. Tracking these metrics will indicate your progress toward protecting the company’s information assets and establishing the neighborhood watch. If you train staff on the information security policy that applies to their areas, and measure their ability to respond, then you’re moving toward your objective of influencing the culture toward security. That’s it. These metrics are simple, make sense, and resonate with company management.
These two metrics may seem like the same metric, but they’re not. The first one requires the recognition by staff of policy violations as they occur in their area of responsibility. For example, a system administrator might recognize that a system is not protected by two-factor authentication, and either reach out to the InfoSec team for help or integrate the system into the company’s two-factor tools and processes. This recognition of policy violations requires that they have knowledge of the applicable policies that apply to their responsibilities for InfoSec.
The second metric applies to all staff and is specifically focused on the staff’s ability to recognize and report fraudulent emails. This metric measures the degree of skepticism held by staff as they process company email.
Both metrics require staff training. Metric 1 requires information security policy training relating to their specific area of responsibilities. Metric 2 requires training that enables staff to analyze and identify fraudulent emails that make it to their inboxes. The goal of a phishing program is to raise the level of skepticism held by staff as they approach their email.
The remainder of this chapter focuses on the simple implementation of these two metrics. I provide specific ways you can capture these numbers, the fun you can have with them, and the benefit they provide to other areas of InfoSec policy. I believe that once you see the wisdom in their use, you’ll join me in using them exclusively with company leadership. As I said, I’ve used them for 20 years, with very positive feedback.
Recognizing Policy Violations
Most staff at your company make one broad assumption about InfoSec: they wrongly believe somebody else is doing it for them. Most staff believe it’s somebody else’s job to protect systems, data, and intellectual property. They assume you and your team are “stirring the sauce” to keep things secure. You know this isn’t true, and you must correct this misassumption.
Outside of IT or engineering departments, nobody spends much or any time on system and data security. Again, they assume someone else is doing it for them. They assume security is built into the systems they use. That the network firewalls protect the company. That laptops are secured by tools provided by the IT department. Without education, staff operate with a sense of carelessness toward InfoSec, because they believe security is baked into the systems or processes. They conclude that everything is OK, because somebody else has taken care of it. Most staff members are focused on getting their jobs done, and they assume IT is protecting their systems and data.
To influence the culture, you have to address this assumption and the resulting consequence—namely, that employees don’t view InfoSec as their responsibility. Making InfoSec everyone’s responsibility is the focus and goal of your security awareness and education program; your awareness program will go hand in hand with your metrics program. The two are inseparable.
As staff become more educated on their roles and responsibilities, you influence the culture toward greater degrees of security, and move the organization in the direction of self-defense. But as you educate staff, you have to measure the improvements in staff awareness as they apply to their responsibilities. Employees need routine checks or report cards to see whether they understand their responsibilities for InfoSec. When you start the awareness program, you’ll want to catch the baseline of knowledge among the staff. This will be your starting point.
To move the organization toward the neighborhood watch, you need everyone doing their part, and this means being able to identify and report security policy violations when they happen. My test to understand the staff’s ability to detect policy violations and report them aims to answer this question: is a staff member at the farthest office from headquarters able to identify and report on simple security policy violations when they happen? If they can, you’ve done your job and achieved your objective.
Why do I target the farthest office from headquarters? I assume the staff at headquarters will get a disproportionate amount of awareness training because that location usually has the biggest component of the InfoSec team. Therefore, testing those farthest from headquarters will give you a good sense as to the effectiveness of your education and awareness program.
The Mother of All Metrics: Phishing Tests
Phishing should have a special place in the life of all InfoSec organizations. For more than 15 years, we as an industry haven’t made much progress in the area of phishing. The numbers as reported by Symantec’s Internet Security Threat Report from 2019 showed that phishing was the initial entry point for over 90% of all company breaches. This statistic dropped in only the last five years or so. Since 2015, phishing stats have fallen into the mid-80th percentile, which still makes phishing the most attractive tool for hackers. Conducting phishing tests and phishing training exercises among staff is a valuable exercise.
The few companies I’ve found that do have phishing programs do it to check a box. This is unfortunate. The phishing program, if done properly, offers many benefits to the InfoSec program and to protecting information assets throughout the company. If the industry metrics are true that nearly 85% of all breaches are traced back to a phishing email,1 understanding your staff’s ability to recognize and report a phishing email is a metric well worth your time.
Considering this fact, I suggest you make phishing exercises a top priority. If rogue organizations phish your organization every day, why wouldn’t you prioritize this as well? We can see from the data that having a phishing program is important to an overall InfoSec program. But if this is true, why don’t more security programs focus on it? I believe it’s because this work isn’t sexy to most InfoSec types. Second, it requires the hard work of end-user training. It’s easier to implement a tool instead. However, I believe that your staff’s ability to defend against phishing emails is the key metric to monitor.
Each year, I faithfully read the internet threat reports published by Symantec, Mandiant, Verizon, and Cisco to understand the numbers of our industry, phishing being one of them. The industry isn’t making great progress in this area because identifying a phish remains primarily a human endeavor. It requires the recipient of the email to make a judgment call regarding the authenticity of that email. No tool on the market does this yet. And teaching staff how to detect fraudulent emails isn’t exciting work that most CISOs want to do. I’ve not heard CISOs at conferences standing around bragging about their phishing programs. But mention the latest cool tool from XYZ Company, and they all have it!
As a result of its importance, a phishing metric is always a part of my presentations to the board of directors. When I explain why I capture the phishing metric, every member of senior management and the board gets it. It makes sense. They also get the concept of the neighborhood watch and how phishing relates to that goal. I further explain that the true goal of any phishing program is to raise the level of skepticism among staff when they approach their email. If staff members look with skepticism at each email they open, and they know how to identify the markers of a phishing email, then you’ve achieved your goal.
I use the company’s phishing program along with some key pentests and red team exercises to drive and assess the company’s overall maturity in InfoSec. My goal is to phish the entire company every day, and to achieve a company-wide failure rate of less than 3%. I track every staff member’s scores, and focus additional training on what I like to call the “repeat unrepentant felons” who click everything that comes to their inbox. Your company has them too. For whatever reason, they click all emails, then open everything. Once you identify this short list of staff members, you can offer them more training or place additional security tools on their laptops to compensate for their tendencies.
The details of how to run a phishing program are outside the scope of this book, but you can do many creative things with the program that provide many benefits to the company, the staff, and its general contribution to the overall security posture of the company. Needless to say, the money I devote to our phishing program is some of the best money I spend.
Social Engineering and Staff Training
Back in the early 2000s when Kevin Mitnick was enjoying some notoriety, I hired an outside firm to run an anonymous social engineering assessment. I asked this firm to call 500 employees posing as an IT support person attempting to solve a network problem that required their password to fix. The results were not good. When staff were asked, by an unknown caller, to provide their user ID and password, nearly one in two gave up that information without hesitation; 46% of our employees gave their login ID and password to a stranger on the phone! Yikes. With two phone calls, any intruder could have access to our corporate network.
What did I do? I started a security awareness campaign. It was a blitz, really. It was like dropping leaflets from helicopters across the company. We handed out mouse pads and pens with little security reminders on them. We set up a table at every company event to hand out security swag. We held classes to teach employees about securing their home PCs. The lobby of every building had bowls of fortune cookies, with cute security messages instead of fortunes inside them. We made presentations at every all-staff meeting that would have us. We developed online courses that were entertaining and lasted for less than five minutes. We held lunch-and-learns and provided tips on how to buy a home computer. You name it, we were doing it, and it really paid off.
I hit security awareness hard for a few years, and at the end of each year we brought that same consulting firm back to dial a new set of 500 users. Each year, the numbers dropped dramatically. The responses from staff members were also noted. After the second year of our awareness program, we had staff members asking the caller for their names and phone numbers! We also had staff members telling the callers it was against policy to share their password with anyone! Some staff were even noted to slam down the phone on the testers.
After three years, only 4% of the employees gave up any type of sensitive information to unknown callers. Throughout this time, the number of calls to our InfoSec hotline skyrocketed. I had so many people reporting security policy violations to the InfoSec team that I couldn’t keep up with the calls. Dropping from a 46% failure rate to a 4% rate in 2 years was amazing. The money I spent on education and training was peanuts compared to what I was spending on security technologies. The difference was staggering.
A quick word of warning, however. When you blitz an entire company with security awareness training and information, you may take an employee culture known for being helpful and turn it into a group of very suspicious people. Employees will ask questions and be less trusting of requests for information, even when the requests come from within the company.
I was provided this feedback, that as a result of our awareness initiatives, employees had become ultra-vigilant about not sharing information with others over the phone, even when the call originated from internal extensions. The fear was staff were not as friendly as before, and much more suspicious of the authenticity of callers; I believe it’s a worthwhile trade-off given the alternatives.
Technology Versus Training
I’ve introduced you to the key metric I track to reflect progress toward establishing a self-defending company: staff members’ ability to identify security policy violations when they occur, and their understanding of how to report them. This metric reflects your progress toward achieving the neighborhood watch.
The key question is, do your employees know those parts of the information security policy that apply to them, and how to report policy violations when those parts are violated? If the answer is yes, you’ve done your job as a security manager and leader. If the answer is no, you have more work to do in training staff on InfoSec policy as it relates to their job responsibilities. Therefore, simple training of staff on the parts of InfoSec policy that apply to them is key. This will take time for you to understand, but it’s worth the effort.
Once you know the InfoSec policy that applies to the various teams, you must craft a training program to teach those responsibilities. I’ve found the use of humorous video training to be the best tool.
Considering how much companies are willing to spend to keep external adversaries away from their crown jewels, why is there so much resistance to spending a small fraction of that money educating the people who have direct access to those jewels? Think of all the technology you and others within the IT department have implemented to protect the company. All technology is rendered useless if one employee isn’t aware of their responsibilities to InfoSec in their day-to-day routine. I can’t think of a security control implemented that couldn’t be bypassed in some way through the ignorance of staff members.
Conclusion
My claim is simple: devoting time and resources to educating staff on these two metrics is a great use of your time. Nothing will protect your company’s information assets like a well-educated employee. Not only is awareness cheap, but the ROI is staggering. No other InfoSec expenditure pays back like a few dollars spent on awareness does.
I believe the job of every InfoSec group is to influence the company culture toward greater degrees of security, as the company allows you to do so. If you can begin to measure and pursue the simple metrics laid out in this chapter, you and your organization can make a significant leap in reducing the attack surface of your environment, without buying a single piece of hardware or software. A well-trained staff member is your best defense, with the neighborhood watch as your goal.
1 See “Verizon Says Phishing Still Drives 90% of Cybersecurity Breaches” at the Graphus Blog.