In Chapter 5 we look at how to create a metrics program that will help you measure the performance of your entire organization and determine what to report to your management and your board of directors. Each of the authors has a bias toward objective measurements and sees that as key to fulfilling the role of the trusted authority on your organization’s risk posture. They collectively emphasize the value of using widely adopted security frameworks to create a comparable baseline from which to measure improvement and extoll the virtues of being disciplined in the performance of preventive and periodic controls.
Bill begins with a brief historical review of tying measurement to business objectives and briefly discusses the evolution of control coverage to measuring the impact on service delivery. He provides several recommendations for frameworks you can use to establish your baseline. To conclude his section on measuring process effectiveness, he offers a helpful set of principles for deciding the metrics reported and how to maximize the impact of the reports. Bill then pivots to a discussion on the CISO’s role in risk management and how to measure the effectiveness of this strategic function.
Matt points out that there is no shortage of things to measure and helps the reader understand how detrimental an unchecked onslaught of raw data can be. He skillfully guides the reader through an analysis of key categories of risk and the relevant measurements to capture and report. Some of the categories he covers include legal, financial, human resources, vendor management, software, data, and system hygiene.
Gary focuses on how to effectively frame information for management and the board of directors to, in his words, “tell a story.” After outlining the criteria for developing the set of metrics the CISO will collect and share, including sample metrics and a formula for creating a useful metric, Gary pivots to organizing the information for consumption and action. He brings all of this home for the reader by sharing lessons learned, including the types of reports and dashboards to share (and with whom), establishing relationships with the recipients of the dashboards, and putting the information into context before they even see the report.
Some of the questions the authors used to frame their thoughts for this chapter include:
♦ What are metrics? Why are metrics important? What steps should the CISO and security team take to create valid metrics for their program?
♦ What are some examples of dashboards you can develop as strategic assets?
♦ What types of reports should a CISO create to educate executive management and sponsor a more resilient, cyber-aware corporate culture?
Cybersecurity Metrics – Stamper
We live in a noisy world, one where the amount of information that crosses our desk, overloads our inbox, or distracts our attention from more meaningful activity is overwhelming. For those of us who work in IT and cybersecurity, our world is exceptionally noisy, and the signal to noise ratio is overwhelmingly noise. Just look at the log and event ID detail with which we work.
As a case in point, Cisco’s ASA reference material includes over 500 pages of Syslog detail (this is just one platform). Combine firewalls with routers, switches, servers, operating systems, applications, VPNs, LDAP or AD, and facility systems from multiple vendors, and you get the picture. Information blinds us.
As CISOs, we are simply overloaded by the amount of information that we are expected to absorb and respond to in a timely and technically-accurate manner. The tools we have to simplify and order this noise are also challenged. The basic legacy signatures and rules-based approaches to securing our infrastructure cannot keep pace with the talents of those looking to compromise our organizations.
This information overload is the reason why so many attacks are successful. The bad guys know how overwhelmed traditional security and IT departments are and can craft exploits that take advantage of this signal to noise ratio. They can simply send a well-crafted e-mail with a weaponized URL link or attachment. Advanced Persistent Threats (APTs) are mainly below the radar, overlooked in this noisy environment. We need to be more efficient in reducing the noise associated with our security operations.
The Value of New Approaches, Techniques, and Technologies
There are ways to improve our security operations and enhance our capabilities to find threats to, and within, our environments. On the technical front, there have been fantastic enhancements to automating security analysis, including tools to automate the collection and surfacing of specific event IDs that warrant attention – essentially indicators of compromise (IOCs). Complementing and extending Security Incident and Event Management (SIEM) tools are newer approaches that leverage network and user behavioral analytics to triage anomalous behavior. Anomaly detection and reporting offers an innovative and practical approach to focusing on what puts our systems and organizations at risk. The value of these systems is that, when engineered correctly, they leverage machine learning that mitigates the requirement for extensive rules writing and manual intervention.
Apart from the technical improvements we see in the realm of anomaly detection, there is also an increasing maturity in security operations related to agreed-upon security controls and metrics. As discussed previously in this book, the FTC’s enforcement of Section 5 of the Federal Trade Commission Act – focused on unfair and deceptive trade practices – has had the effect of creating a minimum baseline standard for security practices, at least within organizations that have a consumer focus.
There is also precedent from states attorneys general, including Kamala D. Harris (former Attorney General for California and now U.S. Senator), recommending the adoption, at a minimum, of the Center for Internet Security’s Critical Security Controls. Essentially, there are now widely-agreed-upon frameworks – including the recent NIST Cybersecurity Framework – that set the minimum bar for security operations and can be used to evaluate and baseline your organization’s security practices.
Security metrics validate the effectiveness of our security operations and controls and provide actionable detail on where organizational improvements are required. Similar to logs, event IDs, and other data points, not all security metrics are created equal. The goal is to have a tailored set of crucial security metrics that are appropriate to your organization’s size and complexity as well as commensurate with the regulatory environment in which your organization operates. Effectively, as a CISO you want to focus on the return on security metrics employed.
To that end, I strongly recommend grouping metrics into functional areas and focusing only on those that are truly important to the organization and your security operations. Too many metrics can feel like a logging environment without a SIEM… too many distractions and nothing upon which you can act. Too few metrics and you overlook key performance and risk indicators. A balanced and thoughtful approach to security metrics is required to ensure that you align the signal to noise ratio with your organization’s risk tolerance.
I recommend grouping metrics into functional areas. There should be metrics that provide insight into administrative functions such as training, policy review and approval, and non-technical indices. Other metrics should focus on the operational and technical side of security. The development of your organization’s metrics dashboard should involve colleagues from business units and executive management. Their insights and requirements will inform the types of metrics you ultimately create, implement, and review. These metrics should be consistent with the core view that the CISO role is transforming into a lead risk management role – evaluating information risk across the entire organization.