Information Security Metrics Insanity, The 3Rs And Dashboards!


I just finished a research document titled Measure The Effectiveness Of Your Data Security And Privacy Program for the The Security Architecture And Operations Playbook. This was a lot of fun to write, because I was able to look back at the 50-plus interviews conducted over the last year, all of them focused on the security metrics issue. This seems like such a hard question to answer. My conclusion is that many security organizations are measuring the wrong things.

There are several reasons for this.  Here are a few of my observations:

  1. We always measure this.
  2. It’s too hard to get any other data.
  3. Our budgets are fixed so we just do the best we can.
  4. Etc…

The list continues pretty much in the same vein. Security officers complain they don’t get the recognition, budgets, and attention from senior leadership, yet our metrics don’t really tell senior leaders anything they want to hear about.

At the end of the day, it’s top-line growth and bottom-line profitability that senior leaders care about. Anything that aligns with these goals will have their attention. Anything else is just noise. Yet many security officers still throw the same old information at senior leaders and expect different results. This is a sign of insanity.

In the paper, I outline the need to refocus on security on three dimensions that have driven security for centuries: readiness, response, and recovery. These are the 3Rs of security. Added to this is a fourth – financial.  We still need to manage our business like a business. This means making financial tradeoffs.

Lastly, in the paper, I talk about how to present information. Dashboards are key here. Because everyone is so busy, presenting information graphically, showing trends, and demonstrating effectiveness is key. I attended a demo from Core Security (, a maker of vulnerability assessment tools that has added some great dashboards to their tool set, at the recent Black Hat conference. It was a good demo and the company showed off their security vulnerability dashboard.  It allows users to see the effectiveness of their counter-measures as compared to vulnerabilities over time and most importantly to tailor the information to the audience. This is a great feature of the tool.

Based on what I outlined above I see this type of dashboard capability as a real need for security officers. Core is not the only company doing this, but it is a good example of tools that can help security officers show off their efforts. As I like to say: “You get what you measure.” Metrics change behaviors; that’s their value. Sharing those measurements so people know the value of your efforts is a best practice. If security officers are going to be able to get the necessary attention of senior leadership for their initiatives, they need to show that their efforts are effective, and how they contribute to the business. Check out the paper when it publishes and let me know your thoughts.


Core Impact

I attended a demo of Core Impact a few years ago as well. The program was full of wonderful tools and great reports and is even on DoD's GSA for authorized purchase (for around $28k). After the sales demo, I did what I do best; ask lots of questions. The Core salespeople were not able to answer my first set of questions so I handed off to Core's technical team. They were not able to answer too many questions either so I was introduced to their Lead Research Scientist. I asked how updates were handled and if they were pushed to the clients or required downloading.
The scientist was not sure about updates but he would contact the Core Impact developers in Argentina. Argentina!!!
Here is a Hacker's dream program, sold across the entire DoD and the program was written by folks in Argentina. I did my best to alert DoD about Core Impact's authors but it didn't seem to interest anyone at that time. Granted, this demo happened back in 2009 however, I would hate to pay that much money for one big Trojan Horse.

Core Security

Thank you for the feedback. The demo I saw was done about 6 weeks ago. I do think that proper vetting of vendors is very important. My comments were specific to the ability of the product to do dashboards well. We at Forrester have not been concerned with this vendor's security profile, and the product has acceptance in the marketplace.

Best regards,

Ed Ferrara

Reply Core Impact Comment

Commenting on B Monroe's concerns about Core's business model and locations. Core Security is headquarterd in Boston, Ma with labs in Argentina. Core has been in business for 16 years and the Impact product in question on this blog comment (distinct from the solution E. Ferrera was commenting on--Core Insight) has shipped version 12.5, with version 13 in development. (It has a long and well-established track record in the market.)The company has a sterling record with regard to its security practices. Core products are available on the GSA-schedule, and sold widely into the government and law enforcement communities. Hopet his helps address his concerns.

Core Communications Dept