In my day job I am Solutions Architect for the McAfee Risk and Compliance Business Unit. Our products cover compliance management, risk management, IT security risk analysis, and a range of application and change control technologies.
[3/2/2010 note: when I wrote this article I worked for McAfee. I do, however, no longer. :)]
In a former life I evaluated complex, multidimensional data sets using models and metrics - I was a scientist performing quantitative analysis of noisy data. The result is that I've spent many years and a lot of energy thinking about and using models, metrics, and measurement. That background gives me certain views, opinions, and expectations.
Now that you know where I'm coming from let's talk about risk, measurements, metrics, and uncertainty. ;)
IT security is a relatively young field. Our ability to create and leverage models that consume data (measurements, metrics) and predict when a Bad Thing™ might happen is not where we really want it to be.
I have written on this topic before. In that article I propose that our best path to mitigating IT security risk is to be able to estimate with some confidence that a Bad Thing™
Measurement - collection of events from our environment - can help us with the first task. If we are nimble and quick on our feet then measurements and metrics can even help us with the second task by triggering alarms in an efficient manner.
Sadly, data alone cannot help us with the third. Before we can get there we need enough of the right kinds of data so that we can analyze, correlate, and model.
The Center for Internet Security have organized a boat load of knowledgeable people to agree on and promote a collection of measurements and metrics that we can use. They call them the CIS Consensus Information Security Metrics.
There are twenty measurements and metrics listed in the white paper grouped into six different categories - I have include the list below but I encourage you to download the white paper and read through the details. If you do you'll find that they are all things you should be tracking in your security environment today and that they are all "good common sense".
Several of the metrics in the collection are for coverage of key metrics across your organization - that is, they are measures of possible holes in your risk management program. As I've argued before, RISK is better than UNCERTAINTY and uncertainty arises from a lack of data. If you improve your security posture by closing holes revealed by these metrics then you will find yourself moving your organization from the realm of managing UNCERTAINTY and into the realm of managing RISK.
Now, if I could get my hands on a large, anonymized data set that includes these metrics I think we can make some progress toward predictive metrics and models.
-Eric
p.s. Here is that list of categories and metrics. Enjoy!