Saturday, 28 June 2008

Security Metrics

What I would see as a simple starting point for metrics.

What is needed is a measure that is not going to be subject to the views of the individual making it. That is, an expert or a trained monkey should be able to get the same results.

What I would propose.
I would leverage the existing work of the CIS for a start. Each of the metrics that are produced using the CIS test tools are a start.

For instance, you can run router tests, Windows system tests on servers and workstations, Unix/Linux servers etc. At least this way we have a starting point.

In the event that there are many systems, we could add a simple random sampling routine. We have 500 servers, 15 are selected (by software) randomly to be tested. The CIS tool is run on these and the results added for the week, month… whatever.
Next we can add other statistics. Network traffic, router logs etc. Again the idea being to keep them as objective as possible in measure. This is, remove subjective bias. There are many ways of doing this in IT metrics. This is but the start, there are 100’s of metrics such as this we can add and start to use to get some really effective results.

This is far from ideal, but it is a starting point where we use existing CIS tools and create metrics that we can improve over time.

As metrics are improved and we start getting data, we can make a level 2 tool that has forecasting and other enhanced capabilities. This is:
1. Stage 1 – Simple Frequentist approach with results from CIS tools
2. Stage 2 – Bayesian predictive model
3. Stage 3 - …

For stage 1, we can do this now.

I will even write the tool to do something such as this.

From the simple boxplot (and there are many ways to visualise this, this is just a quick and nasty) we can see an improvement overall, but also a trend in measures 4-6 where the security metric of some systems has decreased. This not only gives an indication of improvement, but aides in discovering where problems may lie.

The good stuff will not be there In phase 1, but at least it is a measure that is going to be the same each time any person runs it.

No comments: