Tuesday, 14 May 2013

On Trust and Risk

Security matters, but not so we can eliminate all risk, but so we can have trust. Even if we could eliminate nearly all risk (we cannot ever remove risk entirely) we would have to ask whether it was worth it to do so.

Risk IS quantifiable.

This is a statement like many others that is true, not always in the ways we assume, but it is true none the less.

We can always measure risk. This does not make a difference what field you are referring to, risk is a quantifiable metric.

The problem is not if we can measure risk, but how and with what results. These results come to:

  • reliability,
  • precision, and
  • accuracy.

These are not the same, but each has a bearing on how well we report on risk. The first of these, reliability comes down to whether we can repeat the same results again when we do an experiment. It refers to an ability to have either or both precision and/or accuracy stay within predictable bounds.

Precision is how true we are to the mark each time we make a risk measurement. This is, how close to the real value we lie and in effect it comes to the level of variance we have. We can actually be imprecise with the mean value right on the bulls-eye and results that have a large variance or spread. This would be centered on the expected mean on average but with results that vary widely.


Accuracy is how close we are to the mean or other value we see as the measure of risk. We can say it is a measure of how close we are to the bulls-eye.

To have a good measure of risk, we need to aim for both precision as well as accuracy. It is also important t5hat we can reliably have a measurement that we can have others examine and produce.

Qualitative measures of risk.

There are always people who will tell you that risk cannot be measured. What they are really saying in effect is that risk cannot be measured using a scientific process and is an art.

There are reasons that people hold these views. Some have the idea that metrics are not possible and that only skilled people can create a metric. The flaw in this argument is that this is a form of metric and it is one that can be measured and tested. When we look at the results of how risk comes out over time, we see that the art based approach does not work well.

In science, we make predictions and the ultimate test of these predictions is the result that the real world delivers over time.

Risk can be measured. In doing so, we hold those making predictions to account. We can start to measure the actual predictions made. Is a system secure, well time does tell and in checking the “predictions” of risk and security people against time we can make measurements.

In making models, we also see how well we model a system and the feedback from inaccuracy and imprecision allows us to improve over time.

Next time somebody states to you that risk cannot be measured, remember it is. Think instead what they are telling you is that they do not want to have their ability tested in case they come up short.

No comments: