Saturday, 1 March 2008

Security Stats and how I got them.

Mark Palmer asked a good question in respect of the stats. This is the extension of the comment to enable an understating of how they were (and still are) collected.

Working with a Chartered Firm we get to see a large number of clients. At the least there is a requirement for "information gathering". This is where we are engaged with a Financial Audit, but need to do a quick and nasty risk assessment of the financial systems for the client.

I run the team, so I (much to the disdain of many others) get to spend a little more time then is billed and have a lower productivity to offer a better result. Most auditors do not do this as it has direct financial consequences. My salary and bonus is tied to productivity. Most are. In the Big 4 firms this is even truer. This comes to a factor known as materiality that I will discuss later in the post. I prefer to offer value and hate not reporting on security issues. So in effect I do more than I am meant to.

There are over 500 clients, but we see some of these only every 2-3 years. Any of this type is excluded.

What is materiality?
Materiality is the point where the auditor really cares. This is set a fraction of the client profit; turn-over or whatever is being checked. If the audit was of stock and the client had a total value of $10,000,000 held in stock and materiality was set to 5% (a standard limit), materiality would be set at $500,000. We as auditors care about a loss of GREATER then the materiality limit. That is if the loss is less then $500,000, it is a SEP (somebody else’s problem).

This also applies to the IT audit side when conducted as a component of the financial system. In the case of what we care about – it is current losses. So a data breach that could occur or a loss of private data is an MLP (management letter point) and no more. Financial Audits do not (and this is why I am naughty for doing the extra) care about potential losses.

Where the Stats come from
I have to do or oversee all of the IT work in the state. This is that for any complex (and now also non-complex) job, I have to verify the work and ensure that it was completed correctly. So even when I did not complete the audit, I have oversight and I trained those doing them.

As a statistician as well, I have an issue with the stats. They are naturally subject to bias. This is not personal bias, but rather these are representative of a chartered firm (being a mid-tier accountancy). They are not necessarily representative of the population; they are representative of the type of organisations that use a mid-tier firm. I can not extrapolate these to the population without comparable data from a Big 4 firm, and I do not see this occurring. (I do see that there are not many Snr Managers in the Big 4 willing to lower their productivity however ;)

I do see reports from Big 4 firms. I review them on a weekly basis. Again, these are reports for organisations that are not necessarily representative of the population. What I would really need is to have a TRUE random sample of organisation to make the statistics valid to the population. Even a small sample could be used to make an inference of the existing stats – but this would mean choosing 50 companies at random and doing an audit without being their auditor – never happen.

For the moment, these statistics are valid for the type of organisation that will use a mid-tier chartered accounting firm.

Complex - this is a client with a relational database, ecommerce, multiple systems etc
Non-complex - this is a client with a simple accounting package (eg MYOB or quickbooks).

No comments: