Thursday, 23 May 2013

On testing and causal statements

Always unintended consequences.

Whenever we start going down a path of implementing policy we need to think about the consequences. This is both the seen and the unseen.

It is perhaps most importantly the unseen. We cannot say that an intervention has achieved the best outcome and that is better than something when we have not actually compared it. This means when we look at a risk reduction process in information technologies or implement new economic policy or for that matter any other intervention we need to investigate it fully.

In business, good project managers and more importantly portfolio managers will investigate the various results and compare these against what is in effect a null hypothesis. That is they will compare a sample project against the status quo. When doing this we need to also contemplate all the alternatives. This is where many people fail.

More importantly when we are doing correlational studies and investigations that do not allow for experimentally controlled trials (this is gold standard double-blind testing), it is critical that we investigate causation in a rigorous process.

Austin Bradford Hill (1897-1991), was a British medical statistician whose great contribution to science was to leave us with a set of minimal conditions that are required to establish a causal relationship between two events. Hill's criteria has become the basis of modern epidemiological research. It is one of the mainstay methodologies that I use in my research both in evaluating economic effects and when investigating malicious software and other security controls. In particular it is useful in evaluating the human aspects of information security and risk. It is not just epidemiology but other fields such as economics that can benefit from this approach.

Hill's criteria is the basis of good scientific research where we are seeking to establish a causal relationship amongst social phenomena and in particular ones where we cannot engage in controlled trials. In some instances it is in fact better than a controlled trial as the process of creating a controlled trial changes the environment and creates a bias in many of the results. Although it is true that this controlled trial has provided the best answer to a particular problem is not always true that we are investigating the same problem. One example would be looking at studies of irrationality. The University controlled trials testing the reactions of students generally biased the results. In selecting risk trials for instance we take selective forms of risk that bias the results towards male or female risk takers in the study. Later studies have now shown that these original studies into irrationality have been the result of poor methodology with both women and men exhibiting similar levels of risk. What was demonstrated is that the forms of risk taking differ between men and women but overall the levels of risk a similar.

If we are to make a claim that population growth results in poverty or that capitalist governments cause poverty in developing nations or even that Keynesian spending results in the long-term growth them to be scientific in our approach we need to demonstrate causal relationship. Hill's criteria provides one of the ways in which we can do this.

For example, as an economic investigation we can formulate a strategy and hypothesis based on welfare based systems such as a guaranteed minimum wage.

It seems a good process and we have created a safety net. Again an issue is that we are also not thinking of the unseen events. This is what will occur if there was no welfare. In starting such a welfare system, the differential incentive to work decreases. This is not hard to explain. 

Basically it is not poverty but a differential between income that people see as the greatest disparity. Even in Western nations where there is no need to be poor and in fact where most of the poor are wealthier than the middle-class 100 years ago what we see is a desire for more against those others in society.

So, the end result of such a policy is to have more people enter welfare. The result, those earning need to pay more for the increased welfare state. Less incentives for productive work. The differential decreases further making welfare more attractive and incentive's more onto state support. The result more welfare... A degrading cycle of creating more and more welfare.

This takes us back to Hill's criteria. Here we have nine criteria to measure any correlation effects against. These are:

  1. A temporal relationship where the cause always precede the outcome. If there is some factor that is believed to cause an event and this must always necessarily precede the event. This first criteria is the most critical and essential of all of Hill's criteria. If this one is not true then that we have a correlation alone and no causal effect.
  2. Next we need to consider the strength of the relationship. This is a statistical measure of the strength where the factors are highly related. We can look at the Pearson number for correlation as a means of testing this value.
  3. Next there is a effect response relationship. This is a measure of input. As we increase the amount of one factor the other must also increase. For instance if we put more time into training people in security awareness then naturally for this to be causal in relationship we would have to have improved security. The improvement is not required to be linear and we may find that each incremental expense returns less but it must return something more than it would've if it wasn't there.
  4. The fourth relationship is consistency. The results need to be replicable and repeatable. They should apply in different population groups and samples.
  5. Next we look at plausibility. The association that we are purporting to exist needs to be supported by a valid theoretical basis. There needs to be some phenomena that can act in a manner that causes the result or event.
  6. The sixth criteria is that we consider alternative explanations. Many so-called scientists fail here. They just assume a relationship matches with their understanding. It may be true that we can dismiss many arguments out of hand as they have already been investigated and shown to be false, but this does not mean that we do not consider alternative explanations. We must always consider multiple hypotheses prior to making any conclusion a about a causal relationship between events we seek to explain and investigate.
  7. Experimental evidence is also important. Even though we cannot expect to completely re-create an event we should be able to implement an appropriate experimental regime that supports our causal argument.
  8. Next there is a requirement that the causal effect is specific. This is one of the weaker criteria and we can demonstrate causal effects without it. The absence of specificity does not negate a causal relationship but the existence of specificity between associations does add additional support to the existence of a causal relationship. Here is important to always examine specific causal relationships with in a larger systemic environment.
  9. Lastly we have coherence. Ideally any association we are purporting to exist should fit within the body of existing theory and knowledge. Their are ways of course to introduce new theory and Thomas Kuhn referred to these changes to the excepted theoretical basis of science as a “paradigm shift”. To reject the existing theretical basis of science we need to have a particularly good and strong proof and evidence supporting our new claim of causality.

The third of Hill's criteria, the effect response relationship is one that seems to be missing in much of the so-called science we see. For instance in carbon studies we should see a related increase in atmospheric CO2 leading to corresponding increase in global temperatures (all other things being equal). We should also see a corresponding and commensurate decrease in global temperatures as atmospheric CO2 levels decline. This is been something that is rarely investigated enhance which still relegates much of the climate study to pseudoscience.

In a couple of my publications for instance we looked at the effects of economic sanctions on criminal groups involved cyber crime. Two of these papers are:

“Criminal Specialization as a Corollary of Rational Choice” (2012)


Territorial Behavior and the Economics of Botnets” (2012)

In demonstrating the economic effects of a policy designed to reduce cyber crime we need to investigate all of Hill's criteria. In this instance what we find is that cyber criminals are rational actors. Like most other people in society when they're acting individually they act in their own rational interest. When they are offered opportunities that provide better returns for low risk they will take these over the antecedent of a poor return or one with high risk.

Of course the current answer to this and the mainstay of many Keynesian economists that propagate government circles is to argue that what is rational for the individual may not be rational for society as a whole. They argue that irrationality comes of collective rationality.

Of course what they are saying is not that the collected actions of the many are in fact irrational but that people choose things that they did not desire. This is a typical political fallacy that is designed to appear scientific. Many economists make very poor scientists.

This is of course rational and itself. For when we see government favoring Keynesian aligned big government economics and rewarding those who support the idea of big government we also see incentives given by government to those who promote big government. So the individual behavior of these rational economists is to support in irrational policy itself. It is of course rational to support a biased policy when you gain from it at the expense of others.
What could be termed irrational is a libertarian policy that exists with little support and certainly none from government. As a libertarian one has to fight harder and do more. But here it again comes to what is a subjective value. When comparing rationality we need to look at the values of the individual. In my case my greatest value is freedom and that cannot be given through big government and subjugation. In this case economic constraints and an acceptance of lower benefits comes at the cost of upholding one's values.
One further part of this rant is that we need to start excepting that it is not irrational to hold one's values. In fact the only rational choice is to uphold one's values. In this we start to see where one's values lie.

No comments: