Using appropriate metrics to judge interventions

Once upon a time, there were  two good friends. One of them got passionate about body building and  hence started going to gym. He also dragged his friend along. The second person wasn't interested in body building but his incentive to go to gym was to network with people. They both regularly went to gym. The first person developed a well built body at the end of six months, and the second person has developed a very good network of friends. Thus, both these friends were happy for achieving their objectives.

There was a security guard at the entrance of the gym, little does he know about the two friends' motivation to attend gym. He used to daily observe these two friends coming to gym. At the end of six months, the guard judges, 'the first person has achieved success while the second person hasn't built any body, he surely must have been wasting time in the gym.'

Now, who is correct? The friends or the security guard? This clearly is a case of lack of information and difference in metrics being used to measure success. Similar situations exist in interpreting interventions too, when the impact of the intervention is talked about, with each person having presumptions of his/her own metrics.

For example, the mid-day meal scheme. As per this scheme, all school attending children are provided with free lunch. The effect of this intervention can be measured using different metrics -enrollments, nutrition, attendance, and learning outcomes. There is evidence that midday meals result in substantial increases in primary school enrollment. There is also evidence that midday meals scheme has been successful in compensating for early nutritional deprivation. While this intervention can be called a success as per the metrics of enrollments, nutrition, it may not be so if the metric is learning outcomes. Thus, different people calling this as a success/failure may be actually interpreting it based on different metrics.

Some other examples include
  1. Microfinance - reduction in poverty or smoothening consumption?
  2. Toilets - enrollments or outcomes?
  3. Provision of free bicycles - enrollments or outcomes?
We see a clear pattern here. There are two types of metrics, universal and intermediate. Metrics like 'learning outcomes' are the universal metrics, the final aim. Metrics like enrollments and attendance are the intermediate metrics which eventually lead to outcomes, coupled with other factors.

The question then arises, how do we decide the metric to judge an intervention?

The problem with using universal metrics is that these metrics are often a function of several factors. As we saw in the fourth caveat of evaluations, each factor is a necessary but not a sufficient condition. When we administer an intervention addressing one particular factor and measure it based on the universal metric, without addressing the other factors, the negative results may not precisely indicate the failure of this particular intervention. It can also be because, the other factors weren't upto the mark though this particular intervention addressed one particular factor. Thus, it is advisable to use intermediate metrics to judge these interventions. However, if the intervention shows positive results, it shows that the particular factor that is being addressed in the intervention is a major bottle neck and thus is a useful information.

In summary, use the intermediate metrics primarily to measure interventions unless they show positive effects on the universal metrics. This takes care of the disaggregation problems when the results as per the universal metric are negative.

No comments:

Post a Comment