How to read evidence from RCTs? - I

One of the common criticisms of RCTs is that they aren't externally valid, meaning the results can't be extrapolated to other settings. This is sort of a naive 101 criticism. The utility of RCTs is not from a single study but from a collection of studies.

RCTs provide evidence of the extent of binding constraints which can be used to build theory. Further, the mechanisms of impact, rather than the particular intervention is scalable. What does this mean?

Let's consider this example. An RCT finds that helping households build rainwater harvesting tanks increases girls enrolment. Increase in girls enrolment is the effect here. The mechanism of the effect is that girls in households with tanks spend less time on water collection, which is then used to attend school.

Consider another study. In this study, m-Pesa is found to have reduced gender gaps in Kenya. Reducing gender gaps is the outcome here. The mechanism is that m-Pesa reduced the time spent by women in paying bills. Paying bills usually used to take them lot of time, including the time to travel, staying in queue etc. The time gained through mpesa is employed in productive activities.

The interventions in these two studies are different. Probably the outcome is also different. But, the mechanism is similar. Easing women's bandwidth in doing some household works can help them utilise their time for productive activities. This is the learning. It can be used to devise solutions as per the context.

In a given context, you look for activities on which women spend significant time. Suppose it's for collecting drinking water and fire wood. You devise interventions to reduce the time spent on these. It can be either by providing piped drinking water or subsidized LPG cooking stoves. You thus build solutions by learning principles behind successful interventions, as documented by RCTs.

Sendhil Mullainadhan has a good discussion on this in his paper "Mechanism experiments and Policy evaluations". He advocates for testing mechanisms. For instance, in the above example, saving time is the mechanism. You look for other activities on which women spend time (firewood collection). Try to save their time there through an RCT intervention. If it's a positive result, then it's a robust mechanism.

On a different note, the two studies discussed above can be incorporated in Indian policy. The union governments recent initiative to provide free LPG stoves to women can have a significant effect due to this particular "time saving" mechanism. The environmental benefits are additional. The next step should be towards providing piped water or at least some form of ensuring easy availability of water.

No comments:

Post a Comment