RCTs on key inputs tell what isn’t enough, NOT what NOT to do

RCTs are of various kinds. One type of RCTs involves testing the impact of key inputs. Key inputs are those which we know are necessary to achieve something. For example, we do know that textbooks are necessary for children. We don’t need evidence to tell that they are necessary.
The other type of RCTs test interventions or ideas. These are typically of the form — where a particular tool or a particular government scheme or policy is effective. These programmes are essentially different ways of utilising “key inputs”.
This post is on interpreting the RCT evidence of the first kind — the ones that test key inputs. RCTs on the effect of distributing free textbooks is a good example. Generally, these kind of RCTs find no effects. This is interpreted in two ways.
One section of people interpret the result of “no effect due to distributing free textbooks” as “money shouldn’t hence be spent textbooks”.
The other section will interpret the result as “Why do you even need to test the effect of free textbooks? Isn’t it obvious that kids need textbooks?” This form of criticism questions the futility of research on such questions.
Both these are incorrect interpretations of the “no effect”.
Distributing free textbooks didn’t show results doesn’t mean that we should stop distributing textbooks. May be the textbooks are of poor quality or that the other necessary complementary inputs required to put these textbooks to use are missing. We should focus on that. When we know that some “key input” is obviously necessary, we can’t do away with it.
Similarly, the arguments of futility of research testing effects of such obvious inputs are not valid. There is definitely a purpose to such RCTs.
The purpose of RCTs testing effects of “key inputs” is to demonstrate the incompleteness of efforts. It means that the purpose of RCT on textbooks is not necessarily to show whether textbooks are required, it is to show that only textbooks are not enough. Much more has to be done, which isn’t being done.
Some context will put this in perspective.
Often, journalists, policy makers, commentators and politicians are clouded by some specific pain points. Some specific pain points appear so big to them that they end up phrasing these pain points as “the major hurdle”, addressing which will lead to significant outcomes.
This typically happens with journalists who visit schools and find students with no textbooks. They come back to campaign that “lack of textbooks is the reason for poor outcomes”. The entire policy discourse ends up being about “textbooks” and all efforts end up being only about “textbooks”. Note: Textbook is only being used as a hypothetical example here. It can take different forms.
Politicians and bureaucracy are in comfortable equilibrium by working on textbooks because that’s the way the problem is phrased; they think that they are addressing the right problem.
In this process, all aspects other than textbooks are neglected.
It is when the RCTs are required. The “no effect” due to textbooks tell that this may not be the major hurdle OR the current level of efforts may not be enough. It signals that we need to do much more beyond textbooks to achieve outcomes.
It is the way it has to be interpreted.
The ongoing debate on “hot-policing” is another example that frames the results of “key-inputs” in form of a binary.
Chris Blattman tests the effects of hot policing (focusing on crime intensive areas) in his new paper and finds problems with the approach — it just shifts the crime to other areas, disperses the crime making it to trace later and so on. He proposes to replace hot policing with a different approach.
Gulzar Natarjan strongly disagrees with it. He asks:
Do we need evidence about these two elements? In more simple terms, in a system with scarce resources and several competing needs, isn’t it a natural order of things to prioritise them using the most effective reporting and monitoring mechanism possible?
Have we ever sought evidence on whether great Dashboards are a step in the right direction in the corporate world? Did the first set of corporate users demand evidence from IBM before placing orders for Cognos?
This debate is similar to the textbooks debate discussed earlier. Blattmann is inferring the evidence as “hot policing is not working-hence try different approach”. Natarjan is saying that — why are you even asking such obvious questions? In a resource constrained environment, these are the things that make the most sense. Why do you need to even research them?
Both these arguments may not be appropriate inferences of the evidence. As argued earlier, we should interpret this as “hot policing” alone isn’t enough. We should do more, including Blatmann’s alternate proposal of focusing on criminals instead of geography.
But do governments have enough bandwidth to pursue such comprehensive reforms? The next post discusses this.

No comments:

Post a Comment