Unfair criticism of RCTs
Commenting on the cynicism I noticed in policy circles about other's ideas, I wrote: "An established policy is judged by where it works, while a new policy is judged by where it does not work". I meant to say that often a new policy is dismissed for not addressing "all problems" but the status quo problems are continued and accepted despite addressing a lesser number of problems than the proposed one.
One of my friends read this and commented that it's not just the case with policies but it applies in general to organizations and humans too. Historically under-represented sections of people need to work "extra hard" to receive the same level of praise or recognition the dominant people in status quo get.
Unfortunately, it is true in the case of the criticisms on RCTs. Consider the following criticisms of RCTs
1. Show one RCT that resulted in a policy change.
2. Only certain kinds of questions can be answered with RCTs.
3. RCTs displaced other studies
Each of these criticisms has the status-quo bias. To make it evident, ask the same questions on any other form of study.
1. "Show one qualitative study about the resource constraints in bureaucracy that resulted in direct policy change". Despite the availability of numerous such studies, why does this problem remain unaddressed? Does this mean that qualitative studies are useless?
2. Is there any one particular method of study that is universally applicable to all types of problems? Why is this question asked only in case of RCTs? Why don't people ask the same about other forms of research?
3. The criticism of RCT displacing other studies sees research as a zero-sum game. Esther Duflo has, in fact, demonstrated that RCTs did not displace other studies. Instead, they only added to the existing number of studies.
The underlying reason for these criticisms is the lack of appreciation of the nature of research, misrepresentation of RCT researchers, and non-questioning of the existing state of policymaking.
1. The nature of research in social science is that the attributable change doesn't often happen due to a single study. A series of studies shape an alternative narrative that results in change given the right conditions. Somehow, people miss this while criticizing RCTs.
2. No serious RCT researcher ever says that RCTs are the only necessary and sufficient evidence, and input for policymaking. For instance, consider this policy paper by Prof. Karthik Muralidharan where he clearly says "Policy formulation needs to consider technical, administrative, ethical, as well as political factors and even the best technical studies can only provide inputs into one dimension of policymaking." But people criticize as if RCT researchers profess that only RCT evidence is the necessary and sufficient one!
3. In their criticism of the RCTs, especially the bureaucrats don't question the existing state of policymaking. They need to question if the so-called irrelevance of RCTs is due to lack of relevance or inability to incorporate it in policymaking. I suspect that it's the latter because - forget about RCTs - policymakers often don't even use the other forms of existing research. Worse, a recent survey suggests that the top bureaucrats of the country can't do a simple and basic data analysis. Interpreting and incorporating evidence is a long shot in such case.
Further, on the criticism that "foreign researchers" don't research on relevant problems, the critical constraint, in fact, is the bureaucratic maze around getting necessary data. A prominent India-based researcher had once commented that the only way to do good macroeconomic research in India is to be in good books of RBI because it provides access to research data. This is, in fact, true of the nature of research in social sciences. Change doesn't often happen due to a single study. A series of studies shape an alternative narrative that results in change given the right conditions. Somehow, people miss this while criticizing RCTs.
In such situations, we would be better off throwing open the access to research. I am sure the gates will be flooded because research is where data is.
Overall, the criticism of RCTs being irrelevant, displacing other studies, and not resulting in direct policy change are unfair because every other form of research fails if subjected to these criteria. If other forms of research are not subjected to these criteria, why should only RCTs be?
The most important concern is the lack of access to quality data for the researchers. We would be better off if we focus on addressing this problem instead of debating on the merits of different methods on research.