"Inequality at birth is neither just nor unjust. What's just and unjust is the way institutions deal with it" - John Rawls
"I always had a certain dislike for general principles and abstract prescriptions. I think it's necessary to have an "empirical lantern" or a "visit with the patient" before being able to understand what is wrong with him. It is crucial to understand the peculiarity, the specificity, and also the unusual aspects of the case" - Albert O. Hirschman
When NGO thinking percolates into policy narratives and policy making
The danger of “attributable to me” approach is not in the non-government sphere. The real danger is in its percolation to the government sphere.
Lant Pritchett in his usual brilliant style has mounted a new critique on the narrow RCT led development.
Lant’s primary argument is that development is increasingly pursuing “attributable to me policies” rather than “optimal policies”. As an example for non-attributable research, he quotes the example of research that has built the narrative on necessity for reforms in India’s pre-1991 era. Research of this kind, Lant says, has contributed more to the increase in incomes of people than the narrow RCT led research.
Lant argues that an important consequence of RCT led development is that
individual actors care more about what they can take credit for than whether there is team success
In these arguments, Lant largely focuses on the non-government sector and academia, where there is an
increasing tendency in NGO sector to fund only projects and research that can demonstrate causal effect; and
increasing tendency in academia of development economics to shift towards studying narrow questions (RCTs) rather than the big questions of development (what causes growth?).
In this post, I argue that the dangers of perils of attribution is not in NGO or Academic sphere but it is in the percolation of such thinking into governments’ sphere.
NGOs: NGOs operate in a fundamentally different manner from the government. They are focused on a key problem, often a narrow and local problem. They can pursue only one aspect of the problem, keeping others aside. In such cases, the RCT led approach is useful because without it, we are just shooting in the dark, especially spending the valuable resources.
Academia: In academia too, the increase in RCT led research need not be an issue to worry. It might be a much needed correction. Historically, development research has focused on the macro question of how nations grow but it neglected the question of how do households come out of poverty. RCT research has just plugged in this missing gap.
One might check India’s Economic Surveys to feel the necessity of micro-development research. The quality of analysis on macro issues is of top quality, in the Economic Survey. On the other hand, the quality of analysis and commentary on micro-issues like education and health care is disappointing to say the least. It underscores the much needed appreciation of nuances of micro-level issues, for which RCT is a key lever.
In summary, the advent of RCTs has plugged a necessary gap in the NGO sphere and academia.
The real danger of privileging “attributable to me” policies over “optimal” policies that Lant has outlined is prominent in the sphere of government’s policy making.
The RCT led development has led to the following dangerous consequences in the policy narratives. This is not necessarily due to the RCTs, a significant part of it is also due to the way people misinterpret the RCT evidence.
1) All that’s not attributable is being dismissed: NGOs can afford to pursue “attributable to me” policies because they do “interventions” and RCTs on interventions provide useful feedback.
On the other hand, governments don’t do only “interventions”, they do a lot more than that.
When the NGO approach percolates to policy narratives and policy making, it’s dangerous. The “attributable to me” approach that’s applicable to interventions is being applied mindlessly to “the constraints in inputs”.
For instance, in the “attributable to me” approach, if there’s an RCT evidence suggesting no effects of an intervention, the necessity and design of that intervention is rethought.
Now, apply this to RCTs on critical inputs (infrastructure, nutrition etc.), addressing which is primarily the domain of governments. Viewing it from the NGO approach, if the RCT evidence on any input (say infrastructure or nutrition) doesn’t show positive effect, people are interpreting the evidence as - such an input is not worth addressing.
For example, check this post on “What ails primary education in India?” that attempts to use RCT evidence to address this question. Summarizing Duflo’s lecture, the article says (not sure if Duflo actually said that but the key point is note how people are interpreting it and the narrative is being built):
The reasons for the poor outcomes in learning, as popularly believed, are: a) children are undernourished, b)scarcity of resources in that teacher salaries are low or infrastructure is poor, and c) incentives are not aligned. However, these factors are NOT the reason for poor performance of children. There might be a correlation, but causation can not be established.
In what universe can a government aiming to build a functioning school system not desire to focus on nutrition of children, infrastructure, incentives of teachers etc? Aren’t they obvious? Further, should each of them be measured only by the metric of outcomes? Why should toilets be expected to increase outcomes? Why can’t they be built because they have to be built? Should we not focus on them because an RCT says so?
The no positive effect is often due to implementation challenges and more importantly, lack of complementary inputs. One can’t just isolate each input, say it’s not working and hence it isn’t worth focusing.
RCT evidence is being increasingly interpreted in this manner, promoting a fragmented outlook.
Now, governments being pressurized to follow the narrow “attributable to me” policies, an approach suitable for NGOs.
By telling that there’s only one or two big constraints (and others are worth less), it promotes governments to pursue a check list based approach, where one addresses one constraint. This is fundamentally contradictory to the development process, which is often a result of complex interaction of multiple inputs and processes. By pursuing one at a time approach, we may be achieving local maxima, but we are delaying the process of global maxima.
2) Legitimizing the existing constraints of the bureaucrats and politicians: RCTs often being with the presumption that governments have resource crunch and low bandwidth. It then goes on to take these constraints as given andattempts to find solutions within these constraints.
In this process, the policy advocates almost never focus on the key recommendation — getting rid of those resource and bandwidth constraints. How long can we keep pursuing these incremental policies? At some point of time, these constraints are to be broken to make the necessary big push.
By considering certain constraints as given, and recommending policies only within these constraints, RCTs end up legitimising these constraints. They are the elephant in the room, no one wants to talk about.
3. Creating false binaries: The other important fall out of the fragmented out look of identifying one or few constraints (these days through RCTs) is that it creates false binaries — teacher hiring vs. teacher inspection; nutrition vs. textbooks and so on.
In reality, one has to do all. These aren’t compromisable issues. While trade-offs are part of policy making process, the narrow RCT led approach has percolated this thinking into trade-offing even the basic critical inputs.
All this is pure gold to a government which doesn’t want to increase resources and bandwidth but instead wants to focus only on narrow reform to be appear to be doing something. Now, they have the backing of evidence for what they are doing. Sadly, the neglect of other critical inputs don’t have RCTs to prove that they are worth addressing!
While one may argue that governments have not actually pursued this approach, a strong public narrative has been created. It’s only a matter of time for it to reflect prominently in governments’ actions.
Finally, a word of caution here. Most of it is not necessarily due to only RCTs. Such fragmented approach to reform existed even before the advent of RCTs but RCTs have only exacerbated it. Again, it might not be necessarily due to RCTs but more to do with the way they are interpreted and the way findings percolates into public narratives. I didn’t see any significant effort from the RCT researchers to correct these either!
The attributable to me approach is understandable in NGOs because they often work on narrow questions.
Increasing focus on attributable to me questions in academia is also understandable as it can be seen as plugging the gap in such research.
The real perils of attributable to me approach is when such thinking percolates into policy narratives and policy making. It leads to fragmented approach that result in non-comprehensive policies, legitimizes the resource and bandwidth constraints, and creates false binaries. In long term, this is dangerous.