Case against communicating research inform of laundry list - Proposing a new way to use the evidence

The growing popularity of empirical research in development economics has resulted in a rise in the number of studies on the impact of 'interventions'. All of these are supposed to help policy makers make informed decisions and build a culture of evidence-based policy-making.

The summary of the literature on the impact of interventions is often communicated in the form of a laundry list of interventions that work, didn't work and some occasional cases of negative impacts. For example: increasing inputs - no effects, reducing PTR - mixed effects, teacher incentives - mixed effects, data-driven instruction - positive ve effect, and so on.

Apart from the first and obvious concern of 'will such intervention work in my context', communicating the evidence in the form of such laundry list is of relatively little utility to the decision maker, for many reasons.

1. Laundry lists are of little utility in case of prerequisites

What will the decision maker do with the evidence of interventions on the effectiveness of ''prerequisites"? By 'prerequisites' we mean interventions like building schools (where there are no schools), easing access to schools, building toilets, reducing PTR etc.

If the laundry list says that spending on infrastructure doesn't result in outcomes, does that mean that the decision maker should stop spending on improving the infrastructure? Obviously no. 

If the laundry list says that School Management Committees (SMCs) are ineffective, does that mean that the decision maker should scrap SMCs? Obviously no.

The decision will and should continue improving infrastructure and institute SMCs despite such evidence. Thus, looking at this evidence in the form of laundry list adds little utility.

2. Laundry lists are of little utility in case of mixed evidence

What should the decision maker do with a statement which says that there is mixed evidence on reducing PTR? Does that mean that the decision maker should continue the situation of classrooms with 1 teacher per 100 students? Obviously no.

3. Laundry lists conceal the capability deficiencies.

Most often, the binding constraint in low-capacity systems is the capacity to 'do things'. Thus, prescribing an 'act' to be done to such 'low-capacity' systems may mislead the decision maker and not be effective in long-term.

For instance, prescribing 'data-driven instruction' to low-capacity systems make them pursue the reform with the idea (data-driven instruction) as an anchor. It conceals the capacity deficiencies to pursue that idea. 

The analogy of a patient suffering from vomitings due to digestive system failure is helpful here. Let's say that patient 1 is vomiting whatever he eats because of weak capacity to digest and as a result has developed weakness. Now suppose that you identify the weakness and venture out to strengthen the person. You test a type of food on another weak person (patient 2), who is weak not due to weak digestive system but lack of food, and find the new food to be useful in addressing weakness. You come and prescribe the patient 1 to eat that food. Would that be useful? Most likely patient 1 throws up that too.

'Food' here is equivalent to 'idea' and 'digestive system' is equivalent to 'state capacity'. Pursuing treatment of a weak person by prescribing 'new food' puts food (idea) as anchor of the reform. It is useful in cases where the binding constraint is lack of food but not in case where the binding constraint is the lack of capacity. Since this way of treatment identifies lack of food as the binding constraint, if one idea doesn't work the problem is ascribed to food and pursues treatment by hopping from one food to the other, driving the focus away from the problem of weak capacity.

Similarly, 'prescribing' new ideas to be implemented in form of laundry lists to weak capacity systems incorrectly identifies the binding constraint as lack of 'ideas' and puts ideas as the anchor of reform. Pursuing reform with such ideas as anchors shifts the focus away from identifying and addressing the capacity deficiencies.

How to use evidence on the impact of interventions?

1. Evidence on the impact of interventions to counter cognitive traps like NFC etc.

Psychologist Jerome Kagan posited that human mind is averse to uncertainty and ambiguity. We respond to such ambiguity by spontaneously arriving at conclusions or solutions. When we can't gratify our desire to know, the pressure to arrive at a  'solution' increases. Aries Kruglanski, a social psychologist, coined a term for this called 'cognitive closure'. Such desire to arrive at conclusions is termed  as 'Need For Closure' (NFC). [good summary here]

Probably most people in policy space are subjected to this Need For Closure (NFC) phenomenon. As people working in this space, they are motivated to know the answer for 'reason for poor learning outcomes' and there is a pressure from others reaching out to them to 'prescribe' an answer, creating a ripe moment to be subjected to NFC. The decision makers also at times either don't like to engage with the complexity or don't have the bandwidth to do so and are in search for 'actionable points'. The desire for such 'action' mandates people to arrive at conclusions. At times, it also happens because of the desire to sound unique. Whatever may be the reason, everyone is in a pressure to arrive at the answer. This may hamper our view from looking beyond the visible.

What's more important is that human mind is subjected to 'seizure' and 'closure'. 'Seizure' - to arrive att conclusion; 'Closure' - to believe that it's the solution and closing oneself to other possibilities. This results in deeprooted beliefs about causes and solutions.

For instance, consider a journalist or a policy person who visits a school in order to understand the reasons for poor learning outcomes. The journalist finds that students don't have textbooks. This stark picture leads the person to conclude that 'lack of books' is 'the binding constraint'. It also prevents the person from seeing beyond the problem of 'lack of textbooks'. Similarly, there might be other competing hypotheses at any given point of time, like engaging teachers in non-academic activities, large class size etc. At times, these are also used as alibis.

How do we know 'what's correct?'. Is the problem really with large class size or is it just that it is being used as an alibi? Will these teachers show better performance if class sizes are reduced? Similarly, is the problem really with the lack of textbooks or are there other compelling reasons which we aren't able to think through? 

Evidence on interventions helps us get information on such counterfactuals. Instead of presenting evidence on monetary incentives to teacher in form of laundry list saying that teacher incentives work or don't work, the other way to frame it is to say - given all the constraints that are reported by people which are possibly restricting teachers, they are performing better when they were given money. It means that there is unutilized potential or teachers aren't doing up to their full potential, to the extent that they should be doing given the constraints.

Evidence should thus be communicated in a way to break through the NFC biases, pushing people to engage with the complexity, with an underlying message of "this may seem like a problem but needn't be so OR there's more to the problem, let's explore more and not stop at these conclusions." 

Presenting evidence in the form of laundry lists does little to achieve this.

[My book does such analysis focusing on breaking through NFC biases. Download softcopy for free here. Order hardcopy here.]

2. Evidence on impact of interventions to save the precious 'discourse bandwidth'

In the general discourse, the broad education itself gets limited attention. Within that discourse, there are always competing ideas of 'what has to be done', limiting the 'bandwidth' available to each idea. Any new idea potentially crowds out others in the already precious 'discourse bandwidth'. Thus, it is essential to weed out ideas that consume this precious space.

For instance, there is a strong belief as reflected by many people that salaries of public school teachers aren't enough to attract good people thus resulting in the poor education system. The question however is, is the problem with the lack of good people or not utilizing the people in the system to their potential? Will increasing salaries result in outcomes?

Evidence on interventions which say that contract teachers perform on par with regular teacher suggesting the unutilized potential can be communicated in such form to weed out such perceptions from the discourse.

3. Evidence on impact of interventions to build theoretical underpinnings

Aries Kruglanski who coined the term 'cognitive closure' is also known for developing the 'regulatory mode theory'. It essentially means that there are two character traits of people, which decide their way to puruse a given goal - locomotion and assessment. People geared towards locomotion mode have a bias for action, they want to start to 'get going' and 'get things done'. On the other hand, people geared towards 'assessment mode' take time to think, analyze various possibilities, road blocks ahead and then start pursuing the goal. 

Essentially both of them are trying to venture out into dark in order to figure a way out. Each of the above have their own advantages and disadvantages. Locomotion mode is helpful in situations where there is a possibility of quickly 'iterating' and 'adapting' based on feedback. In cases where it isn't possible, the stakes and opportunity costs are high, it's better to pursue 'assessment mode'.

Education, probably the area where the laundry list of interventions is popular, aptly fits into the category of high opportunity costs and stakes. It is thus useful to build a broad theoretical framework using the evidence available to identify and map the individual components of the system which go into making it work.

There is a practice of thinking of theoretical underpinnings in other areas issues but there is dominance of micro-research and little emphasis on macro approach basing on the evidence from micro-approach in education. Example of one such theoretical framework by me below. Read pages 72 to 92 of my book for its description. [Link directs you to a form where you have to enter your mail id. The book is free to download. Hardcopy can be ordered in India here.]

4. Evidence on impact of interventions to identify the 'type' of capacity deficiencies

Coming back to our analogy of digestive system and food, it is to be noted that it isn't incorrect to prescribe food for people with a weak digestive system but there's a way to do it. First step is to identify that the problem is with the capacity and not food per se. The aim then is to improve the capacity.

How to do that? Give a particular food, instead of just reporting that it isn' being useful, find out the enzymes that are required to digest it and see what's missing and fix that. Consider another food, repeat the process and over the type you built a functioning digestive system.

This approach, while still requires food, considers it only as secondary. The primary emphasis is on enhancing the capacity, in other words, it pursues reform with enhancing capacity as the anchor.

Similarly, in the case of education, while we start with a promising idea, it is essential to identify the types of capacities required to make it work, check if the system has such capacities, try to build that capacity, see if the capacity is built by using the promising idea as an instrument. Slowly, in this manner, while trying to make the idea work, but with emphasis on building the necessary capacities, we are enhancing the long-term capacity of the system. This approach has enhancing capacity as the anchor of reform and idea is only secondary.

No comments:

Post a Comment