Pitfalls of the 'seemingly obvious' - Ensuring a Learning India S3 E.001
'Pitfalls in translating observations to policy' is the third theme in the 'Ensuring a Learning India' series. This theme discusses the commonly observed pitfalls while devising a policy option based on observations or data. It is important to understand and recognize these, moving ahead.
There are three major pitfalls in the process of translating observations into policy. One, focusing on the seemingly evident. Two, arguments based on examples and inferring from successful systems. Three, difference in moral frameworks. This theme will discuss each of these.
This post discusses the first pitfall, focusing on the seemingly evident. Seemingly evident observations are those which are striking and hence seem obvious. This can be in case of people both inside and outside the system. For people inside the system, if something troubles them on a constant basis, then out of this discomfort, they may exaggerate the extent of the bad effects of the issue and highlight that as *critical* constraint in achieving the outcomes. For example, outside school duties for teachers. It may be true that this issue is coming in the way of teachers' functioning but some may overestimate its impact and perceive it to be the only critical constraint. Similarly, for people outside the system; if one visits a school and finds that students don't have uniforms, one of the immediate response could be - this is sad, students need tidy uniforms. Some of these can be touching emotionally and hence remain at the top of our mind even after the visit thus occupying the important space. In such cases, it is possible for the person to feel that, getting uniforms is the *critical* thing that should be done to improve schooling.
What's wrong with these policy proposals? Don't we need uniforms for students? Shouldn't we stop teachers from spending time outside classroom? It is true that all these are necessary but there are five issues with such approach of arriving at policy conclusions based on mere observations that seem obvious.
One, the observed phenomenon might actually be a minor issue. For example, the actual data of time spent by teachers outside school may show that it isn't much. Two, carefully designed studies often show that such policy interventions (free textbooks, infrastructure etc.) don't have any effect on the outcomes, thus showing that these issues are not the critical constraints as they seem to be. Three, occupying significant bandwidth of policy narratives. There is a limited bandwidth for policy narratives in any sector; all the issues and solutions can't be highlighted at the same time. In such contexts, bringing up an issue which is seemingly evident for naked eye but not so in reality, crowds out other important agendas. Four, when there is too much focus on such policy agendas, resolving them gives a sense of great achievement and complacency sets in, limiting our attention to other issues. Five, focus on the seemingly evident issues might restrict us from exploring deeper to identify the core issues.
What's the solution? The first solution is to do a litmus test for all our assumptions and perceptions. If someone says, outside school duties take up large part of teachers' time, it would be useful to check the data of time spent by teachers outside the school. Gulzar Natarjan makes an excellent point on similar lines using the judgments data from High Courts. Most judges complain of overload but when the actual data is seen, it tells the true picture. Some courts are really overloaded but some aren't relatively to others. In the absence of such data, everything becomes a matter of perception and philosophy and the debates are never ending. The second solution is to continuously track the outcomes to quickly cross check. But, such quick cross checks and simple correlations might not be enough if there are high stakes and opportunity costs associated with the decision. For example, if the budget of a department is limited, there are opportunity costs associated with investing money in a policy, as there are other possible suggested alternatives losing out. This underscores the need for *evaluating* these interventions to know their true effect.
An evaluation of this kind should give information about the causal effects between the policy intervention or inputs and the outcome. This information is helpful in numerous ways. One, it tells us if that particular issue is a critical constraint in that context. Two, it structures the policy debates and gives them a direction to the p the debate. As discussed earlier, without such evidence, everything is based on philosophy and perceptions, which are highly subjective. Three, if the evidence is not positive, it forces us to think deeper for the core issues. May be we would have realized this some time later but it generally takes long time to realize such nuances and even in those cases, the realization is often due to evidence based on such evaluations. Thus, having such information based on evidence from evaluations reduces the time involved in closing the feedback loop.
In summary, be cautious of seemingly evident aspects. At some point of time, even the concept of sun moving around the earth was seemingly evident. It couldn't have been more obvious.