Lessons from evaluations - Ensuring a Learning India S6 E.004


[37th post in the 'Ensuring a Learning India' series. 42 posts in total. One post per day. 5 more to go]

The previous section summarized our discussion till now. In this section, before moving on to build a framework for reform, it is useful to once again recollect the lessons learnt from evaluations[viii]. This will help us prioritize right sets of policies.

All the caveats of evaluations mentioned in section 3.2 apply here. The important one being necessary vs. sufficient conditions. If an evaluation doesn’t show any impact, it necessarily doesn’t mean that that policy should be discontinued. It only means that we may do it but shouldn’t be prioritized in the precious limited discourse bandwidth. For example, if spending on infrastructure doesn’t lead to outcomes, it doesn’t mean that classrooms shouldn’t be renovated or better facilities be provided at the school. It just says that these may be the essential things to be done but one shouldn’t stop by merely providing this, thinking that lots of money and efforts have been put. It also shouldn’t be given priority in the policy discourse. We just need to move onto exploring other things.

The second important caveat being external validity. Some argue that evaluations in one context aren’t necessarily applicable in other contexts. It is true that the outcomes of the intervention may vary across the context, if you just look at the final metric. However, the information on mechanism of impacts can be valuable which can be used across contexts. For example, the evaluation outcome may say that distributing free textbooks may not result in increase in test scores but it may also say that textbooks increased scores of only top 10% children in test score distributions. This suggests that students who already have the ability to read are able to take advantage of the books but others aren’t able to.

If one reads the list of interventions and the results in table format, it may seem obvious and may not seem much useful but one should see it from the perspective of a person inferring data to make policy decisions. The intention of the person is to identify issues and prescribe or act on them. Suppose you are looking at DISE data and find out that there are only 3 teachers per primary school in India (let’s say), the immediate instinct is to say that there are less number of teachers and hence you need more teachers. You get convinced that this is the critical constraint and put all your efforts in recruiting teachers and ensuring that there is adequate number of teachers. It is very much necessary. Of course, who disagrees that there should be at least one teacher per class? After few years, you may end up realizing that this didn’t improve the situation. Similarly, if you visit a school and find that students don’t have text books. You again get convinced that this is the critical constraint and then put all your efforts in ensuring textbooks but you may again end up realizing that this didn’t improve outcomes.

Does that mean that teachers shouldn’t be appointed and textbooks shouldn’t be given? They should be done by all means. The difference here is, whether you consider it as a constraint or critical constraint. If you consider this as a critical constraint, you prioritize it over everything else, devote all your energies, and occupy the precious discourse bandwidth but not improving the outcomes. If you consider it just a constraint you will do it but you wouldn’t focus all your energies on that, you would also explore other root causes and address them. It is easier said than done because visible deficiencies tend to have more effect than the invisible root causes. So, how does one distinguish a constraint and a critical constraint? Data or evaluations help us. The results of evaluation presented below should be looked at from this perspective. 

1. Infrastructure – Does spending on infrastructure alone lead to improvement of outcomes? One way to look at is to just look at the expenditure on infrastructure over the years and also the corresponding outcomes. This study looks at the representative sample of 1,250 villages across 19 states and finds no correlation between improvements in infrastructure and learning outcomes.

Increase in spending and spending on infrastructure and personnel is an interesting debate in India. Any efforts towards increase in spending, invites comments arguing that increasing spending doesn’t result in outcomes, which is at times misleading. Any new intervention requires additional money, thus skepticism towards increase in spending may not be appropriate. It is better to argue for spending on appropriate aspects which can yield high returns or those that are necessary and need immediate attention, even if they don’t show concrete returns.

2. Contract teachers – We have discussed this in earlier section too. It is very often argued that in the past few years, untrained contract teachers have been hired instead of recruiting regular teachers, which led to de professionalization of education and hence affected the outcomes negatively. However, the evidence doesn’t seem to suggest that.

In this study[i], researchers randomly allocated contract teachers in 100 schools of AP and find that contract teachers are as bad (good) as regular teachers. In another study[ii], researchers used measured the effects when a regular teacher and a contract teacher teach the same students (different subjects). They find that contract teachers are actually produced better learning outcomes. There is also evidence[iii] to suggest that contract teachers exert higher levels of effort than regular teachers, as measured by teacher attendance and engagement.

Overall, there is no evidence to suggest that contract teachers affected the system negatively. If anything, in some cases they added value.

This doesn’t mean that all teachers should be taken on contractual basis, and that no training should be given. It just means that diagnosis of contract teachers affecting outcomes negatively may not be correct.  It is time to reconsider these arguments which occupy the precious discourse bandwidth.

3. Teacher pay – The fact that contract teachers, who receive a fraction of salary of regular teachers are being more productive in some cases and in worst case aren’t performing lower than the regular teachers, suggest that low teacher pay may not be a critical constraint. Other studies also find no correlation between teacher pay and performance.

4. Pupil – Teacher Ratio (PTR) – There is mixed evidence on the effect of PTR on student test scores. However interpretation of this evidence needs caution. It is true that a small class size is required for effective teaching. Any good private school wouldn’t have class sizes exceeding 30. Many activities especially personal attention require small sizes. Thus there is a rationale for having class sizes less than 30. The problem being, in the contexts of weak state capacity, even if the class sizes are reduced, if the attendance of teachers and their efforts don’t improve then there might not be much effect on students. Thus, from a Return on Investment point of view, this may seem as an unproductive investment, especially when compared to other interventions which yield results with the same class sizes as they are now.

I personally believe that subject to resource constraints, class sizes can be reduced. The rationale being, overcrowded classroom is the common complaint of public school teachers. They strongly believe that this is impeding their teaching processes and there is strong resentment that it is unfair to expect results from them with such constraints. For a person monitoring teachers’ performance, it is hard to refute this reality. If we address this concern, there may be stronger reason to demand accountability of teachers.

4. Information provision on student learning outcomes – School Management Committees (SMCs) are instituted with the aim of involving communities in administration of the school. It is argued that parents can make better use of these channels if they have information about poor learning outcomes of students. It is also argued that providing information on functions of these committees and assisting in organizing meetings can in enforcing accountability and thus better learning outcomes. We earlier discussed an experiment in Uttar Pradesh where providing parents with information on learning outcomes of students and assisting in organizing these meetings didn’t result in improvement of learning outcomes.

Here again, this doesn’t mean that SMCs should be abolished and there shouldn’t be focused up on. Successful performance of any intervention, even in future, requires teachers to do their duty and SMCs can be one of the ways to ensure this. However, this suggests that we shouldn’t completely rely on SMCs to make this happen and also put complimentary efforts to address this issue.

5. Remedial instruction – Providing remedial instruction to students who are behind the class level is one of the interventions found successful to be across contexts, even when implemented by volunteers from local community[iv] [v][vi] and regular teachers outside the classroom[vii].

6. Teaching to the Right Level – We earlier discussed a study in Kenya where students were grouped into sections as per their ability, resulted in increase in test scores and an evaluation in Haryana which used pedagogy aiming to teach the student at the right level, which was effective.

7. Teacher incentives – In the section on monetary incentives to teachers, we discussed that the study in Andhra Pradesh, with appropriately designed teacher incentive mechanism resulted in enduring improvement of learning outcomes. We also discussed the studies in other countries where teacher incentives weren’t effective and the complexities around the monetary incentives.

The existing evidence provides us these pointers which can help us see through the fog of emotions, biases and hypotheses.  The next post builds on the discussion till now, summarized in the earlier section, along with these pointers to build a framework for school education reform in India.



[i] Muralidharan, Karthik, and Venkatesh Sundararaman, Contract Teachers: Experimental Evidence from India
[ii] Kingdon, Geeta Gandhi, and Vandana Sipahimalani-Rao. 2010. "Para-Teachers in India: Status
and Impact." Economic and Political Weekly no. XLV (12):59-67
[iii] Goyal, Sangeeta, and Priyanka Pandey. 2011. "Contract teachers in India." Education
Economics:1-21
[iv] Banerjee, Abhijit, Shawn Cole, Esther Duflo, and Leigh Linden. 2007. "Remedying Education: Evidence from Two Randomized Experiments in India." Quarterly Journal of Economics no. 122 (3):1235-1264.
[v] Banerjee, Abhijit, Rukmini Banerji, Esther Duflo, Rachel Glennerster, and Stuti Khemani. 2010. "Pitfalls of  participatory Programs: Evidence From a Randomized Evaluation in Education in India." American Economic Journal: Economic Policy no. 2 (1):1-30.
[vi] Lakshminarayana, Rashmi, Alex Eble, Preetha Bhakta, Chris Frost, Peter Boone, Diana Elbourne, and Vera Mann. 2012. Support to Rural India’s Public Education System: the STRIPES Cluster Randomised Trial of Supplementary Teaching, Learning Material and Additional Material Support in Primary Schools.
[vii] Banerjee, Abhijit, Rukmini Banerji, Esther Duflo, and Michael Walton. 2012. Effective Pedagogies and a Resistant Education System: Experimental Evidence on Interventions to Improve Basic Skills in Rural India. MIT.
[viii] Priorities for Primary Education Policy in India’s 12th Five-year Plan by Karthik Muralidharan has good summary of evaluations in Indian context and the lessons from it. It was the main resource referred to while writing this section and I would like to acknowledge that.

No comments:

Post a Comment