Do better enrollment rates mean better education outcomes? Analysis of a Cash Conditional Transfer in Honduras

When we think about experiments, we tend to picture microscopes and beakers. The advantage of natural lab experiments is that the research is in highly controlled environments where they can isolate and manipulate variables. In contrast, social experiments present more challenges because the research might use the wrong measurements, miss controlling for an important causal factor, or use an incorrect method.  However, in the policy realm it is crucial to continue measuring the effects and impact of different factors in order to provide politicians with information to design better public policies.

 

 

There has been a great deal of discussion among government officials and scholars about the most effective way to break the cycle of poverty, and one of the most popular methods is using Conditional Cash Transfers (CCT) programs. In these programs, money is given to a household that meets certain criteria. These are usually related to the children going to school or regular medical check-ups.  I am especially interested in testing effects in education outcomes. I want to find out if a CCT has an effect on grade progression and if the increased demand in school enrollment led to success in school, which is measured by grade progression.

Honduras is the second poorest country in Central America. In 1990, they implemented the Programa de Asignación Familiar or Family Allowance Program (PRAF) CCT. The program, PRAF II, was redesigned to improve previous weaknesses. To analyze the effect of PRAF on education performance I used the difference-in-difference technique based on panel data collected in a random experiment. To understand the validity of this experiment, we need to understand its various components.

What was the sample framework?

In 2000, the Honduran government conducted a randomized experiment in 70 of the poorest municipalities of the country. The data was collected on every individual in each program household in 2000 and 2002. The survey from 2000 was collected before the program was implemented and the survey from 2002 was collected for the current year.

Who was eligible for the program?

Education transfer was available to families with children ages 6 to 12 who were enrolled and regularly attended grades 1-4.

Health transfer was available to families with children under 3 and pregnant others who regularly attended health centers.

How are treatment and control groups structured? 

In order to have a good treatment group there is a need to administer a large exogenous shock in a short period of time. The PRAF experiment had a total number of 15,350, which consists of 7,675 individuals (each individual having been surveyed in both 2000 and 2002) and 3,935 households.

Ideally, we would like to be able to compare the same treatment group and run the experiment again without giving them the treatment. Since we are not able to do that because of time restrictions, we need a counterfactual or control group that identifies the outcomes that would have happened in the absence of the treatment.  We hope that random selection makes the groups as similar as possible. In this experiment, child height-for-age was used as a poverty proxy to determine which municipalities would be eligible for the CCT program. They employed a system of geographic targeting to identify the communities where the chronically poor lived. The program drew up a list of municipalities based on the findings of the 1997 census and selected those whose chronic malnutrition indices fall below a standard deviation from the mean.

Based on this preliminary selection of eligible municipalities, a random statistical sample was then made to assign those which fell into the treatment and control groups. As a result, 40 eligible municipalities were assigned to treatment and 30 into control. Any of those households from the selected treatment communities was considered eligible for the transfer if it complied with the component criteria. The quality of the estimate is mainly based on the similarity of the counterfactual to the treatments group. Otherwise, the estimate will have bias.

How to assess impact in Education performance?

The education system in Honduras faces problems of general service provision, low teacher quality, low enrollment rates and low performance.  Children also tend to delay joining school, and their age on entry exceeds the normal age at that grade level.  Due to the levels of poverty in Honduras, it is common that children have to work instead of going to school. Moreover, the quality of the educational service is very poor.

The cash transfers in the treatment group were available for families for each child age 6-12 who was enrolled in the first four years of primary school and attended regularly. In my test, I restricted the sample so that we are only examining children between the ages of 6 and 12. The total observations were 2,648.

I used the difference-in-difference technique because it allows me to measure the impact with the data from the pre and post implementation of the program. This method mimics an actual experiment by measuring changes in average outcome in the treatment group before and after treatment minus the difference in average outcome in the control group before and after treatment.

The program requires enrollment and attendance in school in order to receive the cash. Therefore, it will be expected that these two variables will increase because of the pre-condition and not because it is a successful program. As a consequence, only measuring these variables (enrolled or attendance) will not actually show that the program is changing behaviors in children and improving the education performance.  In order to measure impact, I used two models. The first one measures the difference in enrollment and the second one that of grade progress.

It is important to include in each model some characteristics that affect enrollment and grade progress.  I controlled for years of education of head of household, distance to primary school, and size of household and household characteristics such as floor composition, availability of electricity.  In addition, I added the name of their municipality and if they received support from welfare program that is not PRAF.

In my first OLS regression using the variable “enrolled” I can prove that in fact there was an increase of 0 054 and it is statistically significant.

However, to find impact I applied the difference-in-difference formula to the “progressed” variable. The results show that the difference was of 0.05 and this was not statistically significant.

We can infer that even though the cash transfer motivated parents to enroll their children in school, when accounting for the original differences between the two groups and tracking those differences over time, we see that there is no significance in the results. In consequence, it appears unlikely that the PRAF program led to considerable improvements in grade progression from 2000 to 2002.

Are my results valid?

The key assumption for any difference-in-difference strategy is that the outcome in treatment and control group would follow similar trajectories in the absence of the treatment.

Common trend assumption is difficult to verify but using pre-treatment data could show that the trends are the same. Yet, even if pre-trends are the same we still have to worry about other policies changing at the same time. In this experiment, I cannot quantitatively demonstrate the common trends assumption because I do not have pretreatment data. However, the years previous to the experiment were not unusual. In other words, there was not a big economic shock or other macroeconomic factors that could have affected the municipalities included in this study. I am aware that PRAF was in place in some of the municipalities since 1990, so even if I had data from previous years the effect was already there. In addition, the number of observations and the randomization effect should help to overcome the lack of pre and post treatment data.

Conclusion:

    PRAF II did not show the increase in performance results as expected. The difference-in-difference is a good estimator, but the problem is that many factors can affect the assumption that our counterfactual is almost equal as to our treatment group. Therefore, it is important to use other techniques to prove our findings. CCTs have provided good results in countries such as Mexico with Progresa and should continue to be monitored and improved; hopefully allowing people to break the poverty cycle and better their lives.


Appendix:

I used fixed effects for municipality in order to control for any differences in the municipalities such as rural vs. urban, culture, weather, agricultural conditions, etc.