Evaluation

Evaluation should answer why and how programmes have succeeded or failed and allow desired changes to be planned for improvements in implementation. It is an activity that should allow space to reflect upon and judge the worth (value) of what is or has been done. Evaluation can be seen as a cyclical process as shown in Figure 15.2. You should read this diagram by starting at the top and then following the arrows down the middle. From the top, data are collected, then analysed and then evaluated. If the aims of the programme are being achieved (the arrow to the right), no action is required but results should be reviewed and feed into the next evaluation if there is one. However, if the aims were not achieved (the arrow pointing down), action is required in the form of review of the management plan and changes to the way the programme is implemented. This will then require further evaluation in due course (the arrow going back up to the top) to assess the success of the revised programme.

Figure 15.2 Data collection, evaluation and management intervention cycle.

Unlike monitoring which is a regular activity, evaluation will be conducted only when there are evaluation questions that need to be answered. There is no fixed schedule for this but it may happen:

• mid-term
• at the end of a project
• three to five years after the project is completed (to assess the total impact).

Evaluation questions may arise from monitoring data or any other observations that lack in-depth information to explain observed levels of performance or effectiveness. In such cases, evaluation provides useful data on how and why programmes succeed or fail.

Evaluation as an activity may be related to processes, outcomes or impacts of a programme.

Process evaluation, as the name suggests, looks at process questions and can give insight into whether the project is on track or not, and why. At the end of a project, it would involve a review of all the project processes, from start to finish. Process evaluation aims to explain why things have happened in the way they have and answers questions such as:

• Was the intervention implemented according to expectations and the agreed management plan?
• Did the intervention meet the required standards of quantity and quality?
• Can the reasons for the success or failure of the intervention be assessed?

Outcome evaluation is the assessment of what the intervention has achieved. For example, if the intended outcome was to reduce open defecation in a target population, evaluation questions could be:

• Is there a change in the level of open defecation by the target population? And to what extent?
• What contribution did the project make to the observed level of change?
• Was the change that was attributable to the project adequate to meet project goals?

In most cases, outcomes or impacts are influenced by more than one factor and by other changes in a situation. For this reason, an outcome evaluation needs to be designed so that it is possible to estimate the difference between the current outcome level and that expected if the intervention was not in place.

Impact evaluation is the systematic identification of the effects (positive or negative and intended or not) on individuals, households or communities caused through implementation of a programme or project (The World Bank, 2004). Impact evaluations can vary in scale. They may be large surveys of target populations that use baseline data and then a follow-up survey to compare before and after. They could also be small-scale rapid assessments where estimates of impact could be obtained from a combination of group interviews, focus groups, case studies and available secondary data.

Impact evaluation can be used to:

• measure impacts of a programme and distinguish these from the influence of other external factors
• inform management decisions on whether to modify, expand or abolish programmes
• learn lessons for improving the design and management of future activities
• compare the effectiveness of alternative interventions
• help to clarify whether costs for a programme or activity are justified.

The advantages of impact evaluation are that it provides estimates of the magnitude of outcomes and analyses the impacts for different demographic groups, households or communities over time. It should show the extent of the difference that a programme is making and allow plans for improvement to be made. However, it needs competent managers and some approaches can be very expensive and time-consuming.