How process evaluation can help improve program design and implementation

3ie’s recently-published working paper ‘Incorporating process evaluation into impact evaluation – What, why and how’ by Senior Research Fellows Vibecke Dixon and Michael Bamberger lays down the guidelines that can provide impact evaluators with tools and ideas for exploring and adding relevant elements of process evaluations to experimental and quasi-experimental impact evaluation designs. This blog is the first in a two-part series that looks at different process evaluation approaches and the benefits of using process evaluation.

Process evaluation comprises the tools and techniques used to evaluate the effectiveness with which projects and other development interventions are implemented. A process evaluation will often complement an impact evaluation by helping understand how processes and events during implementation can affect the scale and distribution of intended outcomes. In many cases, the process evaluation may also identify important unplanned (positive and negative) project outcomes. The evaluation can address one or more of the following dimensions areas:

• How well the project was designed [p23, PE guidelines]
• How well the project was implemented [p31, PE guidelines]
• How the organizational structures and processes through which the project was implemented affected the achievement of the intended project outcomes and impacts [p34, PE guidelines]
• How outcomes and impacts were affected by the external context within which the project was implemented [p41, PE guidelines]

A process evaluation can serve several purposes. It can be used to strengthen the impact evaluation design, it can help improve project implementation, and it can provide recommendations on how to improve the design and implementation of future projects.

Process evaluation is not new – but current approaches may need strengthening

Many organizations are already using process evaluation, sometimes under a different name such as formative evaluation or implementation evaluation. For example, UN-Habitat used process evaluation to help explain factors contributing to the success of programs to provide adequate, affordable housing and reduce poverty; UN Global Pulse used process evaluation to assess the effectiveness of using big data to strengthen humanitarian programs, and the World Food Programme used it in various studies, including to assess program impact on children’s nutritional status.

Evaluation is considered formative when conducted during the development or delivery of a program or product with the intention of providing feedback to improve project implementation. Such studies are essential because implementation problems are a common cause of program failure [Mathison, S (editor) page 327, 2005, Encyclopedia of Evaluation, Sage Publications]. Most process evaluations use qualitative methods, many of which require significant amounts of time in the field – examples include observation of community meetings or project activities, long open-ended interviews, focus groups, and social network analysis.

3ie has also used process evaluation to complement an impact evaluation (see Table 1).

However, when process evaluation is used to complement an impact evaluation, it is often conducted under time and budget constraints, which can limit the kinds of qualitative data collection methods that can be used. Most process evaluations use qualitative methods, many of which require significant amounts of time in the field. These include observation of community meetings or project activities, long open-ended interviews, focus groups, and social network analysis. So, while many organizations are already using process evaluation, there is still room for the approaches to be systematized with larger budget and time commitments and with more specialized researchers.

How process evaluation is used

While a process evaluation should ideally cover all four dimensions discussed earlier, it is often necessary to focus on the specific aspects of the project implementation process which are of priority concern to clients. Areas on which a process evaluation may focus include:

• Assessing the quality of the initial diagnostic studies during project design
• Assessing the quality of the project design and the quality of the theory of change on which the implementation strategy is based
• Assessing different aspects of the implementation process (e.g., the implementation of a particular component such as the management of micro-loans, the administration of conditional cash transfer payments, or promoting water conservation), or the implementation of broad cross-cutting strategies such as promoting women’s empowerment or strengthening the agricultural value chain
• Assessing the effectiveness of the inter-agency organizational structures through which the project is implemented
• Understanding how the local contexts in which each project is implemented affect implementation and outcomes

Additional benefits of process evaluation

There are several other potential benefits of process evaluation:

1. It can provide additional information not available from project reports and records which many evaluations rely on. Importantly, this can include information on deviations from the project design, which agencies may be reluctant to report on, and on unplanned and sometimes negative outcomes, which are often overlooked.
2. While many impact evaluations focus on the careful selection of the counterfactual, they often provide less information on the factual (the project or other intervention being assessed). In contrast, process evaluation can provide detailed information on the project and how it was actually implemented.
3. It can also provide a deeper understanding of differential outcomes for different segments of the target population and some of the deeper reasons affecting project outcomes – who benefits, how, why, and when? This can also address the broader question of whether the project design and how it was implemented is able to address the needs of all sectors of the target population. Differential impacts of the project on women and men is an example of how this analysis can be used.
4. It can help understand which particular project components or activities contributed most to achieving particular outcomes.
5. It can also contribute to strengthening the theory of change (ToC) by providing richer information on the project and the context in which it is implemented and identifying critical assumptions in the causal chain.

The new 3ie guidelines provide a step-by-step approach to the design and implementation of a process evaluation. The second part of this blog series will discuss the design of a process evaluation.