What is Evaluation

What is evaluation

There are many definitions of ” What is Evaluation” depending on the context and organization, but most have a common thread. If you’re a Monitoring and Evaluation (M&E) specialist like the members of EvalCommunity, you probably do. 

In this article, we will discuss the different approaches and definitions available in “What is Evaluation”. Definition Made Simple:  from around the world.

Table of Contents

  • Do you know what is Evaluation? (EvalCommunity)
  • What do you mean by evaluation? (Canadian Society of Evaluation)
  • What do you mean by evaluation? (American Evaluation Association)
  • What do you mean by evaluation? (UNEG)
  • What do you mean by evaluation?  (OECD/DAC)
  • What do you mean by evaluation (Australia)
  • Conclusion on “What is Evaluation”

Do you know what is Evaluation? (EvalCommunity Definition) #

Evaluation is the process of examining the performance of an organization, program, project, policy, or any other intervention to determine its relevance, adequacy, effectiveness, efficiency, and progress for the purpose of identifying areas for improvement. Essentially, it refers to the combination of evidence and values to determine whether an intervention has merit, worth, or significance.

Evaluation is a complex and dynamic process that requires the use of a variety of data collection and analysis techniques to assess the performance of an organization, program, policy, or project. Evaluation can be used to inform decisions about how to improve the effectiveness of programs, projects, policies, and organizations.

Evaluation Standards #

Evaluation is the process of judging an intervention based on a set of standards. The 7 evaluation standards are the relevance, adequacy, progress, efficiency, effectiveness, impact, and sustainability of an intervention.

  • Relevance, is the intervention doing what it is supposed to do? Relevance involves looking at differences and trade-offs between different priorities or needs. In order to determine whether the intervention can (or has been) adapted to remain relevant to the current context, it needs to be analysed to see if any change in it has been made in the context. The intervention is relevant if it answers the needs, policies, or priorities it has been designed to meet.
  • Adequacy, is the intervention equal to our needs? The state of being sufficient for the purpose concerned. The intervention is adequate if it meets its requirements and if its activities are being carried out in accordance with a planned schedule.
  • Progress, how successful is the intervention in achieving the goals it was set out to achieve? The movement toward a goal, an improved or more developed state, or to a forward position. An intervention accomplishes progress if its activities are carried out according to the schedule it has set.
  • Efficiency, is the resource being used to the best of its ability? The extent to which objectives are achieved by minimizing the use of resources. Efficiency, the relationship between inputs and outputs in the production of goods and services, is obtaining the best possible value for the resources used. The intervention is efficient if it makes the most efficient use of the resources devoted to it.
  • Effectiveness, is the intervention achieving its objectives? The degree to which objectives are achieved and the extent to which targeted problems are resolved. The intervention is effective if the results obtained are in accordance with the objectives and targets for reducing the dimensions of a problem or for improving an unsatisfactory situation. Its impact is related to the overall effect it has on people and the environment.
  • Impact, what difference does the intervention make? The extent to which the intervention has had long-term positive or negative effects at a higher level, either intended or unintentional. An impact evaluation provides information about the observed changes or ‘impacts’ produced by an intervention.
  • Sustainability, will the benefits last? The extent to which the net benefits of the intervention continue, or are likely to continue.

Different Evaluation Labels #

We also include evaluation known by different labels when we talk about evaluation as part of evaluation as a whole:

  • assessment
  • review
  • social impact analysis
  • impact analysis
  • appreciative inquiry
  • cost-benefit analysis

Process and Outcome Evaluations #

The most commonly used process and outcome evaluations in International Development are:

Process evaluations are looking at how a program works. This type of evaluation tends to focus on the who, what, where, and how.

There are a number of questions that are often asked during a process evaluation:

  • Was the program content delivered as the program developers intended?
  • Was every program session delivered?
  • Did participants attend each session?
  • Who attended the program and were they from the population group the program was designed for?
  • Are staff appropriately qualified to deliver the program?

An evaluation focused on outcomes seeks to determine whether the program has met its objectives, how well (or how much) it has met those objectives, and for whom it has achieved them. It is also possible to conduct outcome evaluations to examine any unintended outcomes or changes that have been produced by a program, policy, or service. These types of evaluations are sometimes referred to as “impact evaluations”. An outcome evaluation often asks:

  • Did the program achieve its intended outcomes or goals? To what extent? Did it have a positive effect? How much of an effect?
  • Who benefited from the program or service? Who did not benefit? Why?
  • Have there been any unintended outcomes for participants? 

In order to understand whether a program or service has achieved its goals, it’s often useful to collect data at different times. This can be necessary to assess whether or not change has actually taken place and, if so, how much change. For example, data might be collected at the start and at the end of a program or at defined time points after a client enters a service (e.g. six months or a year after entry or when they leave).

You can find further guidance on how to design an evaluation that suits your needs here.

What do you mean by evaluation? (Canadian Society of Evaluation) #

Systematic assessment of the design, implementation or results of an initiative for the purposes of learning or decision-making. There are many definitions of evaluation depending on the context and organization, but most have a common thread. If you’re an M&E specialist like the members of EvalCommunity, you probably do. 

Systematic: It should be as systematic and impartial as possible (UNEG, 2005). It is methodical, providing information that is credible, reliable, and useful to enable the incorporation of lessons learned into decision-making process of users and funders (OECD, 2010). It is based on empirical evidence and typically on social research methods, thus on the process of collecting and synthesizing evidence (Rossi Lipsey and Freeman, 2004). Conclusions made in evaluations encompass both an empirical aspect and a normative aspect (Fournier, 2005). It is the value feature that distinguishes evaluation from other types of enquiry such as basic science research, clinical epidemiology, investigative journalism, or public polling.

Assessment: It considers value, merit, worth, significance or quality (Scriven, 1991).  It identify what works, for whom, in what respects, to what extent, in what contexts, and how (Pawson and Tilley, 2004). It examines expected and achieved accomplishments, the results chain, processes, contextual factors and causality in order to understand achievements or the lack thereof (UNEG, 2005). It may focus on a broad range of topics including relevance, accessibility, comprehensiveness, integration, fulfillment of objectives, effectiveness, impact, cost, efficiency, and sustainability (Patton, 1997; OECD, 2010). The evaluation process normally involves some identification of relevant standards, some investigation of performance on these standards, and some integration or synthesis of the results to achieve an overall evaluation (Scriven, 1991; OECD, 2010).

Initiatives: It can focus on any kind of initiative such as programs, projects, sub-programs, sub-projects, and/or their components or elements (Yarbrough et al, 2011; Scriven, 2003).

Purposes: It can be conducted for the purposes of decision making, judgements, conclusion, findings, new knowledge, organizational development and capacity building in response to the needs of identified stakeholders leading to improvement, decisions about future programming, and/or accountability ultimately informing social action ameliorating social problems and contributing to organizational or social value (Yarbrough et al, 2011; Patton, 1997).

What do you mean by evaluation? (American Evaluation Association) #

It is a systematic process to determine merit, worth, value or significance. So what does that mean in practice? Let’s use one kind of evaluation, program evaluation, to illustrate. Programs and projects of all kinds aspire to make the world a better place. Program evaluation answers questions like: To what extent does the program achieve its goals? How can it be improved? Should it continue? Are the results worth what the program costs? Program evaluators gather and analyze data about what programs are doing and accomplishing to answer these kinds of questions.

A program evaluation has to be designed to be appropriate for the specific program being evaluated. Health programs aim to make people healthier and prevent disease. School programs strive to increase student learning. Employment training programs try to help the unemployed get jobs. Homelessness initiatives work to get people off the streets and into safe housing. Chemical dependency programs help people using alcohol and drugs. Community development programs plan initiatives to increase prosperity among those in poverty. Juvenile diversion programs try to keep kids out of jail and put them on a path to becoming productive adults. For each kind of program, an evaluation would gather and analyze data about that program’s effectiveness. But program evaluation is only one kind of evaluation.

What do you mean by evaluation? (UNEG) #

The definition of the evaluation was also developed by the United Nations Evaluation Group (UNEG). The key part of the definition is that evaluation is: “An assessment, as systematic and impartial as possible, of an activity, project, programme, strategy, policy, topic, theme, sector, operational area, or institutional performance. It analyses the level of achievement of both expected and unexpected results by examining the results chain, processes, contextual factors, and causality using appropriate criteria such as relevance, effectiveness, efficiency, impact, and sustainability.

What do you mean by evaluation?  (OECD/DAC) #

The OECD DAC Network on Development Evaluation (EvalNet) has defined six evaluation criteria – relevance, coherence, effectiveness, efficiency, impact and sustainability – and two principles for their use.

These criteria provide a normative framework used to determine the merit or worth of an intervention (policy, strategy, programme, project or activity). 

A systematic assessment of a planned, ongoing or completed intervention to determine its relevance, efficiency, effectiveness, impact and sustainability. The intent is to incorporate lessons learnt into the decision making process. Adapted from OECD/DAC Glossary, 2002)

What do you mean by evaluation (Australia) #

Evaluation refers to the systematic process of assessing what you do and how you do it to arrive at a judgment about the ‘worth, merit, or value of something (Mertens & Wilson, 2013; Scriven, 2003–04). Essentially, evaluation involves taking a series of planned steps in order to better understand a program or service.

Evaluations that focus on process want to understand how a program works. This type of evaluation is typically focused on the who, what, where and how (Centers for Disease Control and Prevention [CDC], 2018).

Conclusion on “What is Evaluation” #

As you can see, the definition of evaluation is broad.

  • There are many different ways that people and organisations use the term “evaluation” based on their diverse professional and educational backgrounds, training, and organisational context.
  • There are many types of evaluation designed for different situations and with different objectives. 
  • Evaluation is an important tool for understanding the effectiveness of a program or initiative.
  • Evaluation helps identify what is working and what isn’t so that solutions can be found to improve the project or initiative. It can also reveal elements of a program or initiative that could be better utilized and determine if alternative methods should be adopted to support program goals.
  • Ultimately, evaluation helps to ensure that programs and initiatives are reaching their desired outcomes.