OUTCOME EVALUATION
by Richard A. Krueger
Evaluating for Outcomes
- In the last five years there has been a major shift towards measuring outcomes
- We need to develop strategies to measure outcomes
- Measurement of outcomes is often more valuable for participants
and planners than it is for financial decision-making
- Measuring outcomes is difficult because:
- Outcomes take time to accomplish and measure
- Definitions of outcomes can vary and change over time
- Others must be involved in determining and measuring outcomes
- Sometimes we have no outcomes because what we thought would create change doesn't
- Many forms of evaluation exist. Don't overlook the value of other types of evaluation such as process, formative or developmental.
Why do we Seek to Measure Outcomes?
- We're told to do so by administrators and those who finance programs
- Future funding might be linked to our accomplishment of outcomes
- Our organization needs evidence of accomplishment in order to survive
- It helps us to know what we've accomplished
- It helps customers know what they've accomplished
- It helps partners know what we've accomplished together
Who Decides the Outcome? A Question Often Overlooked
- Customers
- Local educators
- Community leaders
- County commissioners
- Educational specialists
- Experts
- Foundation officials
- State legislators
- Policy analysts
- USDA officials
- US Senators and Representatives
- President
Different people at different levels will have different perceptions of outcomes. There is a tendency for experts, elected officials, and administrators to dominate the decision-making process. Don't forget that the customer may be the most critical person to listen to. In the classic evaluation of Head Start in the early 1970s, the evaluation showed that the cognitive changes of children washed out by the third grade. Experts and policy analysts along with a number of elected officials wanted to eliminate the program, but they changed their minds after hearing the cries of thousands of parents. Parents along with teachers pointed out that the program had a major benefit in preparing children for the social experience of school and that the program benefited parents and the community as well.
At What Level is the Outcome?
Outcomes can occur at a variety of levels-individual, family, community, state or national. It is important to consider the level that was targeted where you expected to see an outcome as well as other levels where unintended outcomes might occur.
Overview of Terms
The vocabulary of evaluation terms is evolving and occasionally experts will offer alternative definitions. Here are some of the terms in use today:
- INPUTS -- The resources invested in the program such as dollars
or days
- OUTPUTS -- Actions, activities, products, curriculum, etc. of
the organization
- OUTCOMES -- Changes that occur relatively soon after a program
- IMPACTS -- Longer-term changes which are a result of the program
NOTE: The terms "impact" and "outcome" are evolving and several definitions are currently in use. Both terms refer to the consequences or after-effects of programs. For some, these terms are synonymous and interchangeable. Others make subtle distinctions in the use of these terms. Often there is a time dimension between impacts and outcomes, with outcomes usually referring to more immediate consequences and impact to the longer-term consequences.
In addition there are some other terms that are occasionally used. These include:
- FOOTPRINT: Michael Scriven uses the word "footprint" to describe major, long-term impacts. The "footprint" metaphor allows evaluators to think of the program consequences like those of a footprint in the sands of time. Over time a program's influence could be examined by considering it's depth, length, breadth, numbers, direction and location.
- Depth--the importance of the effect on the individual
- Length--the duration of the effect
- Breadth--the number of individuals affected
- Number--the number of discrete efforts within the program
- Direction--an overall indicator of progress
- Location--if the program is located in an appropriate area
For more information on Footprint see: Scriven, Michael (1991) Evaluation Thesaurus. Newbury Park, CA: Sage.
- PROXIMAL AND DISTAL: These terms have emerged from the medical community with "proximal" referring to short-term consequences and "distal" to long-term consequences.
- OUTCOME INDICATOR: Outcome indicators are clear, specific and measurable characteristics that allow the evaluator to monitor if a desired change has occurred. Program outcomes are often complex and frequently are not easy to measure or observe. Consequently, evaluators then identify one or more indicators which seek to measure the outcome or impact. Sometimes there are a cluster of indicators which could be assigned different weights which can then be tabulated for a particular outcome. Other times, an indicator might be a single item, such as the presence or absence of a particular behavior, action, or activity. Outcome indicators work best when they are allowed to be modified, fine-tuned, or changed. Here are some examples:
|
Outcome indicators
|
Outcome
|
|
Performance on standardized tests
|
School achievement
|
|
Number of Arrests
|
Community safety
|
|
Class attendance
|
Learning
|
|
Number of tasks delegated
|
Leadership
|
Important concepts to remember relating to definitions:
- Outcomes" and "Impacts" are different from program "Outputs".
- When using the terms "Impact" or "Outcome" consider offering examples to clarify the consequences.
- Don't be surprised when elected officials, policy analysts, decision-makers, and others use the terms in different ways. The language of evaluation is constantly evolving.
How to Evaluate in an Outcome Mode
- Develop a theory of action
- Depict a causal chain of events that begins with the status quo and ends with your intended outcome
- Listen to others and revise theory of action
- Listen for other definitions of outcome
- Discuss choices of outcome with sponsors, colleagues, partners and customers
- Determine which outcomes are of primary importance
- Share the theory of action with others
- Revise theory and expected outcomes as needed
- Determine measurement points (What to measure?)
- Use the theory of action as your blueprint and determine measurement points. Consider measuring proxy factors or outcome indicators.
- Tip: Consider using levels of evidence
- Tip: Consider setting goals for each measurement point
- Develop your measurement strategy (How to measure?)
- Identify a logical, reasonable plan for measurement
- Consider which methods and instruments to use
- Consider involving sponsors, colleagues, partners and customers
Successful Outcome Evaluation
- Listens to different people about outcomes
- Involves decision-makers in the program---and keeps them up-to-date
- Uses a team evaluation effort to improve credibility and to ensure attention to relevant factors
- Strives for credibility and believability
- credible study design
- accepted protocol
- involves others
- uses independent evaluators
- considers reputation of evaluation team
- uses multiple methods
- uses reporting as a strategy not as a product
Outcome Evaluation Design Strategies
Baseline Strategy
- Set a baseline at a point in time and measure again in the future. Find the difference between the two time periods. Present the case for how and why the program is responsible for the difference.
Comparison or Control Group Strategy
- Find a comparison or control group. If possible randomize participants.
- Measure before and after the program.
- Show how experimental group differed from control group.
Reflective Strategy
- Ask participants and others to reflect back to a baseline level.
- Use open-ended questions and ask what is different or what has
changed. What's changed in the community? What caused the change?
- Use closed-ended questions with a scale. For example, use a
10 point scale (1=low and 10=high) and ask participants to rate
the community or situation now versus at a point of time in the
past (one year ago). If change occurred, ask what caused the change.
Descriptive Strategy
- Describe the outcomes in a narrative manner from the perspectives of the customers and the providers. Use stories or mini-cases.
Assessment Strategy
- Experts review indicators of outcomes.
- Community observers monitor progress toward outcomes.
For additional information
If you wish to read more about outcomes, consider the following:
Measuring Program Outcomes: A Practical Approach (1996) Alexandria, VA: United Way of America.
Osborne, David & Gaebler, Ted. (1993) Reinventing Government. Reading, MA: Addison Wesley.
Back to EVALUATION
|
