Richard A. Krueger

ABOUT US

FOCUS GROUP
INTERVIEWING

EVALUATION

STORY-TELLING

LINKS

HOME

1. Introduction

   A while back I was asked to speak at the orientation session for new extension educators. The staff development leader explained that I was invited to talk because I might add variety to the training program. She said that she had observed my behavior and that of other evaluators at staff meetings and found that evaluators often "thought differently" about programs. She didn't explain how we thought differently, but I took it as a compliment and agreed to share some thoughts about how evaluators think.

   Here are my notes from the presentation. I had intended it to be conversational with a touch of humor-sort of tongue in cheek. But nobody laughed. Indeed, they seemed to take it too seriously. There is a little humor here and I hope you can find it.

2. A thinking process used by evaluators
   • Reflecting
   • Develop a theory of action. A logical sequence that results in change.
   • Begin with what is supposed to happen--the results.
   • Use "backwards planning" (like reverse engineering).
   • Depict the causal chain of events that begins with the status quo and ends with your intended outcome.
   • Listening
   • Share the theory of action with others.
   • Listen to others and revise your theory of action.
   • Listen for other definitions of outcome.
   • Discuss choices of outcome with sponsors, colleagues, partners and customers.
   • Determine which outcomes are of primary importance.
   • Revise theory and expected outcomes as needed.
   • Measuring
   • Determine your measurement strategies--how you're going to look at the program
   • Use the theory of action as your blueprint and determine measurement points. Consider measuring proxy factors or outcome indicators.
   • Adding value to the program
   • What can evaluation do to contribute to the program? How can evaluation make the program better, more enjoyable, focused on results, accountable, satisfying to participants and educators, etc? Redesign the evaluation so that it yields needed information but also adds value.
   • Behaviors that are helpful to evaluators
   • Communicate clearly
   • Tell meaningful stories
   • Share your toys
   • Speak highly of others

    

3. Alternative ways that evaluators think

Evaluators differ because of training, experience and personalities, but here are some of the "mental models" that depict their approaches to program evaluation:

Goal Based Thinkers - "We look for goals"
1. What are the stated goals?
2. What goals are measurable?
3. Measure those that can be measured (ignore or just observer others).
4. Consider unintended consequences--both positive and negative.
5. Compare what happened to what was promised.

 

Audit Thinkers -"We investigate and find out what's wrong"
1. What was supposed to happen in the program?
2. What actually occurred?
3. Identify discrepancies and non-compliance.

 

Utilization Focused Thinkers - "We make evaluation useful"
1. What questions are decision-makers asking?
2. What questions are they not asking but are important?
3. What information will be helpful to those making decisions?
4. Gather that information.

 

Empowerment Focused Thinkers - "We empower local people"
1. What is important to local people?
2. What information do these local people want to collect?
3. Work with local people in providing technical support to help gather, analyze and report what is important.

 

Positivistic Thinkers - "We are scientists"
1. Is there a baseline?
2. Can I find a control group?
3. Measure what I can and compare it to baseline or control.
4. Do statistical tests and determine if differences are significant.

 

Number Thinkers - "We count--and we do it well"
Numbers are "hard" data. Our goal is to find the right numbers.
1. What numbers best reflect what the program is doing?
2. Collect numbers.
3. Analyze using systems approach (input-output, cost-benefit, etc.)

 

Qualitative Thinkers - "We tell stories"
1. How do I know what is important or meaningful to the target audience.
2. Listen to people, observe what is happening.
3. Capture their stories.
4. Determine patterns and trends that explain reality.

 

4. Practical tips for successful evaluation

Here are some practical steps that have helped evaluators.

1. Involve others

Utilization, impact and believability emerge from involving colleagues and clientele. If you want the information used then involve others!

2. Ask yourself: Do I have a program? and, Is it worthy of evaluation?

A program is a logical, sequential set of events or activities that accomplish an agreed upon objective or goal. Sometimes we merely have a string of activities without a goal, or goals without logical events. Evaluation takes time and energy and some programs are just not worth the investment.

3. Consider your purpose for evaluating----
Some possible purposes include:
• Demonstrate accountability
• Identify consequences of program
• Leverage for other programs
• Improve future programs
• Provide feedback to people--professionals & clientele

 

4. Consider who wants the evaluation-Who requested it?

What does this audience find to be credible and believable?

5. Use a variety of evaluation methods when possible.

Multiple methods or triangulation add credibility and ensure that the measurement is on target.

6. Keep costs low by:

Sampling strategically
Keeping samples small
Pilot testing instruments

7. Keep interest high by adding payoff to the participant.
8. Start with goals, but don't be unduly limited by goals
9. Consider "early evaluation"
1. examine the logic of the program
2. determine sources of data
10. Design the evaluation carefully. The evaluation should:
• Enhance the program
• Yield information beneficial to stakeholders
• Conserve resources

 

5. The Logic of Measurement Strategies for Outcome Evaluation

One of the most challenging tasks for the evaluator is to prove that an outcome resulted from a particular program. Different strategies can be used and the challenge is to select the strategy that is both credible and practical. Here are some of the ways that evaluators use when they attempt this task.

1. Baseline Strategy

Set a baseline at a point in time and measure again in the future. Find the difference between the two time periods. Present the case for how and why the program is responsible for the difference.

2. Comparison or Control Group Strategy

Find a comparison or control group.
Measure before and after the program.

3. Reflective Strategy

Use reflective strategies whereby customers and others reflect back to a baseline level.

Use open?ended questions and ask what is different or what has changed.

Use closed?ended questions with a scale.
• Open-ended:
• what's changed in the community?
• What caused the change?

or

• How is the community different today than it was one year ago?
• What caused the change?

 

4. Descriptive Strategy

Describe the outcomes in a narrative manner from the perspectives of the customers and the providers.

5. Assessment Strategy

Experts review indicators of outcomes
Community observers monitor progress toward outcomes

Back to EVALUATION