EVALUATOR AS A STORYTELLER
Suggestions for Evaluators
by Richard A. Krueger
Using Stories for Evaluation
Researchers and evaluators tend to be skeptical about stories. The word "story" is vague and can include many different things from case studies to anecdotes. Generally, evaluators are concerned about:
- how stories are collected,
- what claims are made about the stories and
- how those stories are used.
The evaluator sees the story differently
To a researcher / evaluator the individual story is a fragment of data. A story gives us a glimpse of something at one point in time from one point of view-like one frame of a motion picture. Often the evaluator wants to present the larger picture and multiple perspectives. For example, stories might be used to:
- Depict how people experience a program
- Understand program outcomes
- Get insights into program processes
Sometimes stories might be used alone and other times they are used to supplement other forms of inquiry.
The evaluator uses the story differently
The evaluator / researcher uses the story differently. It is important to know the background or circumstances of the story. This background give the evaluator clues about whether the story is typical or extreme, if the story is authentic, if the story has been changed over time and other factors relating to the story. Think about how stories fit in the evaluation design
The evaluator begins with a plan
The plan guides the inquiry and sets the boundaries for the stories that are sought. This plan, however, is open to serendipitous events or insights.
The evaluator is systematic
Sources are documented. Notes are kept. Care and consistency are needed in data handling. Systematic steps and procedures are used.
The evaluator seeks a verifiable trail
The evaluators keep track of the stream of evidence. Verification is sought from the original source(s)
The evaluator is concerned about confidentiality
Stories will differ in the need for confidentiality and the evaluator seeks to protect the storyteller from direct and indirect harm.
The evaluator uses systematic analysis protocol
The evaluator uses recommended and acceptable methods of analysis, documents the analysis process, worries about limitations, and reports results fairly.
The evaluator usually uses multiple methods
Stories can be used as a primary source or as a secondary or supplemental source of evidence.
Primary = stories are the major source of data
Secondary = stories used to illustrate points derived from other sources
The evaluator prepares a report that fairly reports the results, the process and presents useful insights to those receiving the report
This report gives readers insights about the process and the validity of the study.
What's needed for stories to be given credibility by evaluators?
- Application of Program Evaluation Standards*
- Unbiased inquiry
- Defensible protocol
- Systematic processes
- Verifiable procedures
- Authenticity-- Is it truthful or is there "truth" in the story?
- Representation-- What does the story represent? Is it representative of other people or locations?
- Original intent-Why was the story originally told?
* see the The Program Evaluation Standards. (1994) 2nd edition. By the Joint Committee on Standards for Educational Evaluation, Thousand Oaks, CA: Sage.
Examples of Story Topics Useful in Program Evaluation
- Organizational
- The importance of teamwork
- Going above and beyond what was expected
- A major change and how we handled it
- Who knows the ropes around here?
- Sometimes it's OK to break the rules
- Taking care of customers
- Showing initiative and taking risks
- Programs
- Something happened that was wonderful
- My most successful experience with the program
- The best thing about the program was . . .
- Learning and change
- Learning from mistakes
- Something I learned when I've failed
- I learned something that changed my life
- How I changed the foods I ate
- The biggest change I've ever made was . . .
- Education
- Great teachers I've had
- Something a teacher did that meant a lot to me
- The most important thing I've ever learned was. . .
- The best class I've ever had was . . .
- Customer Satisfaction
- Inconveniences I've had with
- A time when I was delighted with the help I received
- At first I was disappointed, but then they fixed it
Back to STORYTELLING
|
