Richard A. Krueger

ABOUT US

FOCUS GROUP
INTERVIEWING

EVALUATION

STORY-TELLING

LINKS

HOME

Occasionally as an evaluator you will need generic instruments or easy strategies that can be implemented quickly. Caution is needed with these procedures because the instruments may not be sufficiently sensitive to detect what you want to measure.

This section includes:

1. Documenting Participation and Involvement
2. End of Meeting Reaction-Closed-ended Questions
3. End of Meeting Reaction-Open-ended Questions
4. Goal Setting with Feedback
5. Writing a Note
6. Inviting Experts to Assess
7. Reflective Appraisal of Programs
8. Semantic Differential

1. Documenting Participation and Involvement

How many people actually attended? This number is often based on registrations completed at the event or counts of participants.

You may want to also answer these additional questions:

• How many were expected?
• How many could have potentially attended? Or, how many are in the population?
• What kind of people attended? How do the demographics of those attending compare to the target audience?

What was the extent of their involvement? Participants may have participated in different ways.

• How long was the event, workshop or program?
  The length of each session?
  The number of sessions?
• What was the amount of time that participants were expected to attend or be involved? Is there a recommended level of involvement?
• How much time did they actually spend at the meeting, workshop, activity, etc?
• How many people (what percentage) completed the recommended level of instruction?

Back to Easy Evaluation Strategies List

2. End-of-Meeting Reaction -- Closed-ended Questions

End-of-meeting surveys are probably the most commonly used evaluation instrument. Typically, this end-of-meeting form is completed during the final moments of a class or event. The form is designed so that it can be answered within a few minutes and consists of a limited number of questions.

Question categories:

There are four major categories of questions that can be asked in the end-of-meeting survey. Not all categories are asked. Select those of greatest importance.

1. Reactions of the Participants
• To the teaching (How well was it presented? Was it clear? Did instructor know the subject?)
• To the program (Was it useful, helpful, practical, interesting, etc.)
• To the context and environment (How was the parking, meeting space, meals, lodging, etc.)
2. Immediate Perception of Results
• How much did you gain on each objective?
• What were the key ideas that you retained?
• What will you do with what you received?
3. Information for Future Programs
• What should be topics, location, time and length of future sessions?
4. Background Information
• What is your age, education, gender, place of residence, occupation-job title, number of workshops attended in past, extent of involvement in workshop, etc.?

Closed-ended responses

Most questions are constructed to use closed-ended responses. Here are some of the popular examples in use:

Or use a five-point scale which is anchored at each end (1=Lowest and 5=Highest).

Back to Easy Evaluation Strategies List

3. End-of-Meeting Reaction -- Open-ended Questions

The end-of-meeting survey can also be developed to use open-ended questions. This open-ended form consists of a handful of questions and space for the respondent to write in their answers. These forms have potential for giving you more detail but they also require time and effort from the participant. Some participants may be unable or reluctant to write. Limit the questions so they can be answered in a few minutes. Analysis is more time consuming than a survey with closed-ended questions.

Here are some open-ended questions that you might consider:

• Describe your thoughts and feelings at the end of the workshop.
• Describe one thing the speakers did that you liked best.
• Describe one thing the speakers did that you liked least.
• What words would you use to describe the workshop?
• If you had to describe the workshop to your supervisor in one sentence, what would you say?
• What might be done to improve the workshop?

Back to Easy Evaluation Strategies List

4. Goal Setting with Feedback

Goal setting with feedback is helpful in those situations where you do not want to suggest specific action steps but instead want the participants to make their own decisions on the future applications. In these situations the instructor offers background materials or research results and then the participants decide whether or not it is applied or used.

Set aside some time at the end of the session. Ask program participants to set personal goals regarding the future use of the materials presented. These goals should be specific, measurable and in writing. Give participants some time to reflect. It is often helpful to remind them that these are goals that they will accomplish in the near future, such as within the next month or so.

Request a limited number of goals. Often two or three are adequate.

Ask the participants to sign their list of goals and turn the list over to a designed individual. Later the participants will be contacted to determine the extent to which they have accomplished their goals.

The primary advantage of goal setting with feedback is that it reinforces the importance of the participants' personal goals as opposed to the goals of the presenter. Furthermore, participants are often more likely to make changes in their behavior if they have set personal goals.

The success of this strategy depends on the skill with which it is presented. It should be presented as a helpful strategy for participants to make progress on areas they personally consider to be important.

The advantage of this strategy is that it allows participants to struggle with practical application of what was presented.

Back to Easy Evaluation Strategies List

5. Writing a Note

At the conclusion of the meeting, class or event the participants might be asked to write a brief note about the experience. This format allows the respondent to comment on any or all aspects of importance. Analysis is based on clustering responses into categories.

Writing a Note to the Leader of the Organization
Ask the respondents to write a short note regarding the event that will be shared with an influential person in the organization. The request is completely open-ended and respondents can write about anything they want.

Writing a Note to the Teacher
Ask the participants to write a short note to the teacher or presenter about the experience. They might be asked to comment on areas of concern to the teacher, such as: What might be improved? What should be done differently? What did you particularly like?

Back to Easy Evaluation Strategies List

6. Inviting Experts to Assess

Invite experts to observe the event, meeting or activity and then prepare a report or share their findings. Experts might use interviews and observations combined with expert savvy in assessing the event. It is often best to make use of several experts who might work individually or as an evaluation team. The experts might be allowed to use whatever instruments or procedures they deem appropriate.

Back to Easy Evaluation Strategies List

7. Reflective Appraisal of Programs or RAP

Reflective Appraisal of Programs is a style of asking questions and presenting response categories that is based on the Levels of Evidence. (The seven levels of evidence include: inputs, program activities, people involvement, reactions, KASA change, practice change, end results.) The RAP strategy uses standardized interview questions that can be adapted to a variety of situations. It relies on perceptions of the program being studied. Program participants estimate or reflect upon the extent to which a program brought about change. It is useful for a wide variety of programs. The questions build on the Levels of Evidence. For example:

"To what extent did you participate in __________?" (People Involvement)
"To what extent did _________ meet your expectations at the time?" (Reactions)
"Think back to the activities in which you were involved. To what extent did you learn more about _________?" (Knowledge Change)
"To what extent did you become more interested in _________?" (Attitude Change)
"To what extent did you acquire more skill in ______?" (Skill Change)
"To what extent did you become more determined to try ____?" (Aspiration Change)
"To what extent have you put to use the ideas or skills you learned regarding _____?" (Practice Change)
"To what extent were the ideas and skills you acquired helpful to you?" (End Results)

• Response Categories: RAP uses standard response categories, including:
  ( ) to a great extent
  ( ) to a moderate extent
  ( ) to a slight extent
  ( ) not at all
• Size: If the program had 40 or fewer participants include everyone. If more than 40 then select sample randomly and obtain a 75% response rate via mail or telephone.
• Selective Probes: Respondents are invited to provide more information through the use of probes. For example, if the respondent indicated "great" or "moderate" to a question on practice change then the interviewer might ask for an example. Probes could include: "Would you please explain?" or "Would you give me an example of what you mean?"

Back to Easy Evaluation Strategies List

8. Semantic Differential

The Semantic Differential scale can be helpful in evaluating an organization, a program or a service. The scale consists of opposite adjectives and the respondent is asked to check one of the seven boxes that is closest to the word that best describes his or her perception. Here's an example.

XYZ Organization
Listed below are words that might describe the XYZ Organization. Note that the words on the same line have opposite meanings. Place a checkmark in one of the seven spaces that represents how you feel about the XYZ Organization.

Analysis
Later you can assign a number value (1-7) to each column and tabulate the mean score for each set of opposite words.

Several variations are possible.
First, you might want to change the words or add additional rows. Attempt to select opposite words that have clear meaning. Sometimes this is more difficult than it seems because of the ambiguity of language.

Second, you might want to have a dual scale where the respondents are asked to give a rating at two different points in time. For example, you might ask for a rating of how the organization was five years ago versus today. Or, you might ask for a rating for how the organization "IS NOW" and also a second rating for how it "SHOULD BE". In these situations, the gap or difference score between "is now" and "should be" or between different years provides an interesting comparison. When constructing these dual scales, it is often best to place the scales side-by-side and print the page on landscape mode.

Back to Easy Evaluation Strategies List

Back to EVALUATION