How to Use Observation in Evaluation
by Richard A. Krueger
- Begin with a framework, a theory or an idea of what you are observing
Develop your observational strategy--develop a plan
- Something is driving your study. You suspect that something is happening but you don't have evidence. Describe it in writing. This will clarify your thinking and allow others to offer feedback.
- Too much is going on to observe everything. You must focus your attention on a limited number of areas, but stay open to new developments.
In your plan you should briefly describe: Who will observe, how will observers be prepared, what is the location, timing and duration of observation, how the study is explained to the subjects and how the subjects will be selected?
Consider an observation checklist
- Consider if a checklist would be helpful. Often checklists are useful when observing specific, unambiguous behaviors. The checklist usually has a rating scale that asks the observer to indicate presence or absence, frequency, intensity, speed, etc. As the behavior gets more complex checklists are less helpful.
- In some situations you may be attempting to identify patterns, connections, sequences, etc. In these cases, use field notes to document what you observe. This latter case might involve observing sequence of events, how something is done, dexterity, skill, concentration, distractions, etc.
A research strategy that has often been helpful is to make comparisons. Sometimes important findings don't emerge until you compare different locations or situations. Consider comparing:
Take detailed field notes
- from one location to another
- from one time period to another
- from one group (age, class, gender, etc.) to another
Always take field notes, even when using a checklist. Record the date, time, situations and describe what you've observed. Sometimes you can take notes while you are observing, but in other situations the note taking might disturb the subjects. In these cases delay your note taking and then later as soon as possible record your notes in detail.
Issues that often surface in observational studies
- Do participants act differently because they are being observed?
- How unobtrusive should the researcher be?
- What do you tell participants when they ask what you are doing?
Suggestions for quality observations
- Practice observing with others on the research team to be sure that the same things are being observed and recorded in a consistent manner.
- Put full attention on what you are observing.
- Do not draw attention to yourself.
- Write detailed field notes. Describe what you see and also note the hunches or ideas that come to you while you're observing.
- When in doubt, invite other members of the team to observe and offer suggestions.
- Analysis and reporting begin with a brief statement of the problem that gave rise to the study. Write a paragraph describing how the study was conducted. In this paragraph describe the situation, environment, timing of observation, research team, subjects, etc.
- Analysis can be in several forms. Aggregate the results if you are using a checklist. Look over your field notes. Look for patterns, themes, or overarching concepts. Draw sketches of concepts or relationships. Draw a picture of force fields or flow charts or time lines, etc. to illustrate the concepts.
- When reporting results use bullets to highlight key points. If possible, compare and contrast findings across sites, categories, etc.
Patton, M. Q. (1987). How to use qualitative methods in evaluation. Newbury Park, CA: Sage.
Patton, M. Q. (1990). Qualitative evaluation and research methods (2nd ed.). Newbury Park, CA: Sage.
Taylor, S.J. & Bogdan, R. (1998) Introduction to qualitative research methods (3rd ed) New York: Wiley.
University of Wisconsin Extension Collecting Evaluation Data: Direct Observation http://www.uwex.edu/ces/pdande/Evaluation/evaluat.html
Back to EVALUATION