Step 2 of Six Steps to Effective Program Evaluation: Understanding Your Program
In my last installment in this series, I talked about the first step to the effective program evaluation: policy, or your program functions. Many evaluations jump to collecting data and measuring outcomes based on the outcomes that the funder or the CEO want the program to show. Those outcomes are certainly important to measure.
However, for your evaluation to get you results that you can use to paint the “true picture” and make a meaningful difference in people’s lives, it also needs to include the outcomes that matter to all stakeholders. Because all stakeholders matter. This means taking the time, early on, to get stakeholders’ views of how your program works. In scientific terms, this is your program “theory.”
Besides making your evaluation more useful, showing stakeholders’ understandings of your program can also show ideas that you can use to strengthen your program. It also builds relationships and increases buy-in for your evaluation results.
Adjusting Goals, Stronger Program
One idea that nonprofit organizations can learn from the business world is that your strategies and goals should be fluid, not “written in stone.” While your core mission will always stay the same, you sometimes need to adjust your steps to get there.
At a panel discussion at the Hudson Institute on A Decade of Outcome-Oriented Philanthropy, I heard a good story. The speaker talked about a program in Kansas City that provided youth with extra-curricular activities. The program emphasized graduation from college as the goal. However, when program leaders talked with students, parents, teachers, and others, they discovered that graduation from college was not what everyone considered the best outcome. It was not what everyone in the community wanted. Based on this information, the leaders ended up expanding the program to support both college graduation and other routes to economic success.
In this way, before outcomes measurement began, they were able to use insights from the stakeholder interviews to shape the program to add new a new way to help more people. By the way, short and friendly interviews are also a great way to build connections between people and organizations.
Below is a basic stakeholder interview guide that you can use to ask stakeholders about their understandings of your program, as part of planning your program evaluation.
Five-Question Interview Guide
You can use this Five-Question Stakeholder Interview Guide to ask your stakeholders about their understandings of your program. So you can use that information to focus your evaluation, and your program, on what’s relevant and useful.
You may want to modify or add more questions to this basic guide for your specific situation.
You can conduct the stakeholder conversations one-on-one or in focus groups (e.g. groups of community members, groups of staff). Or, you could do a combination of both individual interviews and focus groups.
Before You Start
First, introduce yourself and confirm that it’s still a good time to talk.
Explain the purpose of the interviews and the evaluation and the plans for using the results. Ask if they have any questions.
Ensure confidentiality, or ask if you may have permission to use the person’s name, as appropriate.
Ask permission to record the interview (if recording), so that you don’t miss anything that they say.
Five Questions to Ask
Note: In the brackets below, replace “organization” and “program” with the name of your organization and the program that you are evaluating.
- What is your role in [organization] and [program]?
- What do you see as the organization’s goals for [program]? That is, what do you think the organization leaders want it to accomplish?
- What are your personal goals for [program]? In other words, what do you personally hope it will accomplish? That is, what would you consider success? Any other effects that you expect the program is likely to have, either positive or negative?
- In your view, how will [program] accomplish these goals? That is, what specifically will it do, or could it do, take to make those things happen?
- Anything else that you’d like to add that we haven’t talked about?
Thank the person for participating in the interview. Let them know how you plan to share results from the evaluation.
Ask Before You “Leap”
To make your evaluation more useful for strengthening your program, remember to take the important step of asking stakeholders their views before jumping to outcomes measurement. An easy approach is to use the interview guide provided in this article. With that information, you can better show your program’s ability to meet stakeholder needs and get some stellar ideas!