Step 3 of Six Steps to Effective Program Evaluation: Ask Useful Questions
Computer pioneer Charles Babbage was asked if you could give a machine the wrong information and still get the right answers. He’s quoted as saying that he did not understand “the kind of confusion of ideas that could provoke such a question.” In program evaluation, as with computers, you need to ask the right questions to get the right answers.
No More Useless Nonprofit Program Evaluation Questions
Unlike a purely academic research study, an effective evaluation must focus on getting the information you can use to strengthen your program and benefit the people it serves. For each of your evaluation questions, consider why you are asking it and how you will use the answers.
It’s a hot summer day in Virginia as I’m writing this, so an example that comes to mind is a project to build a new public swimming pool in your community:
Demonstrating Your Program’s Value and Worth
Before you begin, an evaluation can help you show funders and stakeholders the need for a community swimming pool, the likely benefits, and the viability of your plan. Later, evaluation can provide the data you need to make the case for sustaining it and perhaps expanding it to bring swimming pools to more neighborhoods.
Planning Ways to Strengthen Your Program
An effective evaluation reveals promising ways to overcome obstacles and build on successes (e.g. letting more people in the neighborhood know about the pool) so your program can meet its full potential.
Building Effective Collaborations with Partnering Organizations
No program exists in isolation. To be effective and successful , the swimming pool must be connected with transportation (buses) and key businesses (chlorine supplier, ice cream stand). An evaluation can help increase buy-in by including those other stakeholders and asking for their thoughts on how to meet shared goals (e.g. how our organizations might help each other to more effectively serve our common clients).
Four Useful Evaluation Questions to Ask
If your program is supported by a grant award, you may be conducting a program evaluation to meet funder requirements. In that case, you’ll want to be sure to answer the questions that your funder wants to be answered. You can find the funder’s evaluation requirements in the grant information materials and conversations with the grantor.
You will likely also need to ask additional questions to get good and relevant information for starting a successful program or making your existing program more successful. Your understanding of how your program functions (also called your “model” or “knowledge map”) shows ideas and causal relationships that might be important to explore.
Four general questions that a program evaluation can answer are often important:
1. Did We Achieve the Results We Expected?
Your understanding of your program shows the results that people expect your activities will achieve. To get needed stakeholder support, you need to demonstrate whether your program is getting those results.
In our example of the community swimming pool, people may expect many effects from the pool’s many activities and features. To mention a few possible effects:
- Providing a safe and enjoyable experience for pool users
- People meeting and building social connections as a result of providing a space for community members to gather and holding events
- Increasing job opportunities in the community by creating new positions and training for lifeguards and other pool staff
- Improved health of community members from getting more exercise while swimming
When your innovation is pursuing a large number of desired results, you may not have the time and budget to measure every effect. So, focusing on the question of “Did we achieve expected results?” means identifying which of the many expected results to measure.
2. Why or Why Not?
For your evaluation to be useful for making decisions to strengthen your program, it needs to show not just whether results were achieved, but also why or why not.
Your understanding of your program explains why you thought results would be achieved. So, exploring the question of “Why of why not?” means looking at whether results happened for the reasons you expected, and what does explain results. If attendees say they were highly satisfied with your annual pool party, you’ll want to know why they were so satisfied—whether it was because of the friendliness of the other attendees, the games, the music, the food, or something else. So you can use that information to help you decide what to keep, what could be dropped, and what could be improved.
3. What Effects Did We Have?
To show your program’s true effects, in addition to looking at whether it’s achieving the effects that you think, you also need to uncover what effects it is having, including any unexpected effects. You may find that it has benefits that you hadn’t considered. Sometimes, you might discover that your program is having an unforeseen negative “side effect.” If so, that’s valuable information that you can use to make changes to avoid those negative effects.
4. What Would Strengthen Our Program?
Your understanding of your program may suggest questions you need to be answered to strengthen your program. For example, if you see that you need “marketing” of the pool to get people to use it, a useful question might be what kind of marketing is most effective to spread the word. This can help you decide whether to focus on posted signs, mailed announcements, social media, word-of-mouth, or something else. Investigating what would strengthen your program might also reveal new opportunities and ideas that you hadn’t considered.
Garbage In, Garbage Out
Remember, a program evaluation is like a computer—garbage in, garbage out . For you to get useful answers, you must input the information from useful questions.