Making Sense of Formative Evaluation: Remember the Fundamentals
Formative evaluation is a common topic of conversation among grant writers and nonprofit leaders. I think it’s because evaluation can be challenging to do right. You can get a Ph.D. in evaluation, so you know it is not as simple as writing some goals and objectives. There’s more to it than that. So what do we do if we aren’t experts?
Just like in the basketball, it’s all about the fundamentals. Let’s focus on one fundamental for now: formative evaluation. What is it, and how do we use it effectively?
Elements of Formative Evaluation
- Ensure appropriate and timely resource allocation to meet your mission
- Validate that the program’s implementation methods will provide meaningful results for recipients
A National Example of Formative Evaluation
The Center for Children and Technology, funded by the U.S. Department of Education, provides an excellent review of its extensive formative evaluation activities conducted over four years. Their work included a pilot study of their newly designed intervention. They used the results of the pilot study to implement a scaled-up version of the program nationally. They provide a formal, published report after each step in their comprehensive evaluation plan. You may find their examples useful as you consider how you will incorporate formative evaluation in your work.
A Competitive Edge
Like the Center for Children and Technology, you can use formative evaluation to communicate your nonprofit’s successes and innovations while improving your credibility. You will be more competitive in grant seeking and fundraising by implementing formative evaluation for three reasons.
- Few nonprofits use formative evaluation , because most grant makers only require summative evaluation (or outcomes evaluation). You can say you use formative evaluation as well as summative (or outcomes) evaluation. These are the core elements of a comprehensive evaluation plan. You improve your nonprofit’s credibility in evaluation by discussing the results of your formative evaluation in your proposals.
- You need evidence to substantiate your chosen solution . Using formative evaluation, your program will be based on the best evidence in the field and will be tested for appropriateness in your community, with your consumers. Reviewers want to see that you’ve done your homework (i.e. evidence), and formative evaluation is one way you can show you have thought about and planned extensively for their investment.
- Funders and donors want to invest in success . Without formative evaluation, how can you assure your current and potential supporters that your program is a good investment? Even if you’re using a best practice program, how do you know that it will work with high fidelity in your community? Best practice programs don’t always replicate in new communities as planned, so pilot testing is vital.
Formative evaluation is effective, simple and cost-effective to implement, and it saves time and money over the long-term. Position your nonprofit for greater fundraising and programmatic success by building formative evaluation into your work from the start.
Practical Application of Formative Evaluation: A Case Study
To make it easier to break down this information, let’s use a nonprofit case study for our discussion.
Program Purpose: A local veterans’ services organization provides support, programming, and resources to military families. Their goal is to provide an access point for service members and their families to find the tools, resources, and assistance they need to build and maintain strong families while the service member is deployed.
Program Activities: Through monthly meetings, local military families meet others who understand the unique pressures they face and form peer-to-peer support networks. Monthly meetings feature guest expert speakers who provide information on resources available to military families in the community, as well as providing social activities such as movies and games for families.
In this example, formative evaluation means we need to answer a set of core, sequential questions during pilot testing. Here are some examples the veterans’ services organization could use.
- Did staff implement all monthly meeting activities as planned? Why or why not?
- What problems did the organizers encounter when implementing the monthly meetings? How were these problems resolved?
- Were any new objectives added? If so, why?
- Were there any changes in program personnel or volunteers? If so, why were changes made and what were the results?
- What actual costs did the program incur (by line item)? Were they greater or less than budgeted costs?
- Did staff learn any lessons that might be useful to other organizations that serve military families?
Next, our veterans’ organization should collect data—both before and during pilot testing—using two or more of the following.
- Focus Groups will include homogenous groups of service members or their family members (not combined) gathered to focus on a specific topic/set of questions with a trained facilitator. Participants will be encouraged to talk openly about their opinions and interact with other members
- Observations will include video recordings of initial monthly meeting (during pilot test phase).
- Key informant interviews conducted by a trained interviewer asks probing questions of a few recipients or staff members using a set of defined questions.
- Expert analysis and review provided by subject matter experts from other established programs that serve military families and service members.
Once you have gathered your formative evaluation data, analyze it! Then in a timely manner, modify your program to improve effectiveness. Once you know your methodology will get the results you want, you can move on to full-scale implementation.