Six Steps to Effective Program Evaluation: Step 4 - Review Related Research
The Shelf. Also known as the final resting place or “the graveyard” for many articles, reports, and important documents. Some of these items have been used to help shape a program once or twice, if they’re lucky. Many, having been abandoned without anybody ever looking at or learning from them, are lost in the great abyss of useful but untapped knowledge.
Whatever problem you want to solve, there is a vast treasure trove of information from related research that can help. This article is to help you get those ideas off the shelf and make them useful.
Getting Valuable Information “Off the Shelves”
Given the many benefits, you might wonder why people don’t always check what past studies have found before testing a new solution with people. This approach provides at least three benefits:
Strengthening Your Program
Reviewing the experiences of past related projects lets you find “best practices” and effective strategies to consider. It can also point out ways to avoid mistakes made by previous programs that ended up being unhelpful (or even harmful).
Planning a Useful Evaluation
Looking at what others have found shows what’s well supported by research and practice. This lets you avoid retesting what others have proved. Plus, past studies can show methods, measures, and methodological considerations that might benefit your program evaluation.
Making Your Case
Telling people about the existing evidence to support your work helps increase buy-in among stakeholders. It also helps you show funders how your plan is worth the investment.
Five Places to Find Valuable Related Research
Researchers use many venues to publish their results, such as blog posts, fact sheets, and webinars, as well as scholarly papers. We can increase our understanding and improve our ability to solve problems faster by sleuthing as much of the relevant evidence as possible.
Finding related research to inform program planning and evaluation is similar to finding any information. Your search should be as rigorous and complete as possible—depending on the timeline, focus, and scope of your project. If you need fast-turnaround information to present at the board meeting tomorrow, you may only have time for a quick search to identify a few readily accessible articles. When making very high-stakes decisions, a more rigorous and comprehensive review is more appropriate.
Below are five good places to look for related research and materials.
1. Your Own Bookshelf
For some questions, you (or others at your organization) may already know of some sources. These may include related professional magazine articles, conference presentations, books, and other sources you’ve come across. Don’t overlook what might be right under your nose.
Tip: Ask your project team members, staff, and colleagues with knowledge of the topic what sources they think may be helpful. As a bonus, post a question to email lists/online discussion groups related to the topic asking group members what sources they recommend.
2. The Internet
A fast way to get an idea of the related research is to conduct a general internet search, using Google or other search engine.
To develop search terms, we identify the concepts in our question and come up with terms related to each concept. For example, a story I read in the news recently was about protests in Milwaukee following the shooting of a black man by police. It brought attention to concerns about racial disparities and race relations in the city. This reminded me of a research interest of mine, interracial dialogue groups, in which small groups discuss and take action to address race relations and racial equity in their community. Imagine you are involved with a community organization in Milwaukee and your board wants to know what effects interracial dialogue groups had on race relations and racial equity, and what makes them successful?
Start by looking at the concepts in our question. They include interracial, dialogue groups, and race relations/racial equity. We use those concepts to surf:
- Terms related to dialogue groups: dialogue group, dialogue groups
- Terms related to interracial, race relations/racial equity: interracial, race, racial, racism, ethnicity
- Terms related to community: community, communities, community-based
Another helpful trick is to search using various combinations of these terms. For example, a few example queries find many results:
- “interracial dialogue groups” OR “interracial dialogue group” (Google search yielded about 715 results)
- interracial dialogue group* community (yielded about 197,000 results)
- dialogue group* (interracial OR race OR racism OR racial) community (yielded about 881,000 results)
You should experiment with different combinations of search terms. Add to and delete terms as needed to narrow or broaden your search to find the most relevant information.
Tip: To search Google like a pro, check out Google’s “ Expert Search tips .”
3. Key Websites
Another great technique is to browse/search the websites of leading professional associations, government agencies, and other organizations that are active on your topic.
Tip: Oftentimes, an internet search will show potentially helpful websites to explore. One of the first sources I found in the above web searches was a U.S. Department of Justice “Community Dialogue Guide” that included a “Directory of Resource Organizations,” including web links. We could browse those organizations’ websites to look for additional useful resources.
4. Scholarly Literature Databases
Google Scholar is a handy tool to search for academic research articles. Searching Google Scholar is free to anyone with an internet connection. Many of the indexed papers require a subscription or cost to download, and many are available at no charge.
A couple of additional, free-to-search sites for finding scholarly publications are Academia.edu and ResearchGate. Some topic-specific scholarly literature databases, such as the ERIC collection of scholarship related to education, are also free to search.
Tip: To master Google Scholar search techniques, check out Google Scholar’s Search Tips.
5. Your Sources’ Sources
Once you’ve gathered a few helpful articles, reports, and other items, looking at what sources those sources mention can lead to more relevant materials.
Tip: As you review related publications, check to see what sources they mention in their discussion of background information, footnotes/endnotes, and references. Look up any sources that sound relevant to your project.
Take Ideas Off the Shelf
Use these five techniques to discover valuable information that could benefit your organization. The more relevant information you find, the more knowledge you gain, and the greater your chances of success! #Winning