Unpacking the Elements of Effective Program Evaluation Systems

Why is effective program evaluation so important?

Designing and rolling out a new grant or loan program can be an extremely time intensive process. From ensuring that the program will meet the needs of its target audience and funders while staying true to the values of the administering entity to testing the application portal and procurement procedures, there are tons of things to keep track of at the launch of the program. Establishing effective program evaluation systems and expectations should be at the top of this list. We hope that this article can serve as a guide to the best practices for identifying and gathering meaningful types of information to support consistent program evaluation. It is also a friendly reminder of the importance of program evaluation, which will be a critical consideration as you begin to plan for and implement projects or programs funded through the American Rescue Plan (ARP).

What are indicators anyway?

So, you’ve developed your project or program and established a logic model, you might be asking yourself “what’s next?” This is the time to pinpoint which indicators you feel will set up your program for success.

Indicators are kind of like the independent variable in a science experiment: they are the measurable and observable characteristics that demonstrate change or progress towards achieving specific results. They are calculated based on a set of metrics, which we’ll get into in a minute.

Looking at your goals and objectives, what insights are going to indicate that your program is excelling, declining, or at a standstill? Which areas do you want to see transform? Is it the behaviors of program participants, reactions towards a specific topic, resource awareness? Do you have to prioritize various points of interest to comply with your program’s funders? Defining the change you want to see at the outset of your project will provide clarity as you develop your program and will allow for more consistent and effective data collection to support your evaluation.

Why should you pay attention to metrics?

These are specific data points that are tracked periodically–e.g., weekly, monthly, quarterly, annually, etc.–to measure your indicators. Metrics support indicators and should be developed with your specific indicators in mind. Using the perfect set of metrics will help to better craft the story of your program and increase the effectiveness of your program evaluation efforts.

Sometimes collecting and analyzing metrics come with challenges like using inconsistent methods to collect your data, not collecting data at regular intervals, or losing data due to improper maintenance. As with identifying indicators, it is important to develop metric collection protocols alongside program development, as this will ensure effective collection and maintenance systems as well as consistent data.

Why are surveys so popular?

Surveying is a powerful research tool. When developing a survey, think about the questions you are asking and how they are being framed. Survey questions should be unbiased and straightforward to ensure the data attained is a genuine reflection of the respondents’ opinions. You can learn more about survey design here.

Surveys can be used internally with your own organization to collect staff insights or externally to reach a broader audience within a targeted area. This results in data sets that can be analyzed to understand different perspectives and inform future improvements, two key elements of effective program evaluation.

While surveys are great for efficiently collecting feedback on a large scale, keep in mind:

• Confidentiality: Sending personal information out into the world can be daunting. With surveys, minimize unnecessary requests for sensitive information. Also, clearly state how the information collected will be used and/or shared.
• Keep it timely: Survey fatigue is real and requesting feedback too often can be met with disinterest. Minimize requests for any given project and think strategically about timing the release a new survey.
• Tell your story: Build a strong narrative around the need for feedback to explain the importance of providing feedback and build trust between you and your stakeholders.
• The 3 S’s–Short, Sweet, Simple: Be concise! Make questions clear and straightforward and limit open-ended questions.

A few examples of common easy-to-use tools to conduct surveys are Google Forms, Survey Monkey, and Mentimeter, which is a fun option for live polling.

How can you best tell your success stories?

Often, it is easy to get caught up in the numbers, but qualitative data holds its own value as well.  While quantitative data helps demonstrate changes and trends, narratives and anecdotes ground numbers and provide nuance that is otherwise difficult to convey. The graphic below illustrates success through two scenarios in which quantitative and qualitative data are paired.

Qualitative and quantitative data can be mixed and matched to meet your needs. For example, quotes, imagery, case studies and video clips can be accompanied by supporting program statistics. These are great ways to increase your program’s profile in a meeting with potential participants, among funders, or online with a general audience.

What are the benefits of leveraging successes & lessons learned?

Leveraging program and project successes and lessons learned is a great way to advance program sustainability. Exploring strengths and weaknesses can help you identify areas of need to refine program goals or service delivery models. Impactful and meaningful insights that come from evaluating your program will enable you to articulate impact, scale approaches, and replicate models that can:

• Attract funders;
• Cultivate new partnerships or broaden existing ones;
• Draw conclusions to inform the development of new initiatives;
• Secure buy-in (from stakeholders or for program or project continuity).

How Public Works Partners can help you establish effective program evaluation systems.

Public Works Partners’ Program Evaluation and Performance Improvement service areas provide clients with the tools needed to demonstrate their success to funders, prioritize resources and improve programs, and manage accountability to successfully meet program and project milestones. This enables our clients to highlight program success to their target audiences in a way that is impactful. We also help our clients share success stories and lessons learned with wide audiences, positioning their insights as a resource to experts, nonprofits, and governments who seek to address similar needs within their communities, towns, or cities. As funding from the ARP helps spur the economy, drives innovative solutions, and unlocks new opportunities, evaluating the efficacy of programs and projects can inform emergency response and recovery plans for inevitable future disasters.