Evaluation

Why Keep Score?

There are many reasons why measuring progress is vital to the success of an entrepreneurship development effort. Keeping score or measuring results is probably the very best recruiting tool or motivational approach possible. Results get you more volunteers, more resources, more positive visibility. Measuring progress is also a way of building accountability into any project, whether the focus is on supporting entrepreneurs or on any type of community improvement. Keeping score will help you in a number of ways:

Use progress reports to keep your volunteers energized and committed.
Hold people, associations and organizations accountable for the work.
Improve your effort by documenting successes and challenges.
Measuring the results or outcomes of your entrepreneurship development efforts is also critically important if you hope to influence policy in your community or state. If you can show how your efforts to energize entrepreneurs are having positive impacts on your community, you will have an easier time convincing decision makers of the merits of supporting your efforts through new policies, programs and resources. Plummer, Idaho, for example, used project reports to build a case for collaboration with tribal authorities in several areas of community and entrepreneurship development.

However, keeping score about the outcomes of entrepreneurship programs requires considering and measuring impacts a bit differently from more traditional economic development efforts. You need to think more broadly, and more long-term, about how your strategy to support entrepreneurs is changing the culture in your community, building new capacity among citizen leaders and creating businesses that bring jobs and wealth into your hometown.

Linesville, Pennsylvania, provides some insight about how this kind of thinking is somewhat different than traditional outcome measurement for economic development. Envision Linesville, Inc. is a collaboration among local business owners, government officials, the local chamber of commerce and community service organizations. Their mission is to increase citizen interest, involvement and activity in the greater Linesville area and to promote and encourage healthy lifestyles and lifelong learning. One of their major goals is to build on the existing tourism base, which includes avid hikers, bikers and sportsmen and women.

Within two years after their initial meetings the group reports these measurable outcomes:

Established Envision Linesville, Inc. as a 501 C (3) nonprofit organization.
Created a master plan to establish National U.S. Route 6 Visitor Center and Museum and developed a partnership with the U.S. Route Tourist Association.
Created the Envision Linesville website to keep the community up-to-date on the organization’s activities.

Building an Evaluation Strategy

Most evaluations these days incorporate a series of questions that frame the assumptions behind a program. A community group making decisions about what actions to take should be able to answer the following questions about their project plans:

For whom?
What assumptions?
What process?
What outcomes?
What impact?

Called a program logic model, this approach really offers a picture of how your program actually works. We provide an example model under Tools in this section, but we’ll also illustrate the model with an example here. If your community group decides to create a program that will identify local entrepreneurs and offer them some special types of assistance, the program logic model would look like this:

For Whom:	Local entrepreneurs Targeted youth groups

Assumptions:	Locally developed Driven by the needs of local entrepreneurs Combines education and support groups Emphasis on peer learning Requires collaboration among various service providers

Process:	One-stop shopping Easier access for the entrepreneurs Referrals from a variety of service providers Adaptation of many marketing and promotional techniques One-on-one and group activities

Outcomes:	Increased number of entrepreneurs identified Support group created and maintained Connections to funding and specialized assistance programs High visibility for entrepreneurship creates more community interest New business starts

Impact:	Local economy is diversified More dollars generated locally

The program logic model is a way to concentrate effort. This may sound very difficult, but remember that at the community level, this can be done through conversational, informal work sessions. Even if one or two people take responsibility for drafting the answers, a larger group can use the draft for discussion. Getting more folks involved also means that more ideas will be generated and nothing will be missed. The danger of one person designing a project is that sometimes the obvious can be overlooked, while a group discussion and review can provide a lot of oversight.

One mistake that a community group can make is to avoid any evaluation at all or to do nothing with the information. Either it’s assumed that nobody in town is interested outside of the core group of volunteers or else the news opportunity of a progress report is simply overlooked. While there is a time and place for an “external’ evaluation of a community effort, a powerful way to keep track of progress is to develop a local team of three or more volunteers who make it their special focus to measure progress and report on it. You might call this team of volunteers an evaluation subcommittee or a progress monitoring team. Their job is to take evaluation seriously and make the most of the opportunity to measure progress and report back to the community.

Sometimes high school or college students can be very effective as part of this team. A high school, community college or university class can be brought into the picture to help collect information about progress and construct new and interesting ways to report the results. Think about a high school math class that might do a summary of evaluations from participants in an entrepreneurship class. Making a presentation of the results at a city council meeting, with newspaper coverage and photos of the students, would be a great way to get people to support more efforts targeted at entrepreneurs.

Students in the Edcouch-Elsa, Texas, school district have proven that the talents and interests of young people can be put to work in reporting on community progress. There, a group of motivated young people established the Edcouch-Elsa Printing Center, which writes, produces and markets print and video products for their communities. They also established a national electronic network for alumni to make connections for possible future community and economic development in south Texas.

The Progress Monitoring Team

So, what does this team of volunteers do? They should answer some basic questions, always referring back to the goals set for working with entrepreneurs. Here’s a review of the questions that form the work of the Progress Monitoring Team:

WHERE ARE WE RIGHT NOW? This question is all about establishing a beginning point or a baseline. For example, if you have a goal of “increasing the number of entrepreneurs identified in one year,” then the question really means, “How many entrepreneurs have we identified in the last year? What’s our starting point?” Remember, measuring progress is almost impossible if you don’t know that starting point. It’s like starting a diet when you don’t know what you weigh—how can you tell how many pounds you’ve lost or gained? The team has to establish a baseline for whatever the goal might be.
HOW WILL WE KNOW IF WE’VE MADE PROGRESS? What will be an indicator of success? What evidence will we see if we’ve made progress? Indicators might be things like an increased number of participants at entrepreneur workshops or a higher percentage of telephone calls to the chamber that mention entrepreneurship issues. Indicators are simply the evidence that the group decides to watch for since those will be indications of progress.
WHAT MEASURES WILL WE USE? The team not only decides what evidence they will look for (indicators), they must also decide whether they will count something (numbers of participants, for example) or figure some percentages (a decrease in bankruptcy for small businesses, for example). In other words, in what quantitative way will they measure progress toward a goal? Of course, the team also must decide and act on ways to collect information in order to keep track of whatever measures they decide are most important.

It’s most practical for the Progress Monitoring Team to decide on three to five indicators and just a few measures for each one of those. Any more indicators and measures makes for volunteer work that is way too complicated. Most importantly, the team has to guard against collecting too much data. Folks that volunteer for this task are usually people who are really interested and curious about information; it’s all too easy for them to let the task become a vast and complicated data-gathering journey. The indicators and the measures that match them have to be significant, not just interesting! Once again, it’s important for the team to focus.

Here’s a look at a sample goal and the indicators, baseline, and measures a team might use to measure progress toward that goal. The Progress Monitoring Team should work with the goals established for your entrepreneurship development efforts and then brainstorm the types of indicators and measures that make sense for your community.

Goal:	Make entrepreneurship education the focus of our community’s economic development efforts

Indicators:	Increased financial support for entrepreneurship classes Increased diversity among participants in entrepreneurship classes Attracting members to a new entrepreneurship support group

Baseline:	Current spending levels for entrepreneurship classes Age, gender and racial/ethnic background of people enrolled in the previous entrepreneurship classes Members, if any, in entrepreneurship support group

Measures:	Percent change in spending on entrepreneurship classes Percent change in participation in entrepreneurship classes by various demographics, e.g., participants under 30, persons of color Number of members in support group after one year

Balancing Stories and Statistics

Information about progress can take the form of either stories (qualitative) or numbers (quantitative). The Progress Monitoring Team should try to gather both kinds of information and use both to report to the community. Stories that feature an individual or a business can be inspiring proof that everyone’s efforts are paying good dividends to the local community. However, reporting numbers of participants in entrepreneurship classes or a percentage of change over time in the number of entrepreneurs counseled or visited can be very compelling information at a budget meeting! Don’t rely on just one type of information for a progress report. Broken Bow, Nebraska, has done a great job of getting media coverage that combines facts and human interest pieces to tell the whole story of each successful development effort.

Measuring Outcomes Over Time

Entrepreneurship development efforts take time. Entrepreneurship is a long-term economic development opportunity —not a short-term fix. However, it’s important to show progress quickly, if possible, to keep the momentum and support for your efforts growing. You should consider both short- and long-term outcomes as you are developing measures of success. It’s very important to match expectations regarding progress for your program with the appropriate time frame. For example, you may not want to use job creation measures to report progress in the first 6 or 12 months of your program. You should be focused on capacity-building measures like number of members recruited for your entrepreneurship steering committee, number of entrepreneurs visited by the steering committee, or number of newspaper articles written about entrepreneurs.

Once you move into the second year and beyond, you can begin to focus on measures that show a quantitative impact on the community, such as the number of new businesses started by entrepreneurs in your community. Later in this section, we’ll introduce an Outcome Measurement Tracking Tool that can be very useful in matching measures and time frames.

Getting Information from Entrepreneurs

You really need two types of information from your entrepreneurs. You need tracking data on the business itself. Is the business growing? Are new jobs being created or existing jobs being saved? Is new capital coming into the business? These data usually come through the service providers that are working directly with entrepreneurs. While you don’t want to overwhelm busy entrepreneurs with too many requests for data, it is important to be able to report back to funders and policy makers about the impact of all your efforts on entrepreneurs. We provide an Outcome Tracking Tool under Tools in this section that you can adapt to meet these data collection needs in your community.

It’s also extremely important to consistently get feedback from entrepreneurs—your customers! Listening to entrepreneurs can help you measure progress and, most importantly, improve your programs. A focus group of entrepreneurs held once a year can be an invaluable tool for measuring progress and for getting ideas on how to expand and improve efforts. This doesn’t have to be a heavy burden for your volunteers either, since it’s a great task to delegate to a high school or community college marketing class. And, it gets the students involved, too. Under Tools in this section we provide an Entrepreneur Focus Group Questionnaire—a tool that you can use and adapt for a focus group with entrepreneurs in your community.

Reporting Results to the Community

What should you do with the information the Progress Monitoring Team collects? Certainly, any funding department or agency gets a report, but we’re concerned with getting the information out to the community. This is a significant way to build political support for entrepreneurship efforts, recruit volunteers and also to market a program to the users themselves! Cheney, Kansas, uses an annual town hall meeting to report progress and set future goals. If a team has collected some information that offers evidence of progress, here are some additional ways to get the word out to the wider community:

NEWSPAPER ARTICLES. Feature stories of successful entrepreneurs, photos of participants in an entrepreneurship class, graphs of increased enrollment, use of services or any type of visual display of a measurement will make the paper.
RADIO PROGRAMS. Taking calls on a local program can be an excellent opportunity to “brag” about your progress.
CLUB AND CHURCH NEWSLETTERS. Feature success stories or even an abbreviated version of the newspaper article can help spread the word about your activities.
SPECIAL MAILINGS. A brief cover letter and a one-page report to boards of directors or officers of service clubs and elected officials is a good way to keep them informed about your progress.
WEB PAGE. Here’s the place to show off all the photos, thank you letters and complete progress reports. Be sure your web address is printed on everything to encourage visitors.
SPEAKERS’ BUREAU. Develop a simple Power Point presentation or series of overheads and recruit two volunteers to make the rounds of the local service clubs with a 15-minute presentation on progress. Have a simple, one-page handout on the report and you’re set to get folks talking about entrepreneurship ideas.

Under Tools in this section we provide five tools to help you measure the progress of your entrepreneurship development efforts:

Program Logic Model
Outcome Measurement Tracking Tool
Entrepreneur Focus Group Questionnaire
Entrepreneur and Small Business Community Metrics Tool
Participant Evaluation for Special Events

You should adapt these tools for your community, to include specific measures that your Progress Monitoring Team decides are important. For example, if you are specifically targeting displaced workers with entrepreneurship education programs, you might want to conduct a focus group with displaced workers and workforce development board staff instead of (or in addition to) a focus group with entrepreneurs.

You can be creative in how you use these tools. Just remember—the most important thing is to make a commitment to keeping score and sharing your progress with the community!

Additional Resources

Make Success Measurable. Douglas K. Smith. 1999. John Wiley & Sons. Although this book is focused on teams in a business setting, the techniques for setting goals and measuring results are easily transferred to a community group or committee.

W.K. Kellogg Foundation Evaluation Handbook. 1998. This is an excellent resource guide to all types of evaluation and measurement, including the use of program logic models with a variety of examples used to illustrate keeping score. www.wkkf.org

Measuring Community Capacity Building: A Workbook-in-Progress for Rural Communities. 1996. Aspen Institute. A version of this workbook is available for free downloading at the Aspen Institute’s website. It includes many examples of how to track and measure various aspects of community capacity that might seem impossible to quantify. www.aspeninstitute.org.

The Center has more resources and can provide additional assistance to you as you support entrepreneurs in your community. To access Center Resources, click on our logo.