From Nonprofit to Social Profit: Exploring Assessment For Social Good

This is an excerpt from the recently released book "The Social Profit Handbook: The Essential Guide to Setting Goals, Assessing Outcomes, and Achieving Success for Mission-Driven Organizations."

By David Grant

Imagine an executive director of a social profit organization. She is working late the night before a board meeting, completing a proposal to a local foundation, one to which she has applied before. The proposal form asks, “By what measurable outcomes will you determine the success of your work?” She repeats the metrics she reported on last year: number of workshops offered; number of people served; scores on evaluation forms at the end of workshops. But she can’t help but feel something is missing. She would like to describe the biggest change the organization made that year, but that would mean describing the big mistake that caused it as well. Maybe not.

Imagine a foundation officer receiving the latest proposal from this social profit organization and re-reading the report submitted on the previous year’s grant. It says the organization served 1,200 people, a 20 percent increase from the year before. She asks herself: "But is that good? Should it have been 30 percent? And how well were they served?" She reads the report, which says in effect that social problems are getting worse but everything is going better than ever with this organization and the foundation should continue in its wisdom and generosity with ongoing funding. The foundation officer thinks: This feels more like a dance than an evaluation.

These people deserve a better assessment system. And so do the people this social profit organization serves.

How we think about assessment is significantly affected by how we think about measurement. Social profit does not have a standard unit of measure, like a dollar, a centimeter, or an ounce, and thus is difficult to measure. Some would say it is impossible to measure. Yet talk of measurement is everywhere in the social sector. The public and politicians question the efficacy of the sector and ask for proof of results. Board members seek ways to assess progress over time. Funders ask for hard evidence of impact.

Much of the talk has to do with “metrics” and “outcomes,” and technology has given us the chance to gather, analyze, and present data as never before. As a result, most organizations in the social sector choose key metrics and keep track of them. We describe our goals in numbers, and indeed, metrics can be important.

The case for further investment in social rather than financial profit is bolstered by a body of metrics associated with key indicators: percentage of a population suffering from diabetes or obesity; percentage of young men in prison in the United States, of which races and ethnicities; parts per million of carbon in the atmosphere; number and size of farms being managed sustainably; a nation’s score on the United Nations’ index of “material well-being of children”; data on infant mortality and life expectancy. The list goes on.

Yes, metrics can focus our attention and increase our motivation. But as we know, setting targets does not necessarily help us meet them. We must ask whether this data is making our organizations any better in our internal workings and in fulfilling our missions.

Who measures what and why?

When I was at the Dodge Foundation, I remember interviewing the founder and executive director of an organization in Newark that offered an arts education program designed to be a deep, transformational experience. Anecdotal evidence suggested that it was; the program seemed to be profoundly changing the lives of the young people who participated. Yet the ED was holding his head in his hands as he told me about a recent visit from another foundation officer, a key backer of this program: “All he said to me was, ‘You have to get your numbers up. You have to get your numbers up.’”

Does the number of kids in a program trump the quality of the program when assessing whether or not it is successful? It can and often does if there is no qualitative assessment that defines a different vision of success. For better or worse, measurement becomes a proxy for intention and values.

It is hard to argue with the foundation officer’s intention to serve as many young people as possible. But that was not the executive director’s intention certainly not his highest aspiration. He did not have the capacity to do that. He did have the capacity, and it was his mission, to change some number of young lives through the arts. But there was no metric for what mattered most to him, nothing with which to make an alternative case. This is a familiar story.

In workshops with social profit leaders, I sometimes ask, “Do you feel you are measuring what matters?” and more often than not, I get a quick no. Frequently, they say they did not think they had the option to measure what matters to them — and besides, they say, it wouldn’t be a valid measure, would it?

This is a critical question. At least part of the moral of the story is that if you do not define and assess what matters to you, someone else will do the assessing of your work, based on what is important to them. The other part involves an assumption and a question: What holds us back from thinking our internal, homegrown assessment would have legitimacy?

Validity and reliability

We live in an age when measurement and its uses in assessment and evaluation have become a serious science. Indeed, when a measure itself has to meet standards — think of the SAT or the Richter scale — we quite properly train our attention on those standards. Is it accurate? Is it reliable, which is to say does it give consistent results? Is it valid, which is to say does it measure what it is supposed to measure? These criteria are extremely important if a carpenter is measuring for a shelf, or a coach is weighing in wrestlers before a tournament, or a doctor is drawing blood to determine levels of uric acid. But does a measure always have to meet strict criteria to be helpful? I think we get confused over whether it is the measure that matters or what is being measured. We get intimidated by the science of measurement, forgetting that, in the words of change expert Michael Fullan, “Statistics are a wonderful servant and an appalling master.”

We find ourselves arguing over whether a measure of levels of quality can ever be accurate or valid. But what if we were able to agree that a measure is accurate enough, or valid and reliable enough, for us to take sensible and appropriate action based on what we learn from it? This question is important because we know any measure of social profit will never have the consistency we seek in standardized measures, nor the precision.

I know from the foundation perspective how difficult it is to compare the outcomes of two different arts education programs, let alone weigh the relative benefits of preserving a hundred acres of farmland or providing “talking circles” for troubled adolescents. But I also know how enthusiastically we responded to our grantees who had a process for identifying community needs, priorities, values, and visions, and who described in detail their aspirations and their plans for achieving them.

We did not talk about criteria of validity and reliability, because they were not the main point. We were basing our social investments on something else — a clear process that led to clear benefits, even if those benefits were described in words instead of defined by numbers.

Measurement and accountability

I remember a time early in my career, when my wife and I founded the semester program at the Mountain School, and we were scheduled to have our first visit from the State of Vermont. I prepared voluminous material about the new educational program and was excited to talk about its innovations: the way the English and science courses were taught in relation to each other and to our rural environment; the use of student journals for personal reflection and group process; the three-day solo camping trip; the work program; the student-led morning meetings. You must be shaking your head at my innocence.

The visit focused on two things: whether the furnaces were safe, and whether the kitchen was up to code. This was my introduction to the idea that while program performance may be foremost in the minds of social profit leaders, it may not be in the minds of their outside assessors.

Indeed, social profit organizations are accountable in at least two major areas aside from program performance: keeping their operations legal; and appropriately managing their finances. And because many social profit organizations receive government funding, they soon learn what governments value most — compliance with minimum standards in these areas, defined by law. The danger of this situation is obvious.

In a sector where many organizations and leaders are struggling to survive, they respond first to the immediate incentives in front of them to be compliant in all legal matters and in financial accounting and reporting. This compliance takes more time than we have, and we get used to the idea that assessment and evaluation come from the outside world.

But the outside world, particularly government, is not pushing on quality of work or level of service to clients and constituents, because they don’t have a measure for it beyond numbers served. There is no law that we have to be great at what we do. We should train ourselves to ask, Whom does this measurement serve? I’ll never forget the first time I realized as a teacher that the primary purpose of the SAT was not to help my students; it was to help adults choose among them.

Similarly, traditional outside measures of the work of individual social profit organizations are not designed for them; they are designed to make distinctions among them. So, surprisingly, measuring program performance seriously, honestly, rigorously, continuously, in a way that can inform and improve our work, is up to us.

It is critically important to do so despite the dearth of outside incentives, because that is where social profit lies.

Measuring what matters

It is helpful when thinking about measurement to remember the lines often attributed to Albert Einstein: “Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted.”

I am arguing that there are particular kinds of social profit that resist quantification, that resist standard measures, that are hard to pin down because they are dynamic in themselves, but are at the heart of how we imagine our missions being realized in the world. Faced with the realization that we will never find a common unit for these visions of social profit, we have two choices. We can say it cannot be measured, or we can measure it ourselves in the manner that social profit demands: a combination of pertinent metrics and a qualitative description of that social profit that can only be created by the people who are providing and receiving it. We must envision and define and then reflect and redefine these visions of success, believing that “if you measure what you value, people will value what you measure.”

David Grant is the author of "The Social Profit Handbook: The Essential Guide to Setting Goals, Assessing Outcomes, and Achieving Success for Mission-Driven Organizations."