Beyond Vanity Metrics: Measuring Real Impact

This article is a summary of Episode 39 of our Designing Tomorrow podcast. Each episode is a conversation between Jonathan Hicken, Executive Director of the Seymour Marine Discovery Center, and Cosmic’s Creative Director, Eric Ressler.

Let's talk about evaluating impact. That may seem like a really simple, obvious thing for any social impact leader. We want to talk about evaluation and we want to talk about measurement, but we want to talk about those a little bit differently. It's worth separating the two of those.

To us, evaluation feels like a more holistic approach and measurement feels more like a tool or a tactic to support broader evaluation efforts.

A broader evaluation can be qualitative in nature. It can resort to stories and it can resort to relationships. Evaluation could be many things. Measurement is that quantitative piece. Sometimes as leaders we kind of mix those two up and there's a place for each of those in evaluating impact.

Why Evaluation Is So Hard

We've worked with organizations that are really good at evaluation and have evaluation experts on their team. We've also worked with organizations that don't seem to really do this at all — which honestly kind of surprised us the first couple of times we stumbled upon that. And then the more that we learned about it, the more that we started to realize that this is actually really hard. This is really hard to do well — especially when we're talking about systemic issues or really complex or wicked problems — and maybe even problems that take years if not decades to be able to even fully evaluate. We started to understand a little bit more about:

1. Why there are experts and consultants that specialize in evaluation as an entire career path, and
2. Why so many nonprofit organizations and social impact organizations in general just don't have a very strong evaluation program or model built into their approach.

It is really challenging, and one of the things that we struggle with in this space is that we're often trying to solve these extremely complex problems and we're being asked to evaluate our own work in terms of solving that huge problem — when in many cases we can't tease apart all of the pieces or the players or the actors or the reasons why a problem might get solved. What we can do is we can ask ourselves: is our contribution doing the thing that it's supposed to do, as opposed to, am I solving the problem as this one singular organization?

Define the Problem — And the Part of the Problem You Solve

We’ve talked previously about getting clear on — What is the Problem? We need to expand the question beyond just "What is the problem, and what part of the problem are we trying to solve?”

It’s the first thing you must do. If you're reading this and haven't gone through that exercise, you should go do that and then come back and restart. We’re gonna talk about this as if you have that problem already defined.

Starting with defining the problem and defining the piece of that problem that you solve is absolutely critical.

An Example from the Seymour Center

At the Seymour Center, they work in the science education space. They talk a lot about creating scientists and launching curious young minds to pursue STEM education and careers. And they know that in 20 years, a 7-year-old who comes through the center — whether that person decides to pursue a STEM education — there are going to be a lot of factors on whether or not they choose to do that. The Seymour Center's contribution might be one of those factors. But they can't credibly claim that the reason that 7-year-old, in 20 years, is pursuing a STEM education is because of their contribution to her journey.

And that is a really good — if not perfect — example of one of the challenges of measuring and evaluating impact, which is that the impact story took 20 years. If you tried to measure that at year two, there might be some leading indicators — this student signed up for extracurricular science activities by choice, or something like that. But if one of the things you're trying to measure is new scientists created, this is a perfect example of just how hard it is to measure those things. 

At the Seymour Center they share one story of a scientist that actually was born out of an experience at the Seymour Center and came out and claimed that. That's such a perfect impact story to have. But there might even be 10 of those in motion right now that they don't even know about. Almost certainly there is, and they may not ever find out about those. Even if we do know, there's no way of proving that the Seymour Center was the only reason that they pursued that education or career. 

And therein lies the challenge for a lot of social impact organizations when it comes to evaluating their own impact — because there are so many factors, whether it comes to housing and homelessness or food security or you name it. There's a million worthy causes out there that are so interconnected with other particular causes or other variables or other levers. It's really hard for us as leaders to stand up and make a claim about our services being the reason that somebody's life was changed. But we have to, because we're asked to do that in order to earn support or earn a funder trusting us.

Evaluating Impact

Let’s look at three key components to evaluating impact:

1. Is our work moving us towards our vision?
2. Is our work allowing us to build a sustainable business?
3. Are the things that we're doing day-to-day successful?

So let's break that down. The first one really comes back to impact. It's the impact statement. If we are doing this work, is that work truly solving a problem or at least solving a part of a problem as we were just talking about? A lot of this comes down to thought experiments. 

Let’s use the Seymour Center as an example. Right now, something like 8,000 young people come through the formal youth programs at the Seymour Center. The question to ask is: if every single public school student in our county came through our program, are we doing a better job of solving the science education problem? The first place to begin evaluating your impact is to ask yourself and be certain that your work is advancing society towards the vision that you've set forth.

If you don't know truly whether that's going to happen fully for 15 or 20 years, some amount of measurement for other signals or stories or leading indicators are needed to show you're on the right track with these students.

And that brings us to the third pillar — is the work that we're doing day to day, are we successful in that work? For this example, the Seymour Center is collecting testimonials. They're collecting heuristic evaluations from teachers, from students, from the instructors in the classes. They're all evaluating their experience there. And ultimately it comes down to whether or not they believe that the students were engaged in meaningful science learning. That's how the Seymour Center is collecting those leading indicators. Did those young people have a great experience at the Seymour Center? And for the Seymour Center leadership at least, that’s the proxy to impact. They have to believe that these meaningful moments of science, when added up over the course of a young person's early life, can ultimately lead or put that person on a journey towards that education or career.

A Framework for Reverse Engineering Impact

A useful mental model is to think about some kind of desired future state — in this case, creating new scientists — and reverse engineering what would have to happen for that to be true. On day one, on experience one, in year one, in year three, in year ten. 

The idea of that thought experiment is a really interesting framing. Because if all you do is focus on the end goal, but don't have a way to track progress — especially in situations where impact isn't necessarily going to happen instantaneously, which is true for a lot of social impact organizations — then you have to deconstruct and reconstruct the answer to that question: 

This is something we think every social impact leader should be thinking about.

This brings us back to the messiness of social impact. We could plot that young person's journey between age seven and age 27, and there are going to be so many touch points on that person's life that are completely out of the control of the Seymour Center. Is there a version of this where they create partners with schools and other organizations to create a seamless journey through her young life? Maybe. That also sounds incredibly daunting.

Watch Out for Mission Creep

This is often where nonprofits get into mission creep or mission drift. Because you start to see — especially when you're in this mental mode of thinking about the big picture and the ultimate impact and the vision and these big problems that you're only solving a part of — you can see all the connections. You can see; If we just tapped into this school network, then imagine the impact we could have there. And if we just tapped into this parent network, imagine the impact we could have there. But then you start to dilute your impact. You're diluting, diluting, diluting. And if you don't have strong measurement and evaluation and the right mental model to really hyper-focus on the part you are uniquely positioned and best at, then you might actually start to hurt your impact and the depth of your impact.

For all the criticisms of academic science and publishing, one thing that is great about academic publishing is there's this section called Next Steps or The Next Study. It’s a model that really works. A scientist goes deep on a particular question, and over the course of their work, they identify the next work that needs to be done. Social impact leaders could do the same thing — here's the thing that we're doing, we're really good at this piece, but through the course of their work, they see that these other gaps exist. Maybe put a call out to your community or to funders to solve that problem. Because you’re not in the position to do it, but you can name that next step.

Component Two: Can We Build a Sustainable Business on Solving This Problem?

This one surprises people. And it's a huge problem that often gets overlooked in the social impact space. We've kind of accepted that this isn't going to sustain itself, but we're just going to laser focus on the mission and it'll work out somehow — that passion-power that happens in this space a lot.

Unless you're an organization that's just going on a five year sprint — we're going to just invest a bunch of money, we're going to solve a problem or we're going to get out. Which if you're going to do that, more power to you.

It's the Navy SEAL model. Get in, solve a problem, get out. If you can do that, great. We applaud that.

But most of us are looking to build a sustainable business because we care about our people and we want to make sure they have stable work, and we are solving problems that aren't going to be gone in five years. And so the best thing that we can do to deliver impact is to ensure that that work is still viable in the years ahead of us.

Measuring impact is not just about external impact but internal sustainability. Because if that's not healthy, then none of the rest of the work is going to be healthy either. It's really a precursor or a prerequisite to doing impact work at all. And doing that well. And of course, you're not going to be able to just pause all of your impact work and make the business sustainable and then get back to it — that's not how this is going to work. But the framing of including some kind of sustainability score or measurement as part of your impact measurement is a really solid idea.

We suggest that when a social impact leader hosts their monthly all-hands staff meeting, they look through all of their measurements and all of their metrics. Look at impact metrics and at financial metrics — because in order to deliver the impact that you seek, you need to take care of yourselves too along the way. Share that message with the team regularly. You need to build a healthy business in order to deliver the impact that you have promised your community. That is inherently part of the impact evaluation.

Component Three: Are We Successfully Doing the Work Day to Day?

Is the work we're doing helping us solve a problem, moving us towards our vision? That's more of an almost theoretical evaluation. The third is: is the work we're doing working?

There’s a system called Objectives and Key Results where organizations essentially break down everybody's jobs and everybody's tasks into discrete, measurable outcomes that tell us if we are doing our jobs successfully. 

Let's take an example from the Seymour Center. They have a contributed income target — that's the philanthropic target — and they break that target up into its discrete pieces. There’s individual giving, major donor giving, and university giving. They also have a membership giving piece. Multiple people are contributing to the membership income piece. They have their membership acquisition team, which is at the front desk, and they have their retention team, which is kind of back of house or their fundraising team.

So they then break down: for the front desk, one of their key outcomes — an indicator of the strength and health of the team — is how well they're able to sell memberships to people who walk in the door. And so they set daily, weekly, monthly targets for membership sales, and they look at those together as a team. That's an example of breaking down a top level metric, which is contributed income — we know we need to have that in order to have a healthy business — and breaking that down into an individual team member's job. At the front desk, one of their tasks is to sell memberships. Are they doing that or not?

These three steps of measurement are really looking from a macro to a medium and then a micro level. At the big picture: are we achieving the goals that we're setting out to achieve? In the middle: are we doing the right things that we think need to be true in order to achieve those goals? And then the last step is: are we doing the things every day that are going to allow us to be successful in doing those right things? 

Now, that's just how one organization approaches impact evaluation.

What We’ve Seen at Other Organizations

We've seen evaluation done in different ways. We've often seen people bring in external consultants and evaluation firms to do this more objectively, or maybe it's a mix of that and also they just don't have anyone on their team who specializes in this. That process is a little bit opaque to us. We just get the results of that work — and often in the process of trying to collect impact for storytelling purposes and for marketing purposes.

We've often also seen a lot of organizations who don't have a very strong evaluation and measurement program in place, and they lament the difficulty of doing this well. It is always in the spirit of: We're working on trying to get better about this, but we're just having trouble because it's complicated, we don't have the capacity, et cetera. And so those types of organizations tend to lean more on qualitative measurements: testimonials, social proof, individual stories — versus more quantitative analytical metrics around number of people served or whatever. Or even sometimes the way that those qualitative metrics are framed can feel a little bit like things that are easy to measure, but maybe not all that impactful in the end. 

For example: Number of people served — did you serve them well? Did you actually create behavior change? Did their outcomes improve or did they just come in and out the doors? This isn’t a judgement. This is just what we see.

Why Is Evaluation So Hard for Social Impact Leaders?

There are probably a number of different reasons. Sometimes people are so focused on doing the work that they don't consider this an important part of doing the work — or it's just something that kind of slips through — or they don't have the capacity or the budget to build this into their work — or they don't have the culture from a leadership level to see how you might build this in with the existing resources and capacity that you have.

And honestly, sometimes we wonder: Is this something that they know they can get away with not doing? Because if you tell good stories — and we do believe in the power of storytelling, and we think it's very powerful — but we do think it needs to be grounded in objective truth. And because more broadly, if you're not in the social impact space, if you're not working at a nonprofit or a social enterprise, there is sometimes a lack of trust in social impact organizations. The perception is that those organizations are scams, that they're not actually making a difference, that they're just paying salaries to their executives — Which we find hilarious because if you actually know anything about the space, how absolutely ridiculous that is. 

Of course, there are situations where there has been fraud and scams and fake nonprofits set up, of course. But those are by far the exception over the norm, and it's actually kind of insulting and degrading to so many people who've sacrificed so much to work in this space. 

Nevertheless, there is a certain percentage of the public that is skeptical of nonprofits. And there is some validity to that skepticism, especially if you start to look at some of these issues that have not really seen much progress despite lots of funding and lots of organizations working on them. Which to us just makes this conversation that much more important. 

We need to be good about evaluating and measuring impact — first of all, so that we can be sure that we're actually making a difference and have data and science backing that up and not just good feelings — but also so that we can have good ammo for building trust and credibility with our supporters and activating and nurturing new supporters.

Do You Need a Consultant?

Some social impact leaders might gag at the idea of bringing in external consultants to help with evaluation and measurement because it may be a lot simpler than maybe we make it out to be.

This is something that any executive leader — if you're not already measuring success and discussing with your team — it's not a ton of work to get started on. This is definitely one of those "start somewhere and keep getting better over time" kinds of situations. You don't need to pay a hundred thousand dollars or more to have somebody tell you how to do it. This is something that you can start measuring, even if it's: We're going to pick three things and as a team, we're all going to measure these three things for a year, and then if we like it, we'll add another. 

There's a simple starting point, and part of it is just building a habit. If you're not looking at numbers with your staff in your one-on-ones, start looking at numbers tomorrow. If you're not doing that at your all-staff meetings, start doing that at your next meeting. Look at a number as a staff and just have a discussion about it and get used to it. Build that muscle. Build that habit. For most leaders, hiring a consultant to begin measurement and evaluation of impact is not necessary.

That said — and this is a counterpoint — this culture of leadership, this way of thinking, is how some people's brains work and they have training in it and they like to geek out on measurement frameworks and OKRs. You’re reading business books about this. Which we respect and applaud. But that's not how everyone's brain works. And so if this feels really difficult to you — and we agree, you should try — but the right outside consultant for the right team actually can make a big difference, especially if that consultant is coming in to help build internal organizational capacity and culture to be better at this.

Should you rely on an outside consultant to do this — think about it as outsourcing the work? That is a mistake, because that is essentially pushing the problem off your desk when it needs to be more integrated into the work that you're doing. But if you as a leader struggle with this and you don't know where to start and you're not the type that is comfortable experimenting, maybe that outside consultant — if you find a good one — is just what you need to get that early framework going and get that behavior-changing culture going.

This is something that we at Cosmic actually struggle with a little bit. As a creative agency, and not everyone here is analytical in that way. Our creative brains fire on all cylinders at all times and get inspired by a lack of structure — which is odd because we run an extremely tight ship and a structured business. So we can hold both. But as we've been trying to improve our own KPIs and measurements and make data-driven decisions, it can be a real struggle. And we're lucky to have other members of the team who are really strong here, and so we've been able to build those up and they take ownership over that.

Bringing in a consultant isn’t always a bad choice. You have to figure out what is your culture, where are you strong, and where are you weak. And if you do bring a consultant in — if that is the right choice for you — to ensure that you do it in a way that's building culture and capacity that then you can take over. Maybe that’s with some additional consulting here and there, but not the idea that you're going to kind of push this off your desk onto a consultant to do.

Don't Forget the Constituent

There’s one pitfall worth naming with any of this evaluation. A lot of impact evaluation can be navel-gazing — very inward looking. There's a time and a place for that. If you're talking about your revenue and your expenses, these are things that really only you and your team are looking at. Maybe some funders, maybe your board, but really it's internal looking. There's a place for those.

The one piece that is often missing from an evaluation dashboard is value to the constituent. Spending time to understand how your constituents — the people you're serving — define value themselves. Are you helping to deliver the value as they define it, not as you define it? And this is actually really interesting for an agency like Cosmic as a business — because value is probably defined a little differently by each client. We can't have a standard metric for success because each client is looking at it slightly differently.

Up to this point, we’ve really been thinking about this from the lens of an executive director or a nonprofit leader. But honestly, this has actually been a huge goal and struggle for us at Cosmic. So much of our work is brand and brand is actually really, really hard to measure — and especially to fully capture the value of brand. 

It can be expensive if you start to think about awareness or affinity — traditionally you would invest $60,000, $100,000, $200,000 to run a huge research study to figure out your brand metrics. Nonprofits aren't going to and should not invest that amount of money (unless they're really big) to measure brand. And so a lot of the branding work and the storytelling work can be really difficult to measure. So there is real empathy — especially for nonprofit organizations that are doing really systemic, intractable work where it can be hard to measure.

We do relate to the struggle of how to put metrics behind all of these different things that are important to measure. What you measure matters. So to that last point about: Are you looking too inward? Is this too navel gazing? Making sure you're measuring the constituent experience and their outcomes and what they're looking for, not just your own internal metrics, is really important. And that's actually been the part that's been hardest for us.

Measuring our own internal stuff is relatively easy — our business model is simple. But measuring the impact that we're having, which can be a little bit more qualitative in nature and can be a little bit murkier — that's the part that we've struggled the most with.

Keep Asking: Are We Measuring the Right Things?

One last tip: As a leader, at least once a year, ask yourself — are we measuring the right things? And be ready to shift those things. Especially once you've gotten your feet under you and you're feeling really confident with measurements, you can get stuck measuring the same thing for years and years and years when actually success may change around you.

Check out the full conversation on our Designing Tomorrow podcast.