Lonely numbers
The most important thing you can do to avoid misjudging something’s importance is to avoid lonely numbers. Never, ever leave a number all by itself. Never believe that one number on its own can be meaningful. If you are offered one number, always ask for at least one more. Something to compare it with. – Hans Rosling, Factfulness
Why was six afraid of seven?
Because nine was six’s best friend, and after seven ate nine, six wasn’t sure what to do with their time. They were sitting up there on a table in the Q3 business review thinking “what’s the point” and “what’s the meaning of it all,” and those are the same questions executives have when you have lonely numbers.
Everyone in marketing has seen a chart like this.
Just as often, I’ve been in meetings where people say things like:
“This program drove 13 qualified leads last month” (ok…is that good? More or less than expected?)
“We generated 596 visitors with this blog post” (again…is that good? Is that enough to meaningfully impact the business?)
“Last month we were down 15% but this month we’re up 20% over last month” (wait…where does that leave us?)
This kind of reporting is everywhere, and it’s frustrating to make and frustrating to read. It’s also part of what keeps people (maybe you, maybe not) from getting what they want at work because it doesn’t answer the two questions your bosses need reporting to answer.
1) How is this going to impact the business?
“If you had a magic wand, and what you were about to do went perfectly — would it still be worth doing? And too often in marketing, that is actually just not the case. You’re going to know the outcome, you’re going to do it well, and it’s still not going to be worth doing. It’s still not going to have a big enough impact” – Kipp Bodnar, CMO at HubSpot
It’s surprising how often people put a lot of work into projects that won’t pay off even if they go perfectly.
I once worked with someone who wanted to get a big influencer and pay them $10k to co-host a webinar for the company. My napkin math said we could reasonably expect ~500 registrants — but even if we had done double or triple that number, 1,500 webinar attendees were simply not going to turn into enough business to justify the $10k in spend.
Try not to do this.
That sounds flippant, but it seriously happens a lot. These are all real examples (not necessarily businesses I’ve worked in):
This quarter’s sales aren’t to target, so the business (which has 50,000 customers and gets 1,000+ new ones per month) puts out a promo. It generates 16 sales. Next quarter the same thing happens.
A business with 500k monthly website visitors wants more traffic, so they publish a few blog posts. The handful of posts get a cumulative 2,500 visitors (0.5% of baseline traffic).
A webinar gets 3,000 registrants — a new record for the webinar program! But overall sales that month still wind up down because the company got that number of free trials every day.
Sometimes in startups you’ll hear “do things that don’t scale,” and you shouldn’t put pressure on brand new channels to be fully scaled up by day one.
But what you really really don’t want is to report on a lonely number (“we influenced 50 sales last quarter”) and leave your executive with the question “is that good?”
Reporting needs to contextualize your numbers so that it’s clear how your projects are contributing to the overall business. When you report to your boss and your boss’s boss, the value of your projects should be self-evident.
And to be blunt — if you have trouble doing that, how sure are you that your projects are in fact the right projects?
2) How is the program actually doing?
Many years ago, David Chambers found a time series on the wall of the office of the president of a shoe company. Here was a simple and powerful presentation of data in context. The caption on the vertical axis was “Daily Percentage of Defective Pairs.”
Intrigued, David asked the president why he had the graph on the wall. The president condescendingly replied that he had the chart on the wall so he could tell how the plant was doing.
David immediately responded with “Tell me how you’re doing.”
Evidently no one had the temerity to ask the president this, because he paused, looked at the chart on the wall, and then said, “Well, some days are better than others!” – Cedric Chin, Referencing Donald Wheeler’s Understanding Variation.
Ok so my incredibly accurate table at the beginning of this article is actually not lonely numbers, right? After all, it’s comparing one month to the next month.
Maybe.
Thing is, you can be looking at all of the numbers relevant to a business area and still not be clear how that business area is doing. The shoe company president in that anecdote has the illusion of understanding because he’s looking at a chart that reports every day’s defects — but how does he know if the plant is producing too many defects, or why?
Month over month comparisons aren’t good enough for many reasons, but to focus on a few:
If you go down one month, it’s easier to go up the next month. There is random variation in business. Lots of businesses wind up in a situation where they think “oh no we’re down!” followed by “oh ok good we’re back up this month,” followed by “oh no we’re down!” Even though nothing in the business has changed at all.
Lots of businesses have seasonality that is not actually accounted for or is handwaved as “oh it’s seasonality.” In theory you could account for this (mostly) just by saying what portion of the effect is expected to be seasonality (e.g. based on historical numbers, we should be down 10% in the first week of May), but this happens less than I would hope.
It’s hard to look at numbers in these charts and answer the question “is this good.” Looking only at MoM (or QoQ, etc. etc.) numbers leans into a panic-relief cycle that could obscure what would happen if you looked at a real time series — which would show a program that’s flat, growing, or in decline. Why it is those things is less clear, but you have a better place to start from.
A lot of business reporting is done because it is what the business does. People often treat it as justification of their work when they should be using it as the backbone for decision-making.
I understand the feeling of defending your work. But also, if the numbers don’t clearly defend your work, your best defense is not to defend bad numbers. It will never last.
What do you do instead? There’s another article in there, but here’s the gist:
Clearly delineate which numbers your work is intended to affect, and why those results will have a meaningful impact on the business if you can realize them
Show the numbers related to your program over time, contextualized against the business impact you chose in step 1.
If you find yourself defending projects that aren’t going to drive the business forward (either because they aren’t impactful enough or they don’t seem to be working), take steps to move on to something else.