How to think about the ROI of data work

In a way that makes you look smarter than your boss

It happens at least once in the lifetime of every data leader or practitioner. You get asked to estimate the ROI of your work, team or of hiring one more person. If you’re lucky you get away with a vague response around making the company more data driven. If not, you may find yourself trying to estimate the $ value of each data project pulling more hairs out for each one.

Your job: make them understand so you get what you want.

People much smarter than me have shared their takes on how to measure analytical work 

• Benn from Mode Analytics thinks that time to complete an analysis is the best measure

• Barry from Hex suggests that your stakeholders should be the ones advocating for your work and telling the ROI story

I have another take (hold onto your hat and glasses as we’re about to get technical). Next time someone asks about data ROI you show them this formula and hold your breath hoping they’re not running for the hills.

Convinced? Probably not. Allow me to elaborate and then let’s put it into practice. 

Not all ROI is created equal

In the article: The most crucial mind shift in a data role? Focus on impact I make the case that data people should focus on impact instead of outputs, align themselves with a key KPI and not forget about the world outside of data. 

There are five roles in a modern data team. The closer you’re to the left, the closer you’re to systems. The closer to the right, the closer you are to directly impacting a KPI. To keep it simple I’ll classify these roles into Systems People and KPI People. 

You should think very differently about impact depending on which role you’re in.

Consider a concept we’ll call Ƞ degrees. Ƞ represents how many degrees away you’re from impacting a top-level KPI.

Here are some examples to bring it to life

Ƞ = 0
0️⃣ A Machine Learning Scientist ships a model that reduces fraud →
💲💲💲

Ƞ = 1
0️⃣ A Data Scientist runs an A/B test to improve signup rate →
1️⃣ An Engineer implements the change →
💲💲💲

Ƞ = 2
0️⃣ An Analytics Engineer improves a data model for the A/B test →
1️⃣ A Data Scientist runs an A/B test →
2️⃣ An Engineer implements the change →
💲💲💲

Ƞ = 3
0️⃣ A Data Engineer improves dbt →
1️⃣ An Analytics Engineer improves… →
2️⃣ A Data Scientist… →
3️⃣ An Engineer… →
💲💲💲 

You get the point. 

The further you’re to the right the more your work should speak for itself through directly impacting a top-line KPI. The further you’re to the left the more the more you have impact by making data consumers downstream more efficient.

Good work, bad work

Why a Machine Learning Scientist owning a metric is good thing

You have positioned yourself at Ƞ = 0 by having built and deployed a fraud detection machine learning model. Using the ROI formula this makes your job simple: Optimise impact (💰) and minimise time spent (⌛️). If you do well, your boss can tell their boss that they need more headcount and easily point to what the business impact will be.

Take a step to the left

In this example the Machine Learning Scientist builds a sentiment model used by one Data Scientist. The Machine Learning Scientist has removed themselves one degree from impact. It’s less clear if this type of work is worth it. 

Now we have to consider the impact from other teams (🖇). Let’s say that there’s a 🖇 = 40% chance the Data Scientist will make a useful recommendation the product team can use and 🖇 = 50% chance this will make it onto the product team’s roadmap. Your work is now diluted down to 20%.

But where does that leave our Systems People friends who always have more steps in-between their work and the KPI? Are they less valuable?

Nope, they’re just as valuable but their sway is somewhere else.

Why an Analytics Engineer working on few end users may be problematic

You’re an Analytics Engineer and make a data model easier to use. This data model is used by one Data Scientist and they are 🖇 = 10% faster because of your work. They use it to run an A/B test and make a recommendation that improves the signup flow.

So far so good. Your work has made life easier for someone. You clearly had an end user in mind and the product team is working towards a KPI - those are all good things.

But your work only impacts one person (🎳 = 1) and the time cost (⌛️) of your work is high. Is this impactful work? Maybe if you spend a week on it. If you spend a month, probably not. 

If you find yourself working in a data role where it’s often hard to gauge what the impact of your work is and the number of data consumers that benefit from your work is low think about if you’re working on the most impactful problems

How else could this Analytics Engineer have spent their time?

Analytics Engineers at scale

You’re an Analytics Engineer who makes a core data model that five data scientists (🎳 = 5) use every day 🖇 = 10% easier to use. You’ve just scaled yourself 5x compared to the previous example.

If you’re a Systems Person constantly evaluate how your work impacts downstream consumers (🖇), how many consumers you have (🎳) and how much time you spend (⌛️).

Why you should invest in your data tools

You’re a Data Engineer who makes dbt 5% faster for everyone in the data team. Now Analytics Engineers work faster and improve more data models. Data Scientists and Analysts benefit from higher quality data models and the product team can implement more ideas. 

You’ve just made life better for 5 Analytics Engineers and 15 Data Scientists. If you’ve just improved dbt performance by 5% you’ve had a massive impact. 

Does this then instead mean that Systems People always have the most impact? Not necessarily. It all comes down to the infamous equation I introduced in the beginning. If you’re an Analytics Engineer that improves a data model which increases the efficiency of 3 Analysts by 30%, that's as valuable as a Data Engineer improving the speed of dbt for 9 people by 10%. It’s obvious when presented this way but in my experience this is not always how data teams think.  

At Monzo where I work we’ve codified the behaviour of the Systems People as a core value and think everyone in the data team has a role to play. We call it Act as force multipliers.

OK, I buy the Graph Theory stuff. Now what?

Great! Here’s how you should think 

• Systems People: Focus on maximising the number of consumers of your work (🎳) and on the impact your work has on each of them ( 🖇)

• KPI People: Focus on reducing the steps between yourself and the KPI (Ƞ) and work on the highest ROI opportunities (💰)

What’s really exciting is that all the work above is part of the same equation. If KPI People get better at picking higher ROI opportunities and reduce the steps in-between themselves and the KPI, the impact of upstream work from the Systems People increases proportionally. 

Here are some practical tips to what you can do.

Now, what? Should you show the formula to your stakeholders or actually put numbers to it? Probably not. But you can use it to think about how your work can have the highest possible impact and which strings you can pull.

Remember that everyone plays a role. 

Data Scientists and Data Analysts work faster if they have high quality data and good data models. Analytics and Data Engineers have multitudes of impact if their data models are used by many.