Experimentation as a company strategy

Running experiments changes company culture - here’s how (and why) you need to embrace them

At Monzo, our Chief Product Officer Mike often talks about the value of thinking like a growth team and steering the company through experiments. And boy, does it strike a nerve with me! 

Just like McKinsey and MBA thinking has shaped company strategy over the past 70 years, I believe experiment-based thinking will shape business over the coming years. Experiments are how the best companies will operate in the future.

What kind of ship are you currently sailing? 🚤

Mike starts almost every presentation with a picture of a tanker. A tanker moves slowly, it doesn't turn well, it takes up a lot of resources and is generally heavy and expensive to run. 

If your company is operating like a tanker: 

• It sets annual plans by committee

• It only takes turns when results come in every 6 months

• It probably has some expensive consultants working with it on setting the course

• The captain plans months in advance

• Only the captain can steer the ship

If you want to build a company that is designed for the past, build a tanker. But if you want to build a company for the future, you need to be sailing a speedboat.

If your company is operating like a speedboat: 

• You take quick turns as you learn through experiments what works and what doesn't

• You review progress every other week

• You give autonomy to each team to set their own direction

• You're not afraid to take a turn when you make new learnings

• It’s fun because every person on your team can see what's going on and feel the movement 

• Anyone has a say in where the boat is going

More importantly, in a speedboat, these qualities start to make up the culture of the company and how everyone operates: 

• Everyone is driving business impact. Almost everything is run as an experiment and the experiment is always tied to the most important metric. This could be revenue, the number of users, cost or some other top KPI. Because experiments are scientific you will drive business impact and if you don't you will know about it. This feels very different to annual planning and looking back on individual factors you can't isolate anyway.

• Anyone can make a change. With an experiment culture, you can take a lot of small bets, learn what works and double down on these. The best part is that these bets can come from anywhere. From a support agent spotting a common customer pain point to a data analyst forming a hypothesis from looking at a dashboard to a user researcher watching users struggle in a user test.

• Your team feels a strong sense of ownership. They're owning their own roadmap and at the end of the quarter, they can look back and point to exactly what impact they had.

Sounds great, right? 

Why aren’t we all in speedboats already then?

Because getting into speedboat mode comes with its own challenges.

Specifically, leadership has to face the fact that they don't always know best and leave their teams with more of the decision power to set the direction. 

Product teams have to come to terms with the fact that a small change in the signup flow may be more important to revenue than building an entirely new and shiny product. It won't always feel great; there may be fewer lines of code, more subtle design updates and often no major public launches. 

Data teams have to be okay with at times putting their fancy toolkit and all those machine learning courses they did throughout lockdown on the shelf and instead focusing on really understanding the business.

Still - the benefits of embracing an experiment-driven mindset by far outweigh the above challenges. 

How to build an experiment culture

In an experiment culture, you'll be running dozens of experiments at any given time to constantly course-correct and double down on what works. 

An experiment is an isolated change you expose to some but not all of your users.

You may decide to show 50% of users a new signup screen with a hypothesis that users seeing the new screen will have a 5% higher signup rate. 

You keep running the experiment until you're sure of your result (also called statistical significance) at which point you know exactly what the impact of the change is. If you do it right it just works.

Here are some signs that you have an experiment culture:

• You realise that more often than not 50% of value comes from iterating and growing existing products rather than focusing on shiny new things

• Your best employees are motivated by business impact at least as much as by technical challenges

• You know that improving your signup conversion rate by 1% means that 100,000 new customers sign up each month and that this is often at least as well worth pursuing as a brand new product launch

• You know that small changes that have a big impact are cheap. They often benefit the customer directly but come with little operational overhead; you don't have to support an entirely new codebase and train customer support on a new product

• Your team knows that numbers win arguments. Everyone is empowered to come up with ideas, test them and see the impact and find that incredibly satisfying. Instead of arguing for a week if one screen is better than another they run a test and the test is now the loudest person in the room.

Where to begin building

On a practical level, here are the key areas to look at: 

• Data quality: Think about this as your compass. The single biggest pitfall of running experiments is if the data you run your experiments on is wrong or unreliable. You will be optimising for the wrong thing and in the worst case, steer in the wrong direction. I'd invest heavily and early here.

• Company culture: Create an experiment-driven culture by rewarding teams and individuals that make an impact happen through thoughtful small bets. Also, reward behaviour where teams take a bet and decide to pull back based on the result of an experiment. This is part of the learning cycle and if the process was right this is a good outcome - you may just have saved half a year of unnecessary work from being done by bringing the learnings forward.

• Leadership team: As a leadership team you have to be willing to give some of your LEGO's away. You can set the overall direction but expect your teams to come up with better experiments than you and give them the freedom to do this. You should recognise when teams are doing thoughtful experiments and approaching it with rigour and trust that they are often closer to customers problems than you are.

Invest in the right tooling for your data and engineering teams.

For data teams it shouldn't take more than a few hours to set up an experiment, they should automatically be notified when the experiment has reached the number of samples it needs to be significant and everything from confidence intervals to dashboard design should look the same each time. 

For engineers, assigning users to an experiment should be as simple as a few lines of code. Engineers should be comfortable thinking about experiments not just on user-facing tools but also for internal products and should at least have a basic understanding of how experiments work

How do you know when you’re in a speedboat?

When you’re a company where anyone can think of an idea, run an experiment, make something happen and see a direct impact. 

This may seem like a subtle change but it's kind of a big deal. 

Your teams are now obsessed with impact and driving the top metrics instead of solving interesting but non-impactful technical challenges. They operate independently with a stronger sense of ownership. 

You are able to take 20 turns per month and will see your top metrics move in the right direction faster than you did before. And your team will be happy about it and come up with ideas you would never have thought of.

If you do this right you'll be operating a team of dozens or hundreds of speedboats that all take a lot of turns but move in the same direction.