Data will not tell you what to do

How disappointing it is to learn that the best ideas rarely come from data after 15 years of working in the field.

Data may give you a conclusive answer that changing the color of a button from yellow to green increases the conversion rate by 0.15ppts but will tell you nothing about the other ideas that would have had ten times more impact.

If you’ve played video games, it’s similar to how you start with a map almost entirely covered in unknowns. Sure, you could keep traversing the 1% of the map you already explored, but the big reward lies beyond when you make it to Onyxia’s lair.

The best ideas are often complex and require persistence, and you may only succeed on the fifth try. Even worse, they look obvious in hindsight, making it easy for outsiders to claim, “What took you so long to get to that.” While data won’t tell you what to do, exploration with a clear goal and taking small steps as you learn may just do.

After years of trying, a fintech losing millions from fraud concludes that machine learning cannot solve the problem. But one team persists in launching experiments, the model is improved, and a user researcher realizes that fraudsters are guiding customers through the in-app warning screen. With this combination of insights, the latest iteration reduces fraud by 90%, the top win for the tech team that year. The aha moment is that it took the right combination of iterations on the machine learning model combined with a radically different design of the app screens.

A young startup concludes that email outbounding to prospects doesn’t work, having sent hundreds of emails and tried dozens of messages without any results. But they missed that only when companies are in the middle of migrating to a new data warehouse do they consider buying the role-based access solution. The aha moment is that customers only buy the software at a specific point in time.

A product manager pushes her team to focus on the signup flow after seeing that the drop-off rate has doubled and that people on the forum frequently complain about it. Her team pushes back, but she knows that a 5% improvement in the signup rate has more impact than launching the new shiny product they had in the plan for the year. The aha moment is yet to be discovered, but the problem is formulated based on intuition and data. That’s a good starting point for exploration.

In the book "The Moment of Clarity," the authors question traditional business problem-solving, made popular by the likes of McKinsey & Co. Their research shows that a nonlinear approach drawing from anthropology, sociology, philosophy, and psychology, is better at getting to the moment of clarity, ending with a “Now I see it,” a conclusion that’s hard to arrive to on a path paved with spreadsheets.

I’m no longer a believer in decision-by-spreadsheet. In a world where most work is more uncertain and complex, I’ve found the principles below to go a long way.

Intuition is underrated. If you have something you care about, whether artisan coffee places, brutalist architecture, or art from Germany’s cold gaze of the 1920s, you have an intuition for it. You can pick up on things you can’t quite express. This intuition is often lost at work, with many layers between your work and the customer. A market report summary is no substitute for using your own product. Spend time where your customers are and make your own conclusions.

Focus on the big problem first. Before jumping head-first into a new problem, find a piece of A3 paper and map out the bigger picture to see where you fit in. If a company loses $10 million on fraud but only $100,000 on faulty chargebacks, it’s 100x more impactful to figure out how to solve the former. It’s obvious in hindsight, but too often, too much time is spent without the bigger picture in mind.

Think in small bets. The best idea comes from iterations and making learnings as you go. But do it with a larger goal in mind so you’re making progress in the right direction.

Don’t let data slow you down. Running an A/B test on a website with millions of daily visitors may give you a conclusive answer in minutes, but getting the same certainty for complex buying journeys or a B2B with a few thousand customers can take months. If data is slowing you down, rely less on data.

Think in terms of one-way and two-way door decisions. In his annual shareholder newsletter, Jeff Bezos talks about decisions as either one-way or two-way door decisions. Two-way door decisions can easily be reversed. One-way door decisions cannot. 90% of decisions are two-day doors, even if you initially think they’re not. Be aware of which group a decision falls into, and move fast on two-way door decisions.

Brief is smooth, and smooth is fast. Long, convoluted documents and meetings slow everyone around you down. Don’t do this to others. Instead, be brief and precise. Make that document into a one-pager and the one-pager into a paragraph. But know that this process takes time. Jason Fried and David Heinemeier Hansson from Basecamp are masters at this. Their book, Rework, just passed one million copies sold. Just before publishing the book, they reduced it from 50,000 to 25,000 words, less than half that of a normal business book, to the great displeasing of their publisher. 

Anticlimactic as it may be, not only will data not tell you what to do, but often, no one thing will. So pack your bags and enjoy the journey through uncertainty.