Is there a human in the loop

Why the cost of data issues increase tenfold without human intuition

As part of building Synq, a common question that comes up is what’s the impact of bad data.

Is it a loss of trust as stakeholders get increasingly frustrated?

Is it incorrect decision-making from dashboards showing wrong data? 

Or is it lost revenue from a machine learning model making inaccurate predictions? 

The answer is, more often than not, it depends.

The obvious answer is that it depends on the cost of data issues. But there’s a more interesting question—is there a human in the loop?

If data is used in a BI tool, for analytics, or sent as a PDF report, there’s a human in the loop. Human intuition tends to be strong. If a number changes unexpectedly or there’s a spike in a chart, people notice. Especially if the data is core to their job performance, such as a KPI they’re measured on.

At Google, my team pushed business unit leaders to understand and look at their dashboards each week and not just rely on analysts. This helped them spot when something is off and build a mental model for how they expect a metric to behave at any given time.

While having humans in the loop is no excuse for not testing your data, it provides a surprisingly strong second line of defense for catching issues.

But if there’s no human in the loop, it’s different. Consider a machine learning system relying on dozens of features to decide if a new customer is too risky to sign up for a scaling fintech – arguably one of the company's most important decisions. The output of the model may have a human in the loop, but it’s near impossible to use intuition to spot if an upstream data issue is causing a feature to have missing values, skewing the predictions. Worse, these issues can persist for months or even years without being caught. 

These issues can cost companies millions.

Today, most data teams' use cases still have humans in the loop. But this is starting to change as the data warehouse moves from important to business-critical, taking a central place at the core of companies’ operations.

So buckle up and brace yourself for a future where angry stakeholders complaining about inaccurate metrics are replaced with a meeting invite titled ‘Critical Data Issue: Urgent CEO Briefing.’