Caught in the Middle: Life as an Analyst in 2021
In just a few years analysts have gone from being focused on business problems to also being expected to know about engineering best practices.
Should analysts be technical and build data pipelines or should they focus on answering business questions? When I speak to data leaders, data roles are converging around data engineer, analytics engineer, data analyst, data scientist and machine learning engineer.
One role still means different things to different people. The analyst.
In some companies analysts work end-end, build data models in dbt, write tests and monitor jobs in Airflow.
In other companies, analysts solely focus on business problems, dashboards and insights while analytics engineers are responsible for data models.
Life as an analyst in 2021
We expect much more from analysts today than we did just five years ago. As companies become more data driven, we rely on analysts to tell us what to do (sometimes too much). At the same time, data is more complex and comes from dozens of systems which has led to data teams starting to adopt engineering best practices such as testing, version control and CI/CD.
If you’re an analyst this means that you’re often caught in the middle. You’re expected to both have a good understanding of how the business works as well as a deep technical understanding.
In other words, the data analyst role has never been harder.
Analyst in 2015
You worked in Excel, used a BI tool and perhaps wrote some SQL that you learned on www.w3schools.com and saved the scripts locally on your desktop
You worked on clearly defined OLAP cubes that were made by the BI team and if you wanted to add new columns you’d have to wait for weeks for the BI team to add them
You spent a lot of time updating spreadsheets and PDF reports for the weekly KPI review meeting. This was tedious but also helped you develop an intuition for the numbers
Analyst in 2021
Your company uses a modern data stack and you’re pulled into everything from writing tests in dbt to debugging Airflow pipelines. At the same time you’re expected to stay close to business problems
Your company uses ELT instead of ETL and all the data you’d ever dream of is easily available. But with so much data you’re struggling to stay on top of the pipelines and data quality has become a real issue. Metrics are defined all over the place and you’re drowning in Slack alerts from dbt and Airflow and don’t know which ones to pay attention to
People keep asking you to read an article about a new concept called the Data Mesh but you don’t really get what it’s about
Being an analyst in 2021 is more exciting but it’s far from perfect.
We’re risking making the data industry look too daunting to new analysts just starting out. In my experience some of the best analysts often come from unexpected places. New data tools should reduce the barriers to enter the data industry, not increase them
Should analysts own data pipelines
Maybe. If it means that they can unblock themselves and work faster, then yes. If it means that they’re removed a step further from understanding the business and spend most of their time on technical work, then no.
Reasons for having analysts own data pipelines
They’re not reliant on analytics engineers to unblock them to get the data they need
They up-skill on technical tools and develop an engineering-like way of thinking about data
They’re more likely to take responsibility for the quality of the data they work with
They can have a seat at the engineering table and be part of helping shape how new products should be designed from a data lens
Reasons for having analysts solely focus on insights, analysis and dashboards
They have more time to do investigative Sherlock Holmes like work
Analytics engineers and data engineers can do what they do best and make sure data pipelines are high quality and consistent
Some of the best analysts I’ve seen have self-taught data skills to solve specific problems they faced in a sales or operations role. If we make the barrier to enter the data space too high with complex tools and workflows this becomes increasingly unlikely
Wish list
We’re at a crossroad and while there’s been some exciting developments in the data tooling space I have three items on my data Christmas wish list
Tools to augment the data workflow. With the modern data stack, data people have to spend too long debugging what went wrong when a dbt test fails or when trying to asses the downstream impact of changing a column in a data model
One place to define metrics. dbt announced metrics at Coalesce last week. This will be the future and having metrics defined in one place will make it much easier to keep one set of metrics that’s used throughout the company
Tools that get out of the way. Data tooling should be so good that analysts spend 90% of their time on insights instead of 90% of their time on debugging data models and checking errors in dbt or Airflow
Watching talks such as last week’s Building an Open Source Data Stack with Katie from Lightdash makes me hopeful that many tools are heading in this direction.
another area that DAs are doing more and more are "DS-lite" roles where they also manage experimentation and perhaps analyses using ML methods (e.g. clustering/regression) but not productionizing the solution. DS unicorns + DA unicorns?
Grea point!
One thing I would add is that the barrier for an analyst to go down the Data Engineering rabbit hole has also lowered, causing Analysts to explore that area and, sometimes, remain there or even find comfort in the technical aspects of it.
Afterall it is much easier to explain to your stakeholder that the insight they want requires a lot of technical work to be completed than to simply say "I didn't have enough time or tenacity to get to an insight on why this business problem is happening". Not that this is the rule, but definitely causes some analysts to solve the wrong problems (making the ETL work vs solving he business problem).
Today one can make simple ETL processes with no-code or even, hire some Saas service that does it for you for 20 bucks a month. 5/10 years back, you as analyst had to go fetch those exel reports by hand because there were no such tools.
But it is indeed a great point: should analysts go down that path or should they focus on the business? And I guess each company has it's balance of where that repsonsibility lies. Plus, the way to keep them focused on what matters depends on stronger leaders to guide them in this journey (which is also getting harder and harder to find)