Trust in data…but, with a caveat.
Data munging process might be painful, but can lead to meaningful observations.
One of the biggest challenges analysts often face is validating that the data points and measures they’ve pulled from source systems are the right values to be using to analyze results and tell a performance story. Sometimes data sources may use similar naming conventions for data fields that have similar values, but, return very different trends when used to analyze results. (eg, sampled or aggregated data, cumulative or trended, etc.) This can greatly skew the accuracy of a performance story.
Often, source information systems may have underlying data table structures that require stitching tables or views together in order to summarize results for the business, requiring knowledge of database structures, programming, or, a tool that offers a user-friendly front end to build necessary connections.
As a result, analysts often spend a considerable amount of time sourcing, configuring, and remapping data just to reach a point where they’re ready to analyze, craft narratives, extract insights, and make recommendations.
Early in my career, I saw this as a problem. I was frustrated by spending 70% of my time on data wrangling, leaving only 30% for generating meaningful insights. It felt like a waste of resources, especially when technology could address much of it. While technology certainly has the potential to shift the balance from 70/30 to 30/70, over time I’ve come to realize that the data-cleaning process is invaluable for shaping insights.
Understanding the underlying data and how to interpret it requires an understanding of the probable expected results, and a way to validate those results. Knowing how source systems are collecting data is a peek into a their business maturity. And, comparing discovery or automation outcomes to pre-existing manual processes can reveal a lot about pain points, and, priorities. Still, it can take years of practice to recognize what might be a wrong measure being pulled in to a data set, how to interrogate the data through comparisons, visualizations, and trend analysis, and frankly, the application of a strong dose of common sense. “Does this result make sense? Do I know what to expect? What if…?”
Asking, and then learning how to answer those questions, is the fire and motivation that drives many analysts to discover, curate, and come with an “aha” that delights, surprises, informs, and makes a difference. There’s nothing quite so satisfying as bringing a data-driven insight that truly empowers the business.
Wishing every analyst and data enthusiast an “aha” moment!
