Validating models with experience and intuition.
Years ago when I had just began my foray into data analytics, I worked in the Circulation Department at The Seattle Times Company. This was when our online presence and digital content was still an experiment code-named “New Media” – and before the rise of Craigslist which decimated advertising revenue for newspapers. That was long, long time ago. Anyway…one of my favorite and most challenging projects was to build a forecast model that would ingest our annual sales plans, apply any channel-specific promotion codes and terms, calculate weekly retention rates by product type for both existing and new subscribers, to predicted subscriber average for the Publisher’s Statement. Publisher’s Statements were traditionally done in March in September, and served as key data points for defining and selling the audience reach a publisher has to advertiser’s, and setting advertising rates. Having a Sales Plan and retention programs that deliver the highest Publisher’s Statement possible allows for higher rates that generate more Advertising revenue.
My first few data model passes were highly manual, with lots of copy/paste actions and tweaking of lookup formulae. I’d run my (very manual) ETL (extract, transform, load) processes to update the back sheets in the model, then compare the projected Publisher’s Statement numbers from the model with the forecasts built by others on the team for sense-checking and validation. While I was building out my data model full of algorithms and links, two key department heads were plotting their projected results, leaning on over 40 years of combined experience and intuition. As we refined our assumptions, inputs, and projections, each iteration brought us closer and closer to alignment, until finally we had a forecast model we could use to compare sales and retention plans that allowed us to optimize budgets across channels.
This project taught me so much about becoming a data-driven team:
- Even the best data models and processes need to validated by someone with the experience and knowledge in the business to interrogate assumptions
- The most sophisticated algorithms can still be wrong – don’t forget to sense-check! (Trending analysis and visualizations can help with identifying anomalies, recognize seasonality considerations, and spot check expected results)
- When looking to optimize the most effective Sales Channels, the data model might suggest cutting low-performing efforts, but, experience knows those other channels target audiences that might not otherwise be reached (eg, Telemarketing may have been the most efficient at the time, but, subscribers that might never answer the phone would respond to a direct mail offer.)
- Comparing my model to nearly 40 years of experience made it so much better!
While intuition can be surprisingly accurate, and the “educated guess” approach had served the business effectively for decades, it’s nearly impossible to scale a process dependent on a couple of individuals with 40+ years of experience. With a working model in hand, we were able to run “what-if” scenarios to optimize the sales plan across channels, products, and promotion offers. And, when the inevitable requests from the business came to cut budgets, we were able to forecast what impact that would have to Publisher’s Statements, and, subsequent revenue. This helped deflect attempts to “cut our way to success” – because, really, has that ever worked?
Shout out to the team still at The Seattle Times, and all of their inspiration, support, and successes!
