The most dangerous forecast is the one that used to be right.

A model gets trained, it performs well, everyone moves on. Then the world shifts. A supplier changes prices, a data pipeline silently breaks, customer behavior drifts. The model keeps producing confident numbers, and the business keeps acting on them. Nobody notices, because nobody is watching the watchman.

This is the unglamorous half of machine learning. Everyone wants to build the model. Almost nobody wants to babysit it. But an unwatched model is a liability with a nice dashboard.

A vision worth building toward

For a while now I have been following the work of Graphite Note and its founder, Hrvoje Smolić. What pulled me in was the clarity of the mission to democratize decision intelligence, so that any business, not just the ones with a data-science department, can predict outcomes, understand what drives them, test actions, and choose the next best move.

One idea from his writing kept circling in my head. Prediction alone is not enough, and causal inference alone is not enough either, the future belongs to systems that close the loop between insight and action. And before you can govern automated decisions, you first have to build the intelligence underneath them. Tolerance bands. Drift monitoring. Exception handling for the models themselves. Statistical governance comes before decision governance.

It is a compelling way to see the field, and I wanted to do more than admire it from the outside. So rather than just read about the problem, I set myself a challenge to build a working piece of that vision, end to end, and measure it honestly. Two projects came out of it.

SKU Sentinel — watching the watchman

SKU Sentinel monitors a fleet of demand forecasts the way an SRE monitors servers. Every week it checks each product against reality, is accuracy inside this product's own calibrated tolerance, is there sustained bias, has the error distribution shifted, are the actuals frozen or zeroed or suddenly ten times larger, which usually means a broken pipeline rather than a demand miracle.

A quick word on what is actually doing the predicting. Each product gets its own forecasting model, a ridge regression trained on calendar and promotion features in log space, kept deliberately simple and frozen after training so I could study how a real model ages in production. The monitor itself is model-agnostic, it only needs the forecast and the actual, so it would work just as happily on top of a gradient-boosted or deep-learning forecaster. To test that claim I also built a small bench of eight forecasting models, from a seasonal naive baseline through Holt-Winters, Random Forest, Gradient Boosting, XGBoost and LightGBM, up to a PyTorch LSTM, and raced them on the same data. The drift detection then leans on classic statistics, the Population Stability Index, a Kolmogorov-Smirnov test, and a Page-Hinkley sequential change detector.

Because I built the test data myself, I could inject 28 known problems at known dates and grade the monitor honestly, not by intuition.

One lesson cost me an evening of confusion and taught me the most. The textbook alarm threshold for PSI, a standard drift statistic, is 0.25. On small monitoring windows, the expected value of PSI when there is no drift at all is already around 0.4. Textbook settings had my monitor flagging almost everything.

Then I tested it on Walmart

Synthetic success is easy. So I ran the identical engine on the M5 dataset, which holds two years of real daily sales for the top 300 food products at one Walmart store in California, under three forecasting regimes. The bars show the share of products the monitor flagged. Lower is better.

That progression rewrote my assumptions. I expected retraining to fix most of it. It fixed about eight points. The dominant diagnosis across all three regimes was sustained bias, which is the statistical signature of a missing driver, not of staleness. Real demand moves for reasons a calendar cannot see, and no retraining schedule compensates for a variable the model was never given. The M5 competition reached the same conclusion by another road, since its winners all needed price and event features plus heavy models.

Meanwhile the data-quality checks caught 33 genuine anomalies in real Walmart history, stockouts and discontinuations and holiday closures, with identical counts in all three regimes, because those checks test the data rather than the model. A good monitor does not just say something is wrong. It tells you which thing is wrong, because fixing a pipeline and retraining a model are very different Tuesdays.

NextMove — forecasts are opinions, orders are money

A forecast on a dashboard changes nothing. So the second project converts forecasts and their uncertainty into order quantities using newsvendor economics. For each product it weighs two costs, what a missed sale burns in lost margin, and what a leftover unit costs to hold or to throw away when it spoils.

The part I find most satisfying is that nobody set the service levels. Detergent earned a 99% target because leftovers cost pennies. Yogurt ran lean because leftovers rot. The economics decided, product by product. And there is a compounding insight hiding in it. Better forecasts narrow the uncertainty, which shrinks safety stock automatically. Forecast accuracy converts directly into working capital, but only if the decision layer actually consumes the uncertainty instead of admiring it.

Three things I believe now

Measure your own detector. A monitoring system that has never been graded against ground truth is folklore with a confidence interval. Respect alert fatigue. A precise alarm stream a team trusts beats a sensitive one they learn to ignore. And uncertainty is not a footnote. Handled honestly, it is the most valuable number in the pipeline.

Both are open source

Full write-ups, reproducible experiments, honest limitations, and web consoles you can run in a minute.

If you work in forecasting, supply chain, or decision intelligence, I would genuinely value your critique. I am always happy to talk shop, and always building toward the next move.