Tokopedia’s Promo Optimization with Machine Learning engine — POML — decided which buyers received which coupons. It was reported on by the conversion rate of the buyers it selected, and that rate was healthy.
But an engine that targets well will select buyers who were more likely to convert in the first place. That is the entire point of targeting. It also means the engine’s own success metric is contaminated by the thing that makes it good: the better it selects, the more credit it takes for conversion it did not cause.
I ran the post-distribution analysis — the measurement layer the engine could not provide about itself. I did not build the targeting model.
Compare targeted buyers against comparable untargeted ones, and treat the difference as the engine’s real contribution rather than treating targeted conversion as the contribution.
The comparison has to survive the objection that the two groups were never alike to begin with. Targeting is deliberately non-random, so the untargeted group is not a control group in any useful sense until you make it one.
Validated savings of $XX million a year — and, more durably, a measurement standard applied to every later iteration of the engine. Subsequent versions had to beat a counterfactual, not just report their own conversion.