Evaluating the potential lift of how-casting.

Introduction

In our previous blog post, we discussed how to move from simple forecasting to “how-casting” by using machine learning and optimization to decide the best allocation of marketing touchpoints across multiple territories and customer groups. Now that we have both a real-world plan (the Actual allocation implemented) and an optimized plan (the Optimal or Counterfactual scenario), it’s time to measure the potential lift we could have achieved if the optimal plan had been followed in the latest period.

In other words: How much ROI was left on the table? This is a fair question that may be asked prior to getting buy-in to use this approach to plan the coming period.

1. Recap: Where We Are in the Process

Below is a flow diagram illustrating our iterative approach. You’ll see a star marking “You Are Here” at the “Evaluate The Potential Lift” stage:

From top to bottom, the key steps are:

Computationally Model the Optimal Scenario – We used machine learning and linear programming to find the best allocation plan.
Evaluate The Potential Lift – Our current focus: comparing the real-world (actual) plan to the model’s optimal plan for the most recent period.
Decide Plan for Coming Period – Once we know how big the potential lift is, we can choose to use or reject the approach. Even with the decision to go ahead
Track and Measure Field Execution – In the next stage, we see if teams are adhering to the plan and measure real results.

2. Comparing Actual vs. Optimal ROI

The first visual comparison is a bar chart showing the Mean ROI (Profit ÷ Cost) for three groups of customers:

Equal: Groups that received approximately the same number of touchpoints in actual vs. optimal.
Less: Groups that received fewer touchpoints in reality than the model would have assigned.
More: Groups that received more touchpoints than the model suggests.

Red Bars (roi_actual): This is the actual ROI achieved.
Blue Bars (roi_optimal): This is the counterfactual ROI if the optimal plan had been implemented.

From left to right, you can see that when the actual allocation was less than recommended, there’s a substantial gap between the red bar and the higher blue bar. Conversely, for segments that were “over-served” (red bar is slightly higher, but not by much), you might see diminishing returns and more wasted spend. Ultimately, the highest potential lift lies where actual and optimal differ significantly.

This plot represents a view of how these evaluations can go in reality. On the equal group, the actual ROI is slightly higher than predicted optimal. You may ask, “Well if they did exactly the optimal plan, shouldn’t the ROI have been exactly the same?”. That’s a fair question, but the model is not meant to be perfectly accurate (as much as everyone would love that), but it is meant to be an improvement on the current state of things. And in this example we see that is true, with customer groups getting less touchpoints than optimal missing some solid ROI and those getting more touchpoints not seeing the ROI improvement that equal had. So we have directional evidence in this example that there is lift to be gained, but how much and how can we get more confident than this directional validation. I’m glad you asked! Let’s see below.

3. Quantifying the Lift with Bootstrap Distributions

Bar charts are helpful, but we can go a step deeper to see the distribution of differences. Below are two histograms generated via bootstrapping at the national level:

Distribution of ROI Differences (Counterfactual – Actual)
Distribution of ROI Ratios (Counterfactual ÷ Actual)

3.1 Difference in ROI

The left chart shows how much more (or less) ROI we could have gained under the optimal plan. The Mean Diff = 0.352 indicates that, at the national level there is potential to gain about 0.35 points of ROI under the optimal scenario.

3.2 Ratio of ROI

The right chart illustrates the Counterfactual ÷ Actual ratio. A Mean Ratio of ~1.086 means the optimal plan delivers about an 8.6% higher ROI compared to the actual plan, on average. That’s a sizable improvement—especially if you consider the scale of marketing budgets across multiple territories.

4. What Does This Mean for Your Business?

Identify High-Impact Opportunities: These charts reveal which segments have the largest potential upside (groups currently under-served or misallocated).
Fine-Tune Resource Allocation: If you have limited budgets, funnel them toward the segments that offer the greatest marginal returns on additional touchpoints.
Assess Risk vs. Reward: The bootstrap distributions show not only the mean difference, but also the shape of possible outcomes (e.g., some groups may have very high upside but also higher variance).
Deeper Dives: You may be curious

5. From Insight to Action: Next Steps

5.1 Decide the Plan for the Coming Period

Armed with the knowledge of “what could have been,” you can now make a more precise plan for the next cycle.

Focus on segments with the highest expected lift.
Re-examine any over-served segments to reallocate budget or resources effectively.
Consider the effects of budgetary and resource changes.

5.2 Track and Measure Field Execution

Even the best plan fails if it’s not implemented correctly. In the next stage, we’ll close the loop by tracking:

Whether the field teams followed the revised plan
Any changes in ROI after adjusting touchpoints

By systematically measuring these outcomes, you can refine and improve the model for each iteration—true test–learn–optimize in action.

Conclusion: Embracing How-Casting to Realize the Full Potential

The difference between Actual and Optimal scenarios underscores the value of a data-driven approach. By:

Modeling Profit directly (instead of just sales),
Applying Optimization (linear programming or mixed-integer programming),
Visualizing Potential Lift through bootstrap analyses,

You get a transparent glimpse of how much ROI was missed—and how to capture it next time around.

Stay tuned for our final blog in this series, where we’ll discuss tracking the execution in the field and measuring real impact once the new plan goes live.

James Young