When a forecast becomes an intervention

A prediction appears to describe something that has not happened yet: demand, attrition, accidents, illness, or default. If an organization can read the future accurately, it can direct scarce people and money to the right places. Predictive models are therefore evaluated first by accuracy. Most organizations still treat a rise in post-deployment accuracy as a clearer reading of an independent future. Once the prediction allocates staff, credit, support, or scrutiny, it is no longer independent of its subject. The more accurate frame is a decision loop: the forecast moves resources, then is judged against the world those resources helped produce. The question is not only whether the future was read correctly. It is which future became easier to occur.

The loop is entered in more than one way. Adding inventory where demand is predicted high creates more chances to buy. Staffing a district more heavily makes hardship more likely to enter the record. Removing important work from a person judged likely to leave changes that person's reasons to remain. Intervention changes the target itself. Observation changes what can be measured. Response changes human reasons. Folded into a single loop, they hide where accuracy is manufacturing its own proof.

A loop that produces its own evidence

Consider a fictional municipality that allocates social-support staff according to predicted consultation volume. Districts with many recorded consultations last year receive more staff; districts with fewer receive less, so hardship is less likely to enter the record. This is not a description of an actual program. What changes first is not hardship itself, but the capacity to discover it. When the next model learns from the new records, areas that received more support appear to contain more demand. Calling recorded consultations demand is already a decision about meaning. The problem treated in Meaning Before Automation arises here too: if that meaning stays implicit, the next model does not remove the confusion; it becomes the premise of allocation. The model reuses data partly produced by its own deployment as though it were independent evidence about the world.

On the observation path, error is not the only danger. A model can remain accurate in the world created after allocation while persistently missing needs that another allocation would have revealed. The unseen future leaves no labels. Intervention counts as a hit the outcome it helped create. Response counts as confirmation the behavior it helped induce. Fitting the future that occurred is not the same as comparing the futures that were possible.

Accuracy after the world responds

Post-deployment accuracy mixes two effects: the model represented the world well, and the world was changed to resemble the model. The second depends on whether intervention, observation, or response moved. Report only the rise in accuracy, and the institution counts its own influence as evidence of correctness. Building Companies That Learn placed the test here: an organization learns only when decision, authority, execution, and observed outcome remain connected. If the outcome that returns was produced by the allocation, that connection does not close learning. It closes a memory of intervention.

When predictions support decisions, they can change the distribution they aim to predict. That phenomenon has been formalized as performative prediction.[1] Retraining then fits the next model to a reality the previous one helped produce. Convergence to a stable prediction does not by itself establish that the resulting social state is desirable. A mathematical model of predictive policing shows how discovered crime can return as input, repeatedly sending police to the same neighborhoods regardless of the true crime rate.[2] The result cannot simply be generalized to every institution. It does make one structure clear: accuracy can count having made something visible as having read the world.

When prediction can remain close to observation

Not every prediction substantially changes the quantity it is scored against. A short-term weather forecast does not cause rain. If rainfall itself is what is scored, the intervention path can be nearly empty. The same forecast still changes whether people travel or buy, so a different target can still move on the response path. A demand forecast by one participant in a sufficiently large market may move neither the market-level target, nor what can be observed, nor anyone's reasons to act. Conventional accuracy remains useful when intervention, observation, and response are all weak for the quantity being scored.

Prediction can still be useful when intervention is real, the purpose is clear, outcomes can be compared, and a mistake can be repaired. Prioritizing inspection by failure risk often meets those conditions. Allocating scarce medical resources by urgency can have a clear purpose while leaving an irreversible loss on the unseen side. That is not a reason to refuse prediction. It is a reason not to count post-allocation accuracy as observation of an independent future. Where all three paths are weak and effects are reversible, accuracy can remain central. Where effects are large and self-reinforcing, another form of evaluation is required.

Instruments after allocation

Self-reinforcing prediction requires designs that recover comparisons lost when one future is realized. Record which allocations changed because of which prediction, and measure access, discovery, and recovery alongside accuracy. Some allocation can be held independent of the prediction to search for labels that observation lost. That measurement is not free. It thins or withholds treatment for someone in order to see what the model would have missed. Where that thinner treatment is itself the harm, identifiability cannot be bought from the people held out. When withholding is not permissible, the remaining comparison is between making selection more precise and widening the chance to be selected.[3] Improving prediction is not always the best route to the objective.

Does a prediction that allocates observe the future? Only when intervention, observation, and response are all weak for the quantity being scored. If any of those paths moved, the answer is no: accuracy is a record of the decision loop, not a reading of an independent future. Human Authority in the Age of Agents is the complementary claim: authority is not preserved by an approval button. What must be accepted is not the report that the model was right, but which evidence was counted and which allocation was made irreversible. Did this future occur partly because it was predicted? And did the organization preserve room for another future to become possible?

Notes

  1. Juan Perdomo, Tijana Zrnic, Celestine Mendler-Dünner, and Moritz Hardt, “Performative Prediction,” Proceedings of Machine Learning Research 119, 2020. https://proceedings.mlr.press/v119/perdomo20a.html
  2. Danielle Ensign, Sorelle A. Friedler, Scott Neville, Carlos Scheidegger, and Suresh Venkatasubramanian, “Runaway Feedback Loops in Predictive Policing,” Proceedings of Machine Learning Research 81, 2018. https://proceedings.mlr.press/v81/ensign18a.html
  3. Juan Carlos Perdomo, “The Relative Value of Prediction in Algorithmic Decision Making,” Proceedings of Machine Learning Research 235, 2024. https://proceedings.mlr.press/v235/perdomo24a.html
Back to writing