Anticipation: Most AI Harm Begins as Drift, Not Malice

Responsible Stewardship of AI | 2 of 15

Image by Sharosh Rajeseekher on Unsplash

Most technological harm does not begin with someone deciding to cause harm.

It begins with a reasonable purpose.

A company introduces an AI system to make a process faster. At first, the system serves the purpose for which it was chosen. Then speed becomes the measure of success. What can be counted crowds out what cannot. Employees adapt to the metric. Managers trust the dashboard more than the experience beneath it.

Each adjustment appears defensible. No one meeting marks the moment when the system’s purpose changes.

But it changes.

Efficiency, once a means, becomes the end. The organization is no longer using the system to serve its purpose. It is quietly redefining its purpose around what the system can optimize.

This is drift. And most technological harm begins as drift, not malice.

In complex systems, harmful outcomes often accumulate through reasonable choices. A data field is added. A threshold is adjusted. An interface makes one option easier than another. By the time the damage becomes unmistakable, feedback loops have strengthened it, and responsibility has scattered across dozens of decisions.

The first dimension of the ASSUME Model is Anticipation: seeing how actions ripple forward and detecting drift before it becomes damage.

Anticipation is not prediction. Prediction seeks confidence about what will happen. Anticipation prepares us to notice what may be beginning to happen. It asks leaders to attend to the future forming inside present decisions.

Before deployment, a responsible team asks not only whether a system will work, but what it could become if it succeeds too well. What patterns might it reinforce? What will the organization begin to value because the system can measure it? What could slowly disappear from view?

After deployment, leaders must keep asking whether the system still serves the purpose they intended—or whether the metric has become a substitute for that purpose.

Fast systems require slow thinking somewhere within them.

That may mean identifying early signs that assumptions are failing, listening to people who experience different consequences, and scheduling pauses to examine whether apparent success is carrying the organization away from its stated purpose.

Anticipation does not give us control. It gives us a disciplined way to remain answerable before hindsight makes the harm obvious.

The Monday Morning Question

How will we know if what we are building ceases to serve the purpose for which we built it?

If an organization cannot answer that before deployment—and keep answering it afterward—it is not anticipating. It is merely waiting for something to happen.

                           *************************************

This series draws upon my book, AI and the Crisis of Control. 

#ResponsibleStewardship #ResponsibleAI #ComplexResponsibility #AIGovernance