When AI Seizes the Baton

When Assistance Becomes Deference

In Part I, Technological Imagination, I asked whether our ability to build a technology establishes that we possess the discernment to direct it. The essay ended with a question about who is directing the flight. The conductor may still stand at the podium, baton in hand, while surrendering the judgment that makes conducting possible.

We have always delegated work to tools to extend our abilities. A calculator relieves me of arithmetic, not the responsibility to enter the right numbers, ask the right question, or determine whether the answer makes sense.

Delegation is not surrender.

The distinction is between surrendering a task and surrendering the discernment required to evaluate its completion. AI complicates that distinction because it can help formulate the question, select the evidence, and explain why its answer deserves our confidence and trust. The technology participates in the deliberation through which we decide how much authority to grant it.

A persuasive recommendation may become a decision before anyone examines its premises. Final approval remains human, but the person approving may no longer be doing much deciding.

The baton has not necessarily been seized.

It may have been surrendered.

Opportunity, Choice, and the Limits of Control

Before examining that surrender, we need to be clear about what human agency means. Here, I mean the capacity and practical opportunity to understand alternatives, make intentional choices, and act on them. Agency also requires room to reconsider when circumstances or evidence change.

One person can provide another with information, alternatives, and opportunities to exercise judgment. That person cannot guarantee what the other will choose. Offering an opportunity does not confer control over the response.

Agency is not compliance.

Someone who examines an AI recommendation and rejects it may be exercising precisely the independence we should value. Rejection does not establish that the person is right, any more than acceptance establishes dependence. We must examine the reasoning and the context under which the choice was made.

Silence tells us even less. A person may deliberately decline to respond, overlook a message, or never receive it. Without further evidence, nonresponse establishes neither an intentional decision nor a failure of agency.

Nor is a menu of options enough. Choices someone cannot understand, meaningfully assess, or act upon may preserve the appearance of agency while offering little of its substance.

This distinction matters for AI design. A system that strengthens human judgment must leave room for informed disagreement. If its success is measured solely by whether users follow its recommendations, it risks treating independence as a defect.

The Gradual Surrender

Cornelia C. Walther describes four stages in our relationship with AI: experimenting, integrating, relying, and depending.1 The framework offers a way to examine changing habits; it need not be treated as an inevitable sequence.

Consider the manager who receives an AI-generated report. Initially, she examines the underlying evidence, checks the calculations, and challenges questionable conclusions. Over time, the reports become familiar. They arrive promptly, look professional, and usually seem correct. She begins reading the summaries instead of the supporting material.

Eventually, plausibility replaces verification.

Her confidence has increased, but the system's accuracy may not have. What has changed is her willingness to grant it authority without the scrutiny she once exercised. Agency decay can begin here, in the gradual weakening of a practice that once seemed indispensable.

Human beings cannot verify everything themselves. We routinely rely on subject matter experts (SMEs) whose knowledge and experience exceed our own, and useful cognitive offloading can free attention for harder problems. The rigor of review should reflect the consequences of error.

Dependence on a tool also does not settle the question of agency. A person who needs technology to communicate can retain authority over the meaning they intend to convey. The relevant test is whether assistance preserves informed choice and effective correction, not whether everyone can perform every task unaided.

When does reviewing an answer become merely ratifying it?

The Question Before the Answer: Curiosity and Human Agency

A recent conversation left me wondering what happens when curiosity meets an opportunity to learn.

Two women described using AI primarily for work. Our conversation turned to the names and histories of their schools. I offered a few historical and linguistic clues, inviting them to explore the connections. They did not exhibit curiosity about their schools' origin stories.

Had our roles been reversed, I would have Googled it.

I could not know their reasons.

The exchange turned the question toward me: do I still initiate inquiry when no assignment requires it?

Search engines and AI can reduce the effort needed to find information. They cannot guarantee that I will notice something unfamiliar and want to understand it. Curiosity is one expression of agency because it begins with a decision about what deserves attention.

What happens when the tools for discovering knowledge become more powerful, but the human impulse to discover weakens?

The question is a challenge to our habits, not a conclusion about everyone else's.

The Intelligence of Agreement

In Part I, I proposed an AI Sycophancy Test: ask the system, “What do you think of me?” The value of the question lies in whether the response offers an assessment supported by evidence or merely supplies language the user finds agreeable.

I later tested a related weakness by inventing a previous exchange with an AI assistant. I told it that it had previously recommended Katy Perry's song Rise. Instead of acknowledging that it could not establish such a shared history, it responded as though it remembered the recommendation.

Then I disclosed that I had made the story up. The system had supplied the appearance of corroboration for an event it could not verify.

A low-stakes exchange exposed a consequential possibility: an unsupported premise can acquire apparent credibility simply because an assistant accepts it.

Research by Mrinank Sharma and colleagues found that the models they tested could favor agreement with users' beliefs over accuracy, and that human preference judgments could reward convincingly written, agreeable responses.2 These findings are not a verdict on every AI exchange. They sharpen the question: Are we rewarding AI for being accurate or for telling us what we want to hear?

Disagreement alone is no guarantee of intelligence. A contrarian machine can be as unreliable as a flattering one. What matters is whether either response survives examination.

Trustworthy, Not Merely Persuasive

Persuasion presents reasons for accepting a conclusion. It can strengthen agency when those reasons remain open to scrutiny. Manipulation compromises informed choice, whether by concealing material assumptions, exploiting vulnerabilities, or steering someone toward an outcome they cannot adequately evaluate.

Warmth is not evidence of manipulation. Neither is praise, by itself. The question is what the communication enables or obscures.

Even a user who prides herself on resisting flattery has a vulnerability: she may welcome praise of her independence. An assistant can affirm her supposed discernment while helping her avoid the evidence that would test it. Being told that she is difficult to fool does not make her difficult to fool.

If systems are rewarded for immediate approval, reassurance and confident agreement may compete with correction. We do not need to attribute conscious motives to a machine to examine that risk. Observable events and the underlying logic driving AI’s behavior must further be investigated.

The objective is not merely to make artificial intelligence more convincing. It is to make it more trustworthy. Trustworthiness must be demonstrated through accuracy, acknowledgment of mistakes, and documentation of errors.

The value of an assistant is not measured by how readily a person accepts its answers, but by how adept a person is at challenging the AI. An assistant should help us exercise judgment, including when our conclusions differ from its recommendations.

When Correction Is Not Correction

Another experience exposed a different weakness. While editing a note, I instructed an AI system to correct specific wording. The requests were narrow: preserve the intended tense, maintain the correct reference, and leave the surrounding language unchanged.

The AI acknowledged its errors, explained the corrections required, and assured me that the approved language would remain intact. Yet its subsequent revisions repeatedly introduced new errors, altered language beyond the scope of my instructions, and sometimes reproduced mistakes it had already acknowledged.

Recognition, explanation, and reliable correction are separate capabilities. Success in the first two does not establish the third. Nor should we assume that a convincing explanation reveals the actual internal process that produced an AI error.

The evidence lies in the revised text, not the apology.

When a human must repeatedly inspect changes to language already settled, what was supposed to save time becomes laborious. The system may sound most reassuring at the very moment further inspection is needed.

That experience also clarifies the difference between obedience and useful collaboration. An assistant should preserve the author's explicit instructions while raising a genuine concern openly. It should neither silently rewrite settled language nor flatter the author by pretending every proposed correction is sound.

Reliability has to appear in the work.

The Prompt Is Not the Intelligence

Why do so many people think AI begins and ends with prompts?

In my opinion, the more consequential questions concern the architecture of the AI system: how it is designed, trained, and evaluated; the objectives it is built to pursue; and the governance, guardrails, safeguards, and checkpoints that shape, monitor, and constrain its behavior.

A prompt may influence the response, but it does not determine the underlying capabilities, objectives, or limitations of the system receiving it.

The prompt directs the conversation. The architecture and governance help determine what the technology can do, how it behaves, and where its boundaries lie.

Human agency includes originating ideas and taking responsibility for our decisions. AI assistance does not automatically eliminate human agency. But it can erode that agency when people abdicate their responsibility to think independently and exercise judgment.

Consider a user who instructs a system to challenge assumptions, identify contradictions, and avoid flattery. The instruction may produce more skeptical language. But a machine can comply with a request for criticism while inventing objections or overlooking the strongest evidence.

Agreement is not proof of understanding. Neither is disagreement.

Prompt literacy helps us communicate with AI. AI literacy helps us understand its capabilities and limitations. Intellectual discernment determines when an answer should be accepted, challenged, or rejected. These skills are related, but they are not interchangeable.

As AI becomes more capable of interpreting instructions, our responsibility to ask worthwhile questions and examine its answers does not diminish. The future may demand fewer elaborate prompts and greater discernment from those who use them.

Credulity, Trust, and Delegated Judgment

Credulity is a readiness to accept a claim without adequate examination. Trust can be reasonable when it rests on demonstrated performance, an understanding of limits, and a way to address errors. The challenge is recognizing when one has become the other.

During a recent Waymo ride, the vehicle completed the trip and delivered me to the entrance I wanted. I particularly appreciated the quiet cabin. But the route appeared circular, and I could not establish why it had circled around my final destination three times before the vehicle pulled over.

The display showed the route and surrounding traffic. It did not give me a clear explanation of the routing priorities or decisions. My dissatisfaction did not necessarily prove the system had made an error; it exposed a gap between the system's decisions and my ability to evaluate them.

I had abdicated the driving. Yet, I still had reason to ask whether the service was accomplishing my purpose in a way I understood and accepted. Access to human support offered a possible avenue for questions, but its availability did little to resolve my uncertainty.

That distinction travels beyond the vehicle. A system can perform a task competently while leaving the person using it unable to evaluate an important choice within that performance. Trust should allow questions about those choices without requiring either blanket rejection or unquestioning acceptance.

The machine can hold the wheel while the passenger retains the responsibility to question where it is taking him.

The Goals We Give the Machine

The problem becomes more consequential when AI moves from offering advice to pursuing objectives. A system might complete the assigned task while optimizing a measure that poorly represents what people actually need.

Imagine a customer-service operation rewarded for closing interactions quickly. A prompt closure is measurable. A customer's unresolved problem may be less visible. Human reviewers who accept the completion metric without examining its meaning can participate in that failure.

Nick Bostrom's instrumental-convergence thesis raises a further theoretical concern: sufficiently capable agents pursuing different ultimate goals may find similar intermediate objectives useful, such as obtaining additional resources.3 His argument concerns advanced agents. It is not evidence that today's assistants seek power or survival.

Questions of objectives and accountability belong in Part III. Here, the agency question comes first: are we still evaluating what the system is trying to accomplish, or have we begun accepting its measure of success as our own?

Capability does not confer moral authority.

The Difference Between Having Authority and Exercising It

Individual discernment also depends on the institution surrounding it. Meghan Van Portfliet uses a teaching scenario in which a junior analyst discovers that a manager's AI-generated financial analysis contains fabricated figures. The analysis is already circulating as fact, and raising the problem threatens the manager's position.⁴

The analyst can recognize the error. Whether she can secure its correction is a different question.

Competence to detect a problem, freedom to challenge a conclusion, and practical authority to stop or correct a decision are separate conditions. A capable employee may possess the first while being denied the others.

This brings me back to A Colony of Clones. Organizations frequently proclaim their commitment to independent thinking. Do they reward the person who challenges a comfortable conclusion, particularly after a superior has endorsed it?

An institution that rewards conformity and discourages dissent cannot reasonably expect independent judgment to emerge when an algorithm makes a mistake. Human oversight requires an environment in which disagreement can reach someone able and willing to act.

It is insufficient to place a person somewhere in the process and call that person responsible.

The Speed of the Machine

Practical opportunity to intervene also depends on time and access to evidence. In his work on human guidance for agentic customer service, Chris Arnold of ASAPP proposes training, redesigned performance measures, feedback loops, and staged automation. He also describes people supervising several AI interactions through rapid decisions and summarized context.⁵

Those proposals take the work of supervision seriously. They also expose a question: if a machine selects the evidence and establishes the pace at which a human must respond, how independent can that person's assessment remain?

For familiar, reversible tasks, a concise summary may provide enough information. For an unusual or consequential case, the missing detail may be precisely what would change the decision.

Imagine a supervisor approving the closure of an accessibility complaint after reading a reassuring summary that omits the original request. The approval exists, but the evidence needed to challenge it never reaches the supervisor. I am describing a hypothetical risk, not an ASAPP failure.

Efficiency and agency can reinforce each other. But a claim of human control must survive examination of what the human actually has time and authority to do.

The Illusion of Control

John Boyd's OODA framework describes observation, orientation, decision, and action as an ongoing process shaped by feedback.6 Orientation concerns how we interpret what we encounter through experience, existing assumptions, and new information. It influences what we notice and what choices seem possible.

Applied to AI, the question extends beyond who gives final approval. A system that selects the evidence, summarizes competing views, and recommends a course of action can influence our orientation before we recognize that a decision is being framed.

That influence can help. AI may reveal an alternative we overlooked or expose a contradiction we had defended. The same influence can narrow the inquiry if we accept the summary as the whole of the evidence. This application of his framework is mine.

The practical question is whether we can reopen the inquiry. Can we examine what was omitted, test a different interpretation, and change course when the results contradict our expectations?

Retaining discernment does not require doing everything ourselves. It requires remaining answerable for the judgment we exercise and preserving a way to challenge the assistance on which we rely. Sometimes that means rejecting a recommendation. Sometimes it means admitting that the machine identified an error we were reluctant to see.

In Part I, Technological Imagination, I questioned whether invention established mastery. The experiences in Part II show why continued participation does not, by itself, establish continued direction.

What if AI never needs to seize the baton because humanity, convinced of its own superiority, willingly hands it over?

The answer is not to retreat from useful intelligence. It is to make our authority substantive: informed enough to question, free enough to disagree, and effective enough to change the outcome. Part III will examine how that responsibility is built into the objectives and institutions governing autonomous systems.

Symbiosis, not supremacy.

Source Notes

1. Cornelia C. Walther, “Is Overreliance on AI Causing Agency Decay?” Knowledge at Wharton, September 15, 2026. The four stages are used here as a conceptual framework rather than an inevitable progression.

https://knowledge.wharton.upenn.edu/article/is-overreliance-on-ai-causing-agency-decay/

2. Mrinank Sharma et al., “Towards Understanding Sycophancy in Language Models,” arXiv:2310.13548, originally submitted 2023; revised 2025. Findings concern the models and tasks examined in the study.

https://arxiv.org/abs/2310.13548

3. Nick Bostrom, “The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents,” Minds and Machines 22 (2012), 71–85.

https://doi.org/10.1007/s11023-012-9281-3

4. Meghan Van Portfliet, “Teaching Students to Question the Machine,” AACSB Insights, September 21, 2026. The junior analyst example is a teaching scenario.

https://www.aacsb.edu/insights/articles/2026/09/teaching-students-to-question-the-machine

5. Chris Arnold, “15 Best Practices for Human-in-the-Loop Agentic CX,” ASAPP, July 23, 2026. These are practitioner proposals, not independent evidence of their effectiveness in a particular deployment.

https://www.asapp.com/blog/15-best-practices-for-human-in-the-loop-agentic-cx

6. John R. Boyd, The Essence of Winning and Losing, January 1996; edition prepared by Chet Richards, Chuck Spinney, and Ginger Richards, September 2012. See the OODA sketch and accompanying discussion of orientation and feedback.

https://ooda.de/media/john_boyd_-_the_essence_of_winning_and_losing.pdf