Chapter 13

Keep the Human Upstream

Daniel received the proposal on a Tuesday morning.

The project offered him an opportunity to help develop an AI-assisted service for small businesses. The team had already built an early version. They wanted someone to examine the workflow, identify weaknesses, and recommend how the system could be improved before a wider release.

The work interested him.

It also presented a problem.

The system was designed to help business owners make decisions. It could analyse customer conversations, compare proposed actions, estimate likely costs, and recommend what a business should do next. Its developers believed that the more decisions it could handle automatically, the more valuable the product would become.

Their presentation contained a phrase Daniel kept returning to:

Remove unnecessary human intervention.

At first, the phrase sounded reasonable. Repetitive work could be automated. Routine calculations did not need to be performed manually. A system could organise information and identify patterns faster than a person working alone.

But Daniel wanted to know what the team meant by unnecessary.

Was it unnecessary for a human to copy information from one document into another?

Or was it unnecessary for a human to understand why the system was recommending a particular action?

Those were different questions.

During the meeting, one of the developers demonstrated the product using a small business scenario.

The system examined a customer complaint, compared it with previous conversations, identified the likely cause, and proposed a response. It also recommended a discount to retain the customer.

The demonstration was smooth. The reasoning appeared coherent. The proposed response was professional.

Then Daniel asked a question.

“What if the system is wrong about the cause of the complaint?”

The developer explained that the model had been trained to recognise common patterns and that the recommendation could be reviewed.

“Reviewed by whom?”

“The business owner, if necessary.”

“What determines whether review is necessary?”

The developer paused.

The system had confidence indicators, but the team had not established a reliable rule connecting those indicators to the consequences of a wrong decision.

Daniel continued.

“What if the complaint involves a safety issue? What if the discount exceeds the business owner's authority? What if the customer is describing a problem that has never appeared in the training examples?”

The room became quieter.

The product could recognise patterns, but the team still needed to decide which actions it was permitted to take, which situations required human attention, and what should happen when the available information was insufficient.

Daniel understood the appeal of automation. He also understood the danger of treating a reduction in human involvement as an improvement in every circumstance.

The meeting ended with the team asking him to submit recommendations.

On his way home, he thought about the distinction between helping a human make a decision and quietly taking control of the decision.

The more capable a system became, the easier it was to confuse those two functions.

---

The Position of the Human

That evening, Daniel met Samuel.

He described the proposal and asked what Samuel thought.

“If the system can make the decision more accurately than a person, why shouldn't it make the decision?” Samuel asked.

“Which decision?”

“Any decision where it has better information.”

Daniel considered the question.

“Better information about what?”

Samuel smiled. “You're doing it again.”

“Doing what?”

“Taking one question and opening five more.”

Daniel laughed, but he did not withdraw the question.

Suppose a system could calculate the cheapest delivery route more accurately than a person. It might be reasonable to let it recommend or automatically select a route within agreed constraints.

But suppose the cheapest route involved a level of delay that would break a promise made to a customer. The calculation would still be correct. The decision would depend on what the business valued and what commitments it had made.

Or suppose the system recommended rejecting a customer because the expected profit was low. The financial calculation might be sound, but the business owner might have obligations or principles that were not captured by the model.

Better information could improve a decision without determining every value that belonged in it.

Samuel nodded slowly.

“So the human needs to define the boundaries.”

“Yes. And those boundaries have to mean something in practice.”

Daniel explained that a person could not preserve genuine control merely by being allowed to approve a recommendation at the end. If the system selected the objective, decided which evidence mattered, determined which risks were acceptable, and narrowed the available options to one preferred answer, the human might have very little meaningful authority left.

The final click could become a ceremony.

The real decision would already have been made upstream.

Samuel leaned forward.

“Then where should the human sit?”

Daniel picked up his notebook.

“Before the system decides what to optimise.”

He wrote three words:

Purpose. Boundaries. Accountability.

The human needed to establish what the system was being used to achieve, which constraints it had to respect, and who remained responsible for the consequences of its use.

The council could help examine those choices. It could reveal contradictions, compare alternatives, and expose the consequences of different priorities.

But the person or organisation deploying it had to own the governing decisions.

This was what Daniel meant by keeping the human upstream.

It did not mean placing a human in the path of every calculation. It meant ensuring that the system's objectives, permissions, and limits remained connected to human judgment rather than emerging unnoticed from the machinery.

---

The Price of a Convenient Answer

A few days later, Daniel returned to the developers with a proposed revision.

He suggested that the system classify actions by their consequences rather than treating every recommendation as equivalent.

Routine, reversible actions could be automated within clearly defined limits.

Actions involving meaningful financial exposure, customer commitments, or significant operational changes would require additional checks.

Decisions involving serious harm, legal obligations, sensitive information, or consequences that could not easily be reversed would need stronger safeguards and appropriate human oversight.

The team listened.

One developer raised a concern.

“If we keep asking the user to review things, the product becomes slower. People want automation because they don't want to think about every detail.”

Daniel agreed that excessive review could make a system frustrating. He was not proposing that every small action required a meeting.

“The question isn't whether a human must approve everything,” he said. “The question is which decisions the system is allowed to make, under what conditions, and with what evidence.”

He gave an example.

A business might allow the system to issue a small, predefined discount when a straightforward service failure had been confirmed. But the same system should not automatically offer a large financial concession simply because its analysis predicted that doing so might preserve the relationship.

The second action involved greater exposure. It required different authority.

Likewise, the system might automatically categorise a routine customer message. But if the message suggested a serious complaint, a safety problem, or a dispute with legal implications, the system should recognise the need for a different process rather than forcing the situation into its usual workflow.

The goal was not to eliminate automation.

It was to define where automation was appropriate.

The developers agreed to test a revised design.

Their next version included clearer action limits, explicit escalation conditions, and a record explaining why certain recommendations required approval. The system could still make routine work faster, but it no longer treated every situation as though the same degree of authority applied.

Daniel was pleased with the improvement.

Then the team asked him to examine a new feature.

It would allow the system to recommend changes to a business's pricing strategy automatically, using customer behaviour and operating data.

The feature looked promising. It could help owners notice patterns they might otherwise miss.

But the developers wanted it to do more than analyse. They wanted it to adjust prices whenever the model estimated that a change would improve revenue.

Daniel asked them what would happen if the system's revenue estimate was correct but the price change conflicted with an existing customer commitment.

The team revised the design again.

The system would have to respect agreed pricing rules, identify exceptions, and distinguish between a recommendation and an authorised action. It would also need to make clear which data supported its conclusion and which assumptions remained uncertain.

The feature could still be valuable.

It simply needed to operate inside a structure of authority rather than becoming the authority itself.

Daniel realised that this was not merely a technical question. It was a question about the relationship between intelligence and responsibility.

A capable system could expand the number of decisions a person was able to examine. It could also expand the number of decisions made without meaningful human understanding.

The difference depended partly on the architecture surrounding it.

---

The Human Who Approves Everything

Daniel thought the problem was solved until he spoke with the team again.

They had introduced a human approval screen. Whenever the system recommended a significant action, the user would see a summary and two buttons: approve or reject.

The developers considered this sufficient oversight.

Daniel tested the interface.

The system presented a recommendation to change a customer's pricing arrangement. Beneath it was a short explanation, followed by a green approval button.

Daniel asked one of the developers to imagine reviewing twenty such recommendations in a row.

“What would happen if the first fifteen were correct?” he asked.

“The user would become more confident in the system.”

“And what might happen to the sixteenth?”

The developer understood.

“The user might approve it without examining it carefully.”

Exactly.

A human approval step did not guarantee meaningful human control. If the system supplied too little evidence, if the user was overloaded, or if the interface made approval easier than examination, the process could encourage automatic acceptance.

Daniel asked for a different design.

The user should be able to see the important reasons behind a recommendation, the relevant constraints, the uncertainty that could affect the outcome, and the conditions under which the recommendation should not be followed.

For more consequential actions, the interface should make the available alternatives visible. It should not present a single recommendation as though all other choices had already been eliminated.

The system also needed to distinguish between information that was unavailable and information that had been checked but remained inconclusive.

Most importantly, the person reviewing the action needed enough time, knowledge, and authority to reject it or request further examination.

A person who could not understand the recommendation, could not change its conditions, and would suffer consequences for delaying it was not exercising much meaningful control.

The developers redesigned the review process.

Now the user could inspect the basis of the recommendation, request additional analysis, compare alternatives, or send the case for specialist review. The system recorded which conditions justified automatic action and which required a human decision.

The change was not perfect. Users could still approve poor recommendations. They could ignore warnings or misunderstand the evidence.

But the system no longer relied on a button alone to represent human sovereignty.

It created a more realistic opportunity for human judgment to influence the outcome.

Daniel began to see a principle that applied beyond AI.

Authority is meaningful only when the person holding it can understand the decision, exercise judgment, and act on that judgment.

---

When the Human Is the Weakest Link

One afternoon, Marcus challenged Daniel's position.

“You keep saying the human must remain in control,” Marcus said. “But people make mistakes. People are emotional. People ignore evidence. Sometimes they make worse decisions than a machine.”

Daniel nodded.

“That is true.”

“Then why insist on keeping them upstream?”

“Because human authority and human accuracy are not the same thing.”

Marcus frowned.

Daniel continued.

“A person can be responsible for a decision and still be wrong. A machine can produce a better calculation and still be operating under the wrong objective. Neither fact removes the need to design the relationship between them carefully.”

He reminded Marcus of the early business test. The AI had helped identify useful measurements, but Marcus had to decide which risks he could afford and what kind of business he wanted to build.

Suppose the council had recommended the most profitable option. Marcus might still have rejected it because it required commitments he was unwilling to make or created a level of operational strain that did not fit his priorities.

That would not necessarily mean he had rejected intelligence. It could mean that the recommendation had optimised for only one part of the problem.

Marcus considered this.

“So the human can be wrong about the goal, too.”

“Absolutely.”

Keeping the human upstream was not a claim that human judgment was always superior. It was a claim that decisions involving human purposes and consequences required an accountable structure for determining what should be optimised, what limits must be respected, and who had the authority to act.

Human judgment itself needed examination.

The council could challenge a person's assumptions, reveal contradictions, and show how a preferred choice might undermine a stated goal. A person could change their mind after seeing that evidence.

That was not surrender.

It was learning.

The danger appeared when the human stopped examining the recommendation and treated the system's output as a substitute for the responsibility to decide.

Marcus looked down at his notebook.

“I think I understand. The human doesn't need to be right every time. The human needs to remain responsible for what the system is being asked to do and how its advice is used.”

Daniel smiled.

“That's a much better way to put it.”

---

The Decision That Could Not Be Delegated

The most difficult test came when Daniel had to choose whether to continue working with the development team.

The project had grown. The team wanted him to take on a larger role, and the compensation would be meaningful. But the next phase involved decisions about which kinds of business recommendations the system should prioritise and how much autonomy it should receive.

Daniel could see the value of the work. He could also see unresolved questions about the product's intended users, its boundaries, and how the team would handle situations where an automated recommendation conflicted with a user's stated priorities.

He asked his council to examine the offer.

The Investigator organised the terms of the role and the project's stated objectives.

The Analyst compared the workload, compensation, dependencies, and likely demands on Daniel's time.

The Challenger examined reasons to decline, including the possibility that his influence over the system's design would be smaller than the team suggested.

The Verifier checked the factual claims he could establish, including the written terms and the commitments made by the team.

The Synthesiser organised the findings and identified the unresolved questions.

The council produced a useful report.

It did not make the decision easy.

The role offered a genuine opportunity. The uncertainty was also genuine. Daniel could not establish in advance whether the team would accept all the changes he believed were necessary.

He considered what he valued.

He wanted to contribute to a system that expanded human capability without quietly transferring human authority to an automated process. He also needed to be realistic about his own time, resources, and influence.

He could accept the role and attempt to shape the project from within. He could decline it and preserve his time for other work. Or he could negotiate a smaller initial commitment, with clear responsibilities and an opportunity to review the arrangement before expanding it.

The council helped him compare those options.

But the final question was not merely which option offered the highest expected income or the lowest immediate risk.

It was whether the terms of the work gave him a reasonable opportunity to contribute in a way he could stand behind.

Daniel negotiated a limited initial engagement. He agreed to help review the next stage of the product, provided that the team documented the boundaries of automated authority, the conditions for escalation, and the evidence required before certain actions could be taken without human approval.

He did not demand that every concern be resolved before the work began. He wanted the unresolved issues to be visible, assigned, and open to review.

The team accepted the initial arrangement.

Daniel knew that acceptance did not guarantee future agreement. If the project later moved in a direction he could not support, he would have to reassess his involvement.

The council had helped him understand the decision.

It had not taken responsibility for the values that shaped it.

After the agreement was signed, Daniel returned to his notebook and wrote:

The machine can help me see the road. It cannot decide what kind of journey I am willing to make.

He read the sentence again.

It was not a rejection of intelligence. It was a statement about the proper relationship between intelligence and authority.

---

Keeping the Human Upstream in Practice

Daniel's experience with the project led him to organise his approach around a few governing questions.

Before using AI for a consequential decision, he would identify the purpose the system was serving. He would make clear which conditions were fixed, which were negotiable, and which values could not be reduced to a simple calculation.

He would determine what the system was permitted to do on its own and what required additional review. He would pay attention to whether the person responsible for the outcome could understand the recommendation, question it, and refuse it without being forced into a meaningless approval process.

He would also examine the consequences of failure.

If an action was easy to reverse and carried little risk, a lighter process might be reasonable. If an action could expose someone to serious harm, create a major financial obligation, or permanently affect a person's opportunities, the system needed stronger safeguards and more appropriate forms of oversight.

The exact design would depend on the situation.

These questions did not guarantee that every decision would be good. They helped ensure that the system's growing capability did not silently expand its authority beyond what had been deliberately established.

Daniel also recognised that the principle applied to his personal use of AI.

He could ask an assistant to help him plan his work, but he still needed to decide what deserved his time.

He could ask it to compare opportunities, but he needed to understand what he was willing to risk.

He could ask it to challenge his beliefs, but he could not use its answer as an excuse to stop examining his own reasoning.

And he could ask it to recommend a course of action without pretending that a recommendation was an order.

The council existed to expand the quality of his judgment, not to make judgment unnecessary.

---

The Architect Remains

One evening, Daniel sat alone and reviewed the four chapters he had just completed.

He had begun Part IV by asking how to design a council. He had learned to match the process to the stakes, assign distinct responsibilities, and specify what a useful output should contain.

He had learned that questions shaped the investigation, that agreement between AI systems did not establish independent evidence, and that verification required more than a convincing answer.

Now he had reached the question of authority.

The pieces belonged together.

A council without structure could become a collection of disconnected opinions.

A council without disciplined questions could investigate the wrong problem.

A council without verification could turn unsupported claims into polished conclusions.

And a council without clear human authority could make all those weaknesses harder to notice because the final recommendation would appear to have emerged from a complete and intelligent process.

Daniel closed the document.

He did not believe that humans should personally perform every calculation or inspect every routine action. That would waste the very capabilities AI made available.

Nor did he believe that human decisions should be protected from challenge. A person who refused to reconsider an assumption was not exercising sovereignty merely because the final choice remained theirs.

The human needed to stay upstream in a more meaningful sense: responsible for the purpose, involved in defining the boundaries, capable of examining the evidence, and accountable for the decisions made under those conditions.

That responsibility could be shared across a team or organisation. It did not have to rest on one individual in every situation. But it could not simply disappear into a system because the system was intelligent.

Daniel returned to the manuscript's table of contents.

Part IV was complete.

He had designed the council, built an architecture for questions, established the need for independent verification, and examined the relationship between human judgment and machine capability.

The next question reached beyond his personal workflow.

What happened when more people learned to work this way? What would change when businesses, communities, and institutions began organising intelligence through councils rather than treating AI as a single source of answers?

Daniel opened the next section and typed:

PART V — THE FUTURE OF HUMAN AND MACHINE INTELLIGENCE

Then he wrote the title of Chapter 14:

The Augmented Mind.

He looked at the words for a moment.

The future would not be determined only by how intelligent machines became. It would also depend on what people learned to do with that intelligence, what they chose to delegate, and whether they could expand their capabilities without abandoning the responsibility that made those capabilities meaningful.

The council had been designed.

Now Daniel wanted to discover what kind of human being could grow through using it well.