23.02 2026 · 8 min read

Part 2: Knowledge in the Loop

Human-in-the-loop isn't about catching mistakes. It's about how organizations actually learn.

Most organizations treat human-in-the-loop as a safety net. A temporary measure until the AI gets good enough to work on its own. But what if that framing is wrong? What if keeping humans in the loop isn't about catching mistakes, but about how organizations actually learn?

That's the hypothesis: human-in-the-loop is not a safety mechanism. It's a knowledge strategy. And treating it as anything less means you're making automation decisions without understanding what you're really giving up.

Let's explore that.

Naive Agents

When companies deploy AI agents, they usually start the same way. They gather documentation, build knowledge bases, write detailed prompts, and feed it all into the system. The assumption is straightforward: if we give the agent enough information, it can do the job.

This is how most of us think about knowledge. It's stuff you can write down. Procedures, FAQs, decision trees, playbooks. Put it in a document, upload it, hand it off. Done.

Agents are well suited for this kind of knowledge. They can ingest enormous volumes of text, search across databases, and follow instructions precisely. And for a meaningful portion of tasks, this works.

But there's a problem. Agents can't learn from experience. Or rather, they can only do so in narrow, limited ways. They don't update their own understanding after handling a difficult case. They don't develop intuition. They don't notice that a process has quietly changed because a team reorganized last month. Every interaction starts, more or less, from the same place.

The agents are naive in that sense. But the approach of deploying them this way is also naive, because it assumes that the knowledge needed to do a job well is the kind of knowledge you can write down in advance.

It's not.

What Knowledge Actually Is

César Hidalgo makes a useful distinction in his work on economic complexity. Knowledge, he argues, doesn't live in documents or databases. It lives in the network of relationships between people, processes, and decisions.

Think about what happens when a company acquires another company's source code. IBM famously acquired codebases that, on paper, contained everything needed to maintain and extend the software. But the people who wrote that code carried context that no documentation captured: why certain architectural choices were made, which parts were fragile, what the unwritten dependencies were. The code was there. The knowledge wasn't.

Organizations work the same way. Knowledge grows through experience. A customer support team doesn't just know the answers to common questions. They know which types of customers ask which types of questions, and why. They know which policies create friction and which workarounds actually work. They know who to ask when something falls outside the normal path. They have meta-discussions in meetings about how processes are evolving. They pick up on patterns that no one explicitly taught them.

This kind of knowledge is distributed across people. A group of humans working together holds more knowledge than any individual, because each person knows their part of a larger process aimed at a shared outcome. And that knowledge is constantly changing, because the organization is constantly changing, because the customers and the market and the product are constantly changing.

Now drop an AI agent into this environment. It can handle the documented, explicit knowledge. But it can't participate in the informal learning that keeps an organization adaptive. It doesn't notice that refund requests spiked after a pricing change. It doesn't develop a feel for which customer complaints signal a deeper product issue. It doesn't learn from its mistakes unless someone teaches it.

The humans working alongside agents need to develop a tacit understanding of where the agent excels, where it fails, and what the edge cases look like. That understanding only comes from doing the work together. Agents need to be included in human knowledge networks, but since they can't learn like humans do, they need a human by their side to improve them.

A Flywheel, Not a Safety Net

This reframes what human-in-the-loop is for. It's not about reviewing the agent's work to prevent errors (though that matters too). It's about building a loop where knowledge compounds.

The real value isn't in the cases the agent handles correctly. Those are already working. The value is in the cases it gets wrong. Edge cases are where new knowledge lives. A customer request the agent misinterprets. A process exception it doesn't recognize. A subtle context it misses.

When a human catches one of these failures, corrects it, and encodes that understanding back into the system (through better prompts, updated rules, new training examples, or refined workflows), the agent gets better. But so does the human. They develop a sharper mental model of the agent's blind spots and the domain's complexities. And so does the organization, because that knowledge is now embedded in both the person and the system.

This creates a compounding loop. Tighter feedback produces better data. Better data produces stronger agents. Stronger agents surface more interesting edge cases (because the easy ones are already handled). More interesting edge cases produce deeper knowledge. And the cycle continues.

The flywheel only works if humans are actively involved. Not as passive reviewers rubber-stamping outputs, but as participants who are learning, catching, and teaching.

How (Not) to Lose Knowledge

So how do you decide where to keep humans in the loop and where to automate fully?

Ask one question: "If humans stop being involved in this process, what do we stop learning?"

If the answer is "nothing important," automate fully. Some processes are stable, well-defined, and low-stakes enough that there's little new knowledge to gain. Routing standard invoices, sending appointment reminders, formatting reports. These are fine candidates for full automation.

But if the answer involves understanding your customers better, learning about your product's weaknesses, or developing domain expertise that informs future decisions, then keep humans in the loop. Even if the agent could technically handle it alone. Even if the accuracy numbers look good enough.

Because accuracy today isn't the same as learning for tomorrow. An agent that handles 95% of cases correctly but never improves is less valuable over time than an agent that handles 90% correctly inside a system that's continuously getting smarter.

The risk of premature full automation isn't just that the agent makes mistakes. It's that the organization stops learning. The knowledge network that used to include human experience, pattern recognition, and informal adaptation goes quiet. And you won't notice what you've lost until conditions change and no one (human or agent) knows how to respond.

Humans are Learners

We started with the idea that humans are teachers for AI agents. That they bring knowledge the agent lacks and feed it into the system. That's true, but it's incomplete.

Through the flywheel, humans aren't just teaching. They're learning. Working alongside agents surfaces knowledge that wouldn't emerge from either humans or agents working alone. The agent processes volume and consistency that humans can't match. The human notices patterns, context, and meaning that the agent can't grasp. Together, they generate understanding that neither could reach independently.

This is what it means for agents to become part of an organization's knowledge network. Not as autonomous nodes that operate on their own, but as partners that extend what humans can perceive and process, while depending on humans to make sense of what they find.

Human-in-the-loop, then, isn't a compromise. It isn't a transitional phase on the way to full automation. It's a knowledge strategy. Every automation decision is a knowledge decision, whether you realize it or not. Choosing where to remove humans from a process is choosing where to stop learning.

That doesn't mean full automation won't come. For many tasks, it will. As agents become more capable, the boundary of what requires human involvement will keep moving. But it will move because humans learned enough, through the flywheel, to make the next stage of automation possible. Not because someone decided the agent's numbers looked good enough on a dashboard.