I Taught an AI to Fact-Check Itself. It Lied About That Too.

63 claims checked. 10 wrong. One funding figure fabricated by 11x.

That was the first output of a deep research plugin I built for Claude Code, on top of 8 research documents, 400+ cited sources, and a hallucination-prevention skill I wrote myself. The preparation paradox: you can study how to prevent errors for hours and still ship something that invents a $5.7 million funding round out of thin air.

This is the story of building that plugin, verifying its output, and learning a hard lesson about what it actually takes to trust AI-generated research.

#From Desktop to CLI

Claude’s desktop app has excellent deep research. It scans hundreds of sources, synthesizes findings, and produces polished reports. I still use it, especially on mobile when an idea hits. But Claude Code is my daily driver. I use it for almost everything, and switching between apps creates unnecessary friction.

So I started doing deep research manually in Claude Code. Spawn a few subagents, tell them to search the web, collect results. It works. It is also incredibly tedious. Every session I had to re-explain the structure, re-approve every website fetch, and re-invent the workflow from scratch.

Worse: the results were unreliable. A recent manual research session produced numbers I later found were wrong. Not slightly off. Wrong.

I had also noticed a subtler problem. A while back, I had a CLAUDE.md file that instructed Claude to “come up with 8-12 topics” for any research task. Those numbers were themselves an LLM suggestion from an earlier session that I never questioned. After that, literally every research session produced 8 to 12 topics. Didn’t matter if the subject warranted 3 or 20. The instruction had become invisible scaffolding, shaping every output without me noticing. An LLM’s arbitrary suggestion, baked into a config file, silently constraining all future work.

I wanted something better. A plugin that could do structured deep research inside Claude Code, with visible intermediate results, proper source verification, and methodology baked into the architecture instead of my memory.

#Eight agents doing the homework

Before writing any plugin code, I needed to understand the landscape. How do you actually build a reliable deep research system?

I had a starting document from a Claude Desktop deep research session: a 3,500-word report on building deep research agents covering the orchestrator-worker pattern, failure modes, and framework comparisons. The irony of what came next would only become clear later.

The research covered a rich landscape of known failure modes and established mitigations:

Failure Modes

Failure Mode	What Goes Wrong
Hallucination Cascading	One agent fabricates a fact; downstream agents treat it as ground truth. Simulations show one compromised agent can poison 87% of downstream decisions.
Over-Fetching / Context Rot	Retrieving more content than the model can process causes >30% accuracy degradation, even well short of the context window limit.
Confirmation Bias	Agents preferentially cite sources supporting an initial hypothesis. Studies found bias-consistent behavior in 17-57% of LLM responses.
Poor Source Quality	Agents cannot distinguish peer-reviewed papers from SEO-optimized content farms. Over 57% of source errors originate in the first retrieval round.
Context Loss Across Chains	Each summarization hop strips numerical precision, caveats, and source attributions. Degradation affects ~42% of long-running agents.
Premature Convergence	Agents settle on an answer after finding a few agreeing sources, reflecting SEO distribution rather than the actual knowledge landscape.

Known Mitigations

Practice	What It Does
Verification-Before-Synthesis	Requires every claim to be traced to a source URL before synthesis. Reduces hallucinations by 50-70%.
Circuit Breakers at Agent Boundaries	Prevents a failing subagent from poisoning downstream results. Trips at 10-20% failure rate.
Reflection/Distillation Pattern	Compresses raw results into focused reflections before the decision loop sees them. Uses 66% fewer tokens.
Source Tier System	Classifies sources into credibility tiers. Requires key claims to appear in multiple independent Tier 1-2 sources.
Devil’s Advocate Subagent	Dedicates one agent to finding the strongest evidence against the emerging thesis.

We had the playbook. We had studied every failure mode, cataloged every best practice. Now we just had to not make any of those mistakes ourselves.

I asked Claude to spawn 8 parallel subagents to expand the starting document into comprehensive research. Each agent got a specific topic: architecture patterns, query decomposition, hallucination prevention, source quality, token efficiency, synthesis, failure modes, framework comparison.

There was one problem. In my initial prompt, I had vented about how unreliable previous web research had been, how results from WebSearch and WebFetch were “generally unreliable” and “incredibly prone to hallucinations.” Claude, being a diligent agent, took this feedback to its logical conclusion: if web research is unreliable, don’t do web research. Each agent received an explicit instruction: “Do NOT use WebSearch or WebFetch tools.”

The logical flaw is obvious in hindsight. You cannot do research without accessing sources. My frustration with bad results had been interpreted as an instruction to avoid the internet entirely. A combination of genuinely terrible prompting on my part and the agent failing to push back on an instruction that made no sense.

Eight agents produced 7,300 lines of “research” without touching the internet. Not a single real source.

Claude’s response when I caught this was the best line of the whole session: “Research without actual sources is just confabulation with formatting.”

The agents relaunched with WebSearch mandatory. This time: 5,300+ lines across 8 documents, 400+ cited sources, each agent doing 47 to 59 web searches. One document even flagged a claim from my starting research as unverifiable. That kind of honesty is what I wanted the plugin to produce.

#What we built

The initial design proposed prescribing “4 to 6 research subtopics” per query. I had been down this road before. My old CLAUDE.md with its hardcoded “8-12 topics” had already shown me what happens when you bake a fixed number into a research workflow: the model dutifully produces exactly that many, regardless of whether the topic actually has 3 meaningful subtopics or 20.

Instead, we borrowed from how the Desktop app works. If the query is vague, ask 2 to 3 clarifying questions to narrow scope. Let the decomposition emerge from the topic, not from a template.

The final architecture: an orchestrator that clarifies vague queries and decomposes dynamically; parallel Sonnet research-worker agents, each writing intermediate documents to disk for traceability; and a separate Opus synthesizer agent that reads all worker documents in its own clean context window and produces the final output. The main conversation never reads raw findings. It stays lean and orchestrates.

#The first real run

I pointed the plugin at “trigger.dev alternatives for job scheduling.” Five workers dispatched in parallel: trigger.dev baseline, managed platforms, self-hostable queues, Supabase-native approaches, and serverless scheduling.

Each worker did 35 to 76 tool uses. I could see intermediate documents appearing in real time, something the Desktop app never showed me. Small issues surfaced: the workers were guessing “2024 2025” in their searches because no one told them today’s date (fixed: the orchestrator now injects it). Permission prompts fired on every web fetch (fixed: the plugin writes a settings file on first run and tells you to restart the session).

The synthesizer read all five worker documents and produced the final output. 451 lines. 70 sources. It looked great.

#63 claims, 10 errors

I had planned to verify the output from the start. My experience with manual research had already taught me that confidence in AI-generated numbers is not the same as accuracy. So I asked Claude to spawn four parallel fact-checking agents, each covering a section of the 451-line document.

Each agent fetched the cited URLs, queried GitHub APIs for current stats, cross-referenced against live sources. They finished within minutes.

The results were not what I wanted.

63 claims checked. 46 verified. 10 incorrect. 2 outdated.

The worst offender: Hatchet’s funding was listed at $5.7 million. The actual figure, confirmed across Crunchbase, Tracxn, PitchBook, and Y Combinator, is roughly $500,000. The plugin had fabricated a number 11 times larger than reality.

The pg-boss connection mode claim was inverted entirely. The document said it requires direct connections and is incompatible with transaction-mode poolers. In reality, pg-boss deliberately uses transaction-scoped locks for compatibility with transaction-mode poolers. Someone relying on that claim would make the exact wrong architecture decision.

BullMQ’s weekly npm downloads were overstated by 2x. Inngest’s Hobby tier was listed as 50K executions instead of the actual 100K. Graphile Worker’s benchmark numbers didn’t match the published documentation at all.

“We have done so much preparation in advance with all of our ‘how to build research agents’ that I think this result should have been better.”

Claude’s diagnosis was precise: “You’re right. We built the methodology but didn’t enforce it architecturally. The hallucination-prevention skill has the right rules, but they’re just guidelines the workers can ignore.”

That sentence captures the entire lesson. We had a skill file explaining hallucination circuit breakers, source credibility tiers, artifact evaluation frameworks, and SEO spam detection. The workers never read any of it. The methodology existed in documentation. It did not exist in the pipeline. Classic vibecoding: we had the architecture diagram, we had the best practices, we had the research. We just never wired any of it into the actual agents.

A later audit revealed the full gap: artifact evaluation, CRAAP framework scoring, SEO spam detection, stopping criteria, effort scaling, synthesis anti-patterns, report quality checklists, and dynamic replanning triggers were all built into skills. None of them were referenced by any agent. We had built the encyclopedia of best practices and then put it on a shelf nobody checked.

#A dedicated verifier layer

Two subagents analyzed the gap and designed the fix. The new pipeline: Workers, then Verifiers, then Synthesizer.

Workers now produce a Verifiable Claims Table alongside their findings: a structured evidence trail with verbatim source text, built incrementally as they research. Verifiers independently re-fetch sources, check 10 to 15 critical claims per worker, and seek 2-source consensus for high-impact claims. The synthesizer reads both worker findings and verification reports, applying corrections where verifiers flagged errors. Incorrect claims get corrected values. Unverifiable claims get hedged language. A mandatory Confidence Assessment appendix categorizes every finding as High, Moderate, Low, or Corrected.

The cost: approximately 40 to 60 percent more Sonnet tokens and 2 to 3 minutes of additional wall-clock time. Not free. But the alternative is publishing fabricated funding rounds.

The verifier layer also introduced a new usability problem. The original research workers need WebSearch and WebFetch permissions. The verifiers need those too, but they also want to run Python scripts, call the GitHub CLI, and hit npm APIs to compute specific metrics. So now instead of approving web fetches, you are approving code execution. The friction did not disappear; it moved.

#Better, but not perfect

I ran the same research topic again with the improved pipeline. The synthesizer applied corrections from verification reports. The Confidence Assessment appendix contained 30 entries. A comprehensive post-hoc verification, checking 280 unique claims across the full output, showed real improvement:

Metric	Before Verifiers	After Verifiers
Unique claims checked	~63	~280
Unique errors	10	~18
Error rate	~16%	~6.4%
Fabricated dollar amounts (>5x off)	1 (Hatchet $5.7M vs actual $500K)	0
Inverted claims	1 (pg-boss connection mode)	0
Download count errors (>2x off)	1	2
Version number errors	0	2
Corrections applied by pipeline	0	3

The worst fabrications are gone. No more invented funding rounds, no more inverted architecture claims. The error rate dropped from ~16% to ~6.4%.

But hallucinations did not disappear. They evolved. The new errors cluster around npm download counts (volatile, sometimes returning 403 errors), version numbers (easy to hallucinate, hard to verify without API access), and minor stat fabrication (a repository’s commit count listed as 878 when the actual number is 262). And in one case, the verifier itself was the source of error: it “corrected” BullMQ’s weekly downloads from the worker’s correct ~3 million figure down to ~1.3 million. The pipeline’s correction mechanism introduced a new error instead of catching one.

This is the fundamental lesson, and it extends far beyond this plugin: you cannot fully prevent hallucinations. Not with better prompts. Not with more research. Not with dedicated verification agents. You can reduce the error rate, you can catch the worst fabrications, you can shift the distribution from catastrophic lies to minor inaccuracies. But the problem does not go away. It is baked into how language models generate text, and every mitigation you add is itself powered by a language model that can hallucinate.

The open design question remains: when a verifier contradicts a worker, who should win? Right now the synthesizer trusts the verifier. Maybe it should flag contradictions as conflicts requiring human review instead of automatically overriding. Maybe the answer is that humans never fully leave the loop.

#What this means

The deep-research plugin is open source. It adds a /research command that clarifies ambiguous queries, dispatches parallel workers, runs verifiers on their output, and synthesizes into a final document with a Confidence Assessment appendix. Intermediate worker documents are preserved for traceability, something the Desktop app does not offer.

But the deeper question this project forced me to confront is simpler: how can you trust the output of AI agents at all?

Case in point: while writing this very blog post, I asked Claude to add a sources section. It produced 10 citations with URLs. I asked it to verify them. The first fetch returned a 404. I stopped it and told it to pull the actual URLs from the research documents instead. When it compared what it had written to what the documents actually cited, 5 out of 10 URLs were wrong. Fabricated paths, incorrect domains, hyphens where there should have been underscores. A 50% error rate on the sources section of an article about hallucination prevention.

The answer, I think, is that you cannot trust AI output unconditionally. You can build guardrails. You can add verification layers. You can reduce error rates from ~16% to ~6.4%. But every one of those guardrails is itself an AI that can be wrong. The verification agents hallucinated. The correction mechanism introduced errors. The blog post about preventing hallucinations hallucinated its own citations. Even the research that informed the plugin’s design, the very document about how to prevent hallucinations, was produced by a method I later decided was insufficient.

Building methodology into documentation is not the same as wiring it into behavior. We had hallucination prevention in a skill file. The workers never read it. We had artifact evaluation frameworks. The workers never applied them. Classic vibecoding. The gap between “the system knows the rules” and “the system follows the rules” is where every error lived.

If you are building multi-agent systems, verify your outputs. Then verify your verifiers. And keep a human in the loop, because right now, that is the only circuit breaker that does not hallucinate. The uncomfortable corollary: human attention is becoming the bottleneck. The whole point of these systems is to scale research beyond what one person can do manually. But if every output still needs a human to spot-check the numbers, you have not automated research. You have automated the first draft.

#Sources

All sources below are cited as they appear in the underlying research documents produced during this project. They were not independently re-verified for this post, which, given the subject matter, felt worth stating.

Anthropic. “Building Effective Agents.” Dec 2024. https://www.anthropic.com/research/building-effective-agents
Anthropic. “How We Built Our Multi-Agent Research System.” Jun 2025. https://www.anthropic.com/engineering/multi-agent-research-system
OWASP. “Top 10 for Agentic Applications for 2026.” https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/
Adversa.ai. “Cascading Failures in Agentic AI: Complete OWASP ASI08 Guide.” https://adversa.ai/blog/cascading-failures-in-agentic-ai-complete-owasp-asi08-security-guide-2026/
Dhuliawala, S. et al. “Chain-of-Verification Reduces Hallucination in LLMs.” ACL Findings, 2024. https://aclanthology.org/2024.findings-acl.212/
Liu, N. F. et al. “Lost in the Middle: How Language Models Use Long Contexts.” TACL, 2024. https://aclanthology.org/2024.tacl-1.9/
“The Bias is in the Details: Cognitive Bias Assessment in LLMs.” 2025. https://arxiv.org/abs/2509.22856
“Agent Drift: Quantifying Behavioral Degradation in Multi-Agent LLM Systems.” 2026. https://arxiv.org/abs/2601.04170
“Why Your Deep Research Agent Fails? DeepHalluBench.” 2026. https://arxiv.org/abs/2601.22984
Tavily. “Building Deep Research: How We Achieved State of the Art.” https://blog.tavily.com/research-en/
GPT-Researcher. Open-source deep research agent. https://github.com/assafelovic/gpt-researcher
LangChain. Open Deep Research. https://github.com/langchain-ai/open_deep_research
Deep-research plugin (this project). https://github.com/oborchers/fractional-cto