Six Months of Running AI Marketing Agents 24/7. Here's What Actually Broke.

The agents have been running for six months now. Eight of them. Across three websites. Generating briefs, writing drafts, rendering images, pinging search engines. Nobody hits a button. They wake up on a schedule and do their work.

This is not a "how to build agents" post. Those are everywhere. This is what I actually learned after letting them run.

The hard part was never the code. It was never the prompts either. The hard part was knowing whether the agents were doing good work or just doing work.

I am not alone in finding this hard. The LangChain State of Agent Engineering survey, with 1,300 respondents, found that quality is the number one production barrier, cited by 32 percent of teams. Not cost. Not latency. Quality. The agents produce output, but is it the right output? That question is harder than any technical problem in the stack.

The Metric That Lies

Every agent platform shows you a dashboard. Tasks completed. Tokens consumed. Posts published. These numbers go up and to the right and everyone feels good.

But "posts published" is a vanity metric when you are the one who has to read them.

In month one, our content agent shipped a post that was technically flawless. Correct word count. Proper schema. Images rendered. Video attached. It also argued the exact opposite of something I had written two weeks earlier. Nobody caught it until I read it.

The agent did its job. It just did it wrong.

This pattern repeated across evaluation agents, research agents, and even the QA agent that was supposed to catch the other agents. Output volume went up. Trust in the output went sideways.

This is a specific instance of a broader pattern. A 2026 analysis of 847 AI agent deployments by Digital Applied found that 88 percent of agent projects fail before reaching production. When they analyzed the postmortems, 94 percent of failures clustered into 7 patterns. The top two, scope creep at 34 percent and data quality failures at 27 percent, are both upstream of the agent itself. The agent was running fine. The question it was answering was wrong, or the data feeding it was bad.

I saw the same thing at a smaller scale. My agents were running fine. The problem was that I had no way to tell whether the work they were producing was any good.

The Eval Stack I Had to Build

After a few of these incidents, I stopped celebrating task counts and built an evaluation stack. It has four layers.

Layer one: deterministic checks. Before anything ships, code verifies the floor. No em dashes in copy. Schema present. Image files exist at the right paths. Video rendered and playable. These checks catch about 30 percent of failures and cost almost nothing.

The key insight here is that deterministic checks catch the failures you can predict. They miss the ones you did not think of. That is why you cannot stop here.

In practice, I built these as a series of assertion functions that run in the agent's deployment pipeline. Each check returns pass or fail. If any check fails, the agent output does not advance to the next stage. The total cost of running all checks is about 200 milliseconds per agent run. There is no excuse to skip this layer.

Layer two: trace review. Every agent run produces a trace. What inputs it received. What tools it called. What it decided. I review a sample of traces weekly. Not all of them. Ten to fifteen per agent. This is where you find the "technically correct but wrong" failures that deterministic checks miss.

The LangChain survey found that 62 percent of teams with agents in production have detailed tracing that lets them inspect individual reasoning steps. Among those with production agents, 71.5 percent have full tracing. I fall into this category now, but I did not start there. The traces existed from day one because the agent framework produces them automatically. What had to change was the habit of reading them.

The trace review workflow takes me about 30 minutes per week. I open the trace viewer, pick the most recent run for each agent, and scan the reasoning chain. I look for three things: did the agent use the right tools, did it interpret the tool output correctly, and did it make a reasonable decision given the inputs. Most runs pass. The ones that do not tell me something about the prompt, the data, or the context that needs fixing.

Layer three: human override gates. For content that reaches customers, a human still approves. The agent drafts. The human reads. This is not because the agent is unreliable. It is because the cost of being wrong on a customer facing page is higher than the cost of the five minutes it takes to read a draft.

This is where a lot of agent vendors will tell you that you are doing it wrong. "The agent can handle it," they say. "You are limiting its potential."

They are selling software. I am running a business. A single wrong email to a 4,000-person list costs more than the agent's annual operating budget. Five minutes of human review is cheap insurance.

The trick is narrowing the human gate to exactly the points where being wrong hurts. I do not review every research brief. I review every draft that gets published. I do not review every image generated. I review every image that goes on a public page. The gate sits at the boundary between internal and external. Everything inside runs on checks. Everything outside gets a human read.

Layer four: post deployment monitoring. After content goes live, I watch for signals. Did the page get indexed? Did anyone link to it? Did it generate any inbound traffic? These signals close the loop between "we shipped" and "it worked."

Post-deployment monitoring is the layer most teams skip because the feedback loop is slow. A page can take days to index. Traffic data takes weeks to stabilize. But without this layer, you never know if the agents are producing work that actually matters.

I check this monthly. The signal I care about most is not page views. It is whether anyone linked to the content organically. If agents are producing work that earns backlinks, that is a signal that the quality is there. If nothing is linking to it, that is a signal that the work is technically fine but adds nothing new.

The stack is simple. But nobody tells you to build it. The agent vendors sell you on "set it and forget it." That is a lie for anything customer facing.

Three Failure Patterns I Did Not See Coming

After six months, three patterns emerged that were not obvious at the start.

The agent learns my slop. If I write a bad prompt, the agent amplifies it. Vague instructions produce vague output. This sounds obvious, but it compounds. A slightly weak research brief becomes a mediocre draft becomes a forgettable published post. The failure is not at the final step. It is in the brief. The agent faithfully executes the weak signal.

This maps directly to the data quality failure pattern identified in the broader industry research. When the input to an agent is incomplete or inconsistent, the agent chains incorrect conclusions. The error propagation multiplier is high. A 5 percent ambiguity in the brief produces a 30 percent drop in output quality.

The fix is brutally simple. I write better briefs. Not longer ones. More specific ones. Instead of "research the competitive landscape," I write "find three competitors who launched a similar feature in the last 90 days and describe what happened after launch." The agent cannot read my mind. It can read a specific instruction.

Agents drift when left alone. The prompt you wrote in January will not produce the same output in June. Models change. APIs update. New data patterns emerge. Without periodic recalibration, agents slowly wander off target. I now run a calibration check every two weeks on every active agent. Compare recent output against a golden set. If something shifted, fix it before the drift becomes visible to customers.

The Digital Applied research on production patterns found that agent drift is underreported because it looks like gradual quality degradation, not a sudden failure. If the agent was producing 4-out-of-5 work and now produces 3-out-of-5, nobody notices for the first week. By week four, the output is noticeably worse but the team has normalized to it.

The calibration check takes 15 minutes. I keep a file with five examples of what good output looks like for each agent. Every two weeks, I run the agent on a test case and compare the output to the golden set. If the match rate drops below 80 percent, I investigate. Usually the fix is a prompt update or a data refresh. Sometimes it is a model update that changed the agent's behavior.

Coordination failures are more common than individual failures. When eight agents hand off work to each other, the failure is rarely "agent four produced garbage." It is "agent two passed incomplete context to agent five, which made a reasonable but wrong decision." The most valuable debugging technique is not checking individual agent output. It is tracing the handoff.

The broader industry research confirms this. Multi-agent systems fail at the coordination layer, not the capability layer. A 2026 analysis of five popular multi-agent frameworks across 150 tasks identified 14 distinct failure modes across three categories. The most frequent was incomplete context passing between agents. The most destructive was silent context overwrite, where one agent's output unintentionally replaced another agent's data.

I see the same thing. When agent two passes a summary to agent five, the summary loses nuance. By the time agent eight finishes, the original intent has been diluted three or four times. The fix is structured handoffs with explicit field schemas. Instead of passing a paragraph of text, agents pass typed objects with required and optional fields. If a required field is missing, the handoff fails and the coordinator knows exactly where.

The Architecture That Survived

I started with a flat architecture. One agent does research. One does drafting. One does QA. They passed work forward like an assembly line.

That broke in month two. The QA agent would reject a draft, send it back, and nobody tracked the revision state. Drafts got stuck. Agents waited for each other. The queue backed up.

What works now is a hub and spoke pattern. A coordinator agent owns the pipeline state. Specialized agents do the work and report back. The coordinator tracks what is pending, what passed, what failed, and what needs human input.

The coordinator does not make content decisions. It manages workflow state. This separation, state management from content judgment, is the single most important architectural decision I made.

This maps to the supervisor-worker pattern from the 2026 agent architecture taxonomy. In that taxonomy, hub-and-spoke is one of eight canonical patterns, and it is one of two multi-agent patterns that reliably earns its cost in production. The other is plan-and-execute. The remaining patterns, like mesh and swarm, introduce coordination overhead that rarely pays off outside niche use cases.

The key decision criteria for hub-and-spoke is whether your tasks decompose into clear sub-tasks with defined handoffs. Marketing content production does. Research brief generation does. Customer segmentation does. Open-ended reasoning tasks generally do not, and those benefit more from plan-and-execute or single-agent approaches.

If I were starting over, I would still begin with a single agent handling one workflow. I would escalate to multi-agent only when the single agent hit a measured ceiling on quality or throughput. Most teams add coordination overhead before their single agent has reached its natural limit.

The Tooling Stack

A few people have asked what tools I use. Here is the short version.

The agents run on a combination of n8n for workflow management and custom scripts for the agent logic. Each agent is essentially a prompt template plus a set of tool definitions. The coordinator is an n8n workflow that checks pipeline state every 15 minutes.

For observability, I use a mix of n8n's built-in execution logging and a custom trace storage layer that captures every tool call and model response. I experimented with LangSmith for a month but found that the agent-level tracing I needed was simple enough to build myself.

For evaluation, I built a small evaluation harness that runs the four-layer stack described above. The deterministic checks are assertion functions. The trace review is manual but structured. The human gates are a Telegram bot that sends me draft summaries and waits for approval before publishing. The post-deployment monitoring cross-references Google Search Console data with the agent's output log.

The total infrastructure cost is about $150 per month. That includes API credits for the models, n8n cloud hosting, and storage for traces and outputs. The agent vendors would have you believe you need a six-figure platform budget. You do not. You need clear evaluation criteria and the discipline to check them.

The Cost of Not Evaluating

A quick data point for anyone building a business case for evaluation.

The average direct cost of an agent project failure, per the Digital Applied analysis, is $340,000. That includes engineering time, model credits, and integration work. When you factor in opportunity cost, the number exceeds $650,000.

My eval stack cost about two weeks of part-time work to build. The trace review costs me 30 minutes per week. The human gates cost five minutes per published piece. The post-deployment monitoring costs about 10 minutes per month.

The math is not subtle. An eval stack that prevents one bad published post pays for itself a hundred times over. An eval stack that prevents a broken agent from running for two weeks pays for itself in model credits alone.

What This Means for Marketing Teams

If you are building or buying AI marketing tools, start with evaluation before you start with agents.

Define what good output looks like for your specific use case. Create ten examples. Then five examples of "close but wrong." Then five examples of "completely wrong." If you cannot articulate the difference between these categories, an agent cannot either.

Build the deterministic checks first. They are cheap and they catch the obvious failures. Add trace review second. Add human gates for anything customer facing.

A simple way to start: pick one agent running one workflow. Spend one hour defining what good output looks like. Build three deterministic checks. Review ten traces. That is a week of work and it will tell you more about your agent's readiness than a month of benchmark testing.

And do not trust task counts. An agent that publishes fifty posts a month is not necessarily more valuable than one that publishes five posts that actually drive results. Measure work, not activity.

The agents are still running. They will wake up tomorrow and do their jobs. The difference between month one and month six is not that the agents got smarter. It is that the system around them got harder to fool.

If you are building production agents and want to compare notes, I am easy to find. The operators who share what breaks learn faster than the ones who keep it quiet.