Zapier has published an account of an internal experiment measuring how AI agents read its website, and the findings complicate a common assumption about how agent traffic reaches a page. The company built a small tool it calls Agent Arm Runner, which runs Claude Code inside a Docker container behind a forward proxy, then directs requests for a single URL at either the live production page or a preview deployment. With the URL unchanged and only the content behind it swapped, Zapier could compare agent responses without disturbing anything else in the agent's environment.
The stated motivation is feedback speed. Search optimization work can take weeks or months before enough data exists to judge whether a page change helped. Zapier says it wanted a faster loop for the equivalent question with agents: when an agent searches for Zapier, reads an integration page and answers a user, how much does the page content actually shape what comes back? The harness is the company's attempt to isolate that variable.
The mechanics are documented at some length. A JSON configuration defines which prompts run, which experimental arms exist and how many repetitions occur. In the control arm, requests to zapier.com reach production; in a variant arm, the proxy intercepts requests for the path under test and routes them to a preview deployment instead. Because state must not carry between experiments, every run begins in a fresh container, and since agent behavior is not deterministic, each arm can be repeated several times.
For the first real test, Zapier chose a deliberately large difference. The control was the live Slack integration page. The variant removed almost everything from the page body, including the triggers, actions and integration content. Zapier is explicit that this was not a change it intended to ship; the goal was to confirm the harness worked and to establish a baseline for how much page content influenced Claude's answer.
The expectation was straightforward: the production page should yield details about the Slack integration, and the gutted variant should yield very little. That is not what happened. According to Zapier, the variant still returned specific details about Slack triggers and actions, including counts of reads, writes and searches and named items such as posting a message to a channel or adding a reaction. Those details were not in the HTML being served for the variant.
Zapier checked the proxy logs to confirm the request was actually reaching the preview deployment, and it was. The logs also showed why the content survived. The request carried an accept header listing text/markdown ahead of text/html, and the server returned a Markdown representation of the page. Zapier had removed the integration content from the HTML version but had not made the same change to the Markdown mirror, so the agent received the original details through that channel.
To trace the path, Zapier added a canary string to the Markdown version with an instruction to include the exact code when summarizing the page. The same canary later appeared in content passing through WebFetch. Zapier frames this as a concrete demonstration that Claude was consuming a Markdown representation rather than the HTML page, and notes that the company had previously treated Markdown mirrors as a readability optimization rather than as part of the request path under test.
A second layer came to light in the logs. Opus 5 was reported as the calling model by the top-level run, yet the captured WebFetch processing showed the fetched page arriving in a separate request to Haiku 4.5, with no tools attached. Before the result was returned to the calling model, the fetched content was passed to that model in the requests Zapier captured. Zapier reads this as meaning the model that answers the user's question may differ from the one directly processing the page.
That intermediate request also carried its own instructions. Zapier quotes an excerpt requiring a concise response based only on the supplied content, enforcing a strict 125-character maximum for quotes from any source document, requiring quotation marks for exact language and forbidding word-for-word reproduction outside quotations. The excerpt also includes rules against commenting on the legality of prompts and responses and against reproducing song lyrics.
Zapier draws a distinction from this: there is the content the company publishes, and there is the representation of that content that eventually reaches the calling model, with an instruction layer in between that the publisher does not control. The quotation and paraphrasing rules matter specifically for how content should be written for this path, because they can change what survives the summarization step regardless of what the page actually says.
The company handles scope carefully. Its statement says the described behavior amounts to what it observed with Claude Code and WebFetch during these tests; it declines to assume that other agents, or even later releases of the same tools, will behave identically. No conclusions are offered; open questions are listed instead. Among them: how a page's structure influences what makes it through the WebFetch step, whether prose and headings or lists survive differently, what depth within a document an agent can still locate reliably, and whether other categories of Zapier pages reproduce the pattern.
For freelancers and developers who maintain sites that agents read, the practical takeaway is that testing only the HTML is insufficient. Zapier's own advice is to test the Markdown representation of a page as well, because in its tests that representation was already live in the request path. Anyone running content experiments for agent traffic should also expect that the model consuming the page may be a smaller intermediary model operating under instructions the publisher cannot see.
There is a tradeoff embedded in the approach. Holding the URL constant while swapping content is what let Zapier separate a content change from everything else in the agent's environment, but it requires infrastructure — containers, a proxy, preview deployments and repeated runs — that is heavier than a browser A/B test. Zapier acknowledges this directly, describing the principle as the same as any A/B test but harder to set up for agents.
The limitations are worth stating plainly. This is a single company's account of its own internal tooling, with no published dataset, sample sizes or statistical analysis in the supplied material. The first experiment used an intentionally extreme variant, so it demonstrates that the harness detects a large difference rather than quantifying how much ordinary page content matters. The observations are tied to specific model versions and one agent harness, and Zapier itself declines to generalize them.
What the account does establish is a concrete mechanism: Markdown mirrors can serve content that differs from the HTML, and WebFetch can route fetched pages through a separate model with its own summarization rules before the calling model sees anything. For teams whose pages are read by agents, those are two places where the published page and the consumed page can diverge, and both are worth checking before drawing conclusions from agent behavior.