# Cloudflare's Browser Run crawl results now flag per-URL outcomes and robots.txt state A changelog entry says Browser Run /crawl responses can carry an outcome for each URL and a job-level robots.txt report covering up to 50 origins, giving developers more detail on why a crawl succeeded or failed. Canonical URL: https://freelancenews.online/news/cloudflare-s-browser-run-crawl-results-now-flag-per-url-outcomes-and-cb957502 Published: 2026-11-03T15:18:10.138Z Updated: 2026-11-03T15:18:10.138Z Source published: 2026-11-03T00:00:00.000Z Event date: Not established Review status: source-reviewed Review method: Automated comparison against retrieved source text; not independent fact-checking. ## Report Cloudflare has updated the output of its Browser Run /crawl feature so that crawl results carry more diagnostic detail, according to a changelog entry on the company's developer documentation site. The entry frames the change as an addition to what crawl results include, and it matters most to anyone who reads those results programmatically rather than by eye. The first addition is a per-URL outcome field. According to the changelog, each URL record in a crawl result can include an outcome that identifies the specific processing result. The examples the entry names are success, robots_disallowed, timeout and page_error. The wording is that a record "can include" an outcome, which describes a capability rather than a guarantee that every record will carry one. A job-level robots.txt report is the second addition. Up to 50 origins that the crawl encountered are covered by this report, according to the changelog, which describes the robots.txt state for those origins. Listed for each entry are the origin, whether usable rules were available, and the HTTP status the crawler received when it got a response. The entry illustrates the shape of the output with a short JSON sample. In it, a robots array holds an object with an origin, a state of available and a status of 200, while a records array holds a URL with a status of completed and an outcome of success. The sample is illustrative; the entry does not present it as an exhaustive schema. For field definitions and the complete set of possible outcome values, the changelog points readers to separate documentation on URL outcomes and robots.txt diagnostics. That pointer is significant: the supplied text names four outcome values as examples, not as a closed list, so the full enumeration lives elsewhere. For developers, the practical difference is diagnostic resolution. A crawl that returns a page_error outcome tells the operator something different from one that returns robots_disallowed, and the robots report separates a site that blocked the crawler by policy from one that failed to respond at all. That distinction matters when a crawl is being debugged, because the remedy differs: a robots exclusion is a policy decision by the site owner, while a timeout or page error is a technical failure on either side. This is editorial reasoning about how the fields could be used, not a claim the changelog makes. The robots.txt report also has a stated ceiling. Because it covers up to 50 origins, a crawl spanning more origins than that will not have every origin represented in the job-level report as described. The changelog does not explain what happens beyond that limit, whether entries are truncated, sampled or ordered in a particular way, so developers working at larger scale should treat the cap as a known constraint rather than assume full coverage. The entry does not describe any pricing, plan availability or configuration flag associated with the new fields. It also does not say whether the fields are always present or only populated in certain conditions, and it does not describe backward compatibility for existing consumers of crawl output. Those are open questions for anyone whose code parses these results. Editorially, the most useful reading for freelancers and small teams is defensive: code that consumes crawl results should tolerate the new fields appearing and should not assume a fixed set of outcome values, since the changelog defers the complete list to separate documentation. A parser that switches on outcome strings without a default branch is the kind of code this change could quietly break. The change also suggests a workflow improvement for client work. When a crawl returns fewer pages than expected, the outcome field and the robots report together let a developer show a client a concrete reason, such as a site's robots rules disallowing access or a page failing to load, rather than a bare count of missing URLs. That is an interpretation of how the fields could be used, not a claim made in the source. What the evidence does not support is any statement about accuracy, performance or reliability of the new diagnostics. The changelog describes what the fields contain and gives one sample payload; it reports no testing, no benchmarks and no comparison with prior behavior. Readers should not infer that outcomes are exhaustive or that the robots report reflects anything beyond what the crawler observed during that job. The scope of the announcement is narrow and worth stating plainly: it concerns the shape of crawl results from Browser Run, not the crawling engine itself, not rate limits, and not any other Cloudflare product. The supplied text contains no statement about changes to crawling behavior, only to the diagnostics returned alongside it. For working developers, the takeaway is to check how their crawl-consuming code handles unknown fields and unknown outcome values, and to consult the linked field documentation before relying on any particular outcome string. The changelog's own pointer to that documentation is the most reliable next step, since the entry does not list every possible value. ## Key points - Browser Run /crawl results can now include a per-URL outcome such as success, robots_disallowed, timeout or page_error, per the changelog. - A job-level robots.txt report covers up to 50 origins, listing each origin, whether usable rules were available, and the HTTP status received. - The changelog defers full field definitions and the complete set of outcome values to separate documentation rather than listing them. - The entry does not state pricing, plan availability, or whether the new fields are always populated. ## Practical implications — editorial interpretation Editorial interpretation: developers who parse crawl output should make their code tolerant of new fields and unrecognized outcome strings, and should read the linked field documentation before branching on specific values. The robots report can help distinguish policy blocks from technical failures when explaining crawl gaps to clients. ## Limitations and unknowns The evidence is a single changelog entry with one sample payload. It does not enumerate all outcome values, explain behavior beyond the 50-origin cap, state pricing or plan availability, confirm backward compatibility, or report any testing or performance data. No independent verification was supplied. ## Sources - [1] developers.cloudflare.com: Changelog https://developers.cloudflare.com/changelog/post/2026-11-03-crawl-result-diagnostics/ Retrieved: 2026-11-03T15:17:28.252Z ## Claim references - Crawl results can include a per-URL outcome identifying the specific processing result, with examples including success, robots_disallowed, timeout and page_error. [source 1] - A job-level robots.txt report covers up to 50 origins and includes the origin, whether usable rules were available, and the HTTP status received. [source 1] - The changelog provides a sample payload showing a robots entry with state available and status 200, and a record with status completed and outcome success. [source 1] - Field definitions and the full list of possible outcome values are deferred to separate documentation. [source 1]