Web Scraping Crawl website API
Crawl a website and get clean text content from up to 10 pages in one normalized response - ideal for feeding sites into LLMs and search indexes.
$1.25/1k results, pay per request, no subscription.
{"url": "https://www.example.com","title": "Example - Build something great","text": "Example is the platform teams use to ship faster…","author": string | null,"canonicalUrl": string | null,"contentType": string | null,"description": string,"domain": string,... 9 more fields}
Pricing and providers
Web Scraping Crawl website API pricing: one API, 3 interchangeable providers
The cheapest provider serves first. If it fails, the next one takes over in the same call.
capped at $0.0125 per request1 result$0.001255 results$0.0062510 results$0.0125 capYou are charged for what comes back. Lower the limit to pay less - and no single request can cost more than the cap, however much comes back.2BisonBackup 150.98% uptime13682ms p5012.6% of calls12.6%52 calls$1.25/1k resultsup to $0.0125 / requestMetered per result returned$0.00125 per result
capped at $0.0125 per request1 result$0.001255 results$0.0062510 results$0.0125 capYou are charged for what comes back. Lower the limit to pay less - and no single request can cost more than the cap, however much comes back.3SalamanderBackup 231.25% uptime14204ms p506.1% of calls6.1%25 calls$1.25/1k resultsup to $0.0125 / requestMetered per result returned$0.00125 per result
capped at $0.0125 per request1 result$0.001255 results$0.0062510 results$0.0125 capYou are charged for what comes back. Lower the limit to pay less - and no single request can cost more than the cap, however much comes back.
Try it
Get Web Scraping crawl website in one request
This sample shows 6 of 31 fields. See every field
{
"found": true,
"data": {
"items": [
{
"url": "https://www.example.com",
"title": "Example - Build something great",
"text": "Example is the platform teams use to ship faster…"
},
{
"url": "https://www.example.com/pricing",
"title": "Pricing - Example",
"text": "Simple plans for teams of every size…"
}
]
}
}Full parameter and response reference - every field, type, and example for this endpoint.
Reference
One request, one response shape.
Every field you send and every field you get back, each with a real example.
Last verified 2026-10-31 · uptime and latency measured over 30d
Request body
JSON, posted to this endpoint.
- urlstring
https://www.example.com
The website URL or domain to crawl. Pages linked from it on the same site are crawled too.
- limitinteger
Maximum number of pages to crawl and return (1-10, default 10). You are billed per page returned, so a lower limit costs less.
- requireFieldsarray
Optional; omit it and routing is unchanged. Name the output fields this request must be able to return, for example `language`, and it is served only by a source that returns every one of them. Fields you do not name are still returned whenever the serving source has them. Every source charges the same per page, so this never changes your price. A named field can still be empty on a page that genuinely lacks it.
Response
JSON, one example value per field.
Pages the crawl reported as failed or skipped. They are never in items and never billed. The list holds what the crawl reported, not every page of the site.
- outcome"failed" | "robots_disallowed" | "skipped"
failed: the page could not be fetched or answered with an HTTP error. robots_disallowed: the site's robots.txt does not allow crawling it. skipped: the crawl did not fetch it, for example because it is outside the crawled site.
- statusCodeinteger
The HTTP status code the page answered with, or null when unknown.
- urlstring
The page's URL.
One record per crawled page: its URL and domain, page title, meta description, language, HTTP status, the page content as Markdown text, and the page metadata the source reports. You are billed for these pages only. Populated whenever the provider has data for the entity.
- authorstringcan require
The author the page declares, or null when unknown.
- canonicalUrlstringcan require
The canonical URL the page declares in its own markup, or null when it declares none or the source does not report it.
- contentTypestringcan require
The content type the page was served with (for example text/html;charset=utf-8), or null when unknown.
- descriptionstring
The page's meta description, empty when the page has none.
- domainstring
The page's host name, without a leading www. Populated whenever the provider has data for the entity.
- jsonLdany[]can require
The JSON-LD structured data blocks the page publishes (schema.org Organization, LocalBusiness, BreadcrumbList and so on), as parsed JSON, or null when the source does not report them. They are the publisher's own claims, not verified facts.
- languagestringcan require
The page's declared language code (for example en-US), or null when unknown.
- modifiedUtcintegercan require
When the page says it was last modified, as a UTC epoch timestamp in seconds (Unix time), or null when unknown. Multiply by 1000 for a JS Date in milliseconds.
- publishedUtcintegercan require
When the page says it was published, as a UTC epoch timestamp in seconds (Unix time), or null when unknown. Multiply by 1000 for a JS Date in milliseconds.
- robotsstringcan require
The page's robots meta directives (for example noindex, nofollow), or null when it declares none or the source does not report them.
- siteNamestringcan require
The site name the page declares (its og:site_name), or null when unknown.
- sourceUrlstring
The URL the crawler requested for this page, before any redirect, or null when unknown.
- statusCodeinteger
The HTTP status code the page answered with, or null when unknown.
- textstring
Example is the platform teams use to ship faster…
The page content as Markdown. Populated whenever the provider has data for the entity.
- titlestring
Example - Build something great
The page title, empty when the page has none (a PDF, for example).
- urlstring
https://www.example.com
The crawled page's URL, after any redirect. Populated whenever the provider has data for the entity.
The page's headings in document order, each with its level (1 for an h1) and text, or null when the source does not report them.
- levelinteger
The heading level, 1 to 6.
- textstring
The heading text.
Counts for this crawl. They describe the pages this crawl visited and reported, not the whole website, and are not added together into a total.
- pagesReturnedinteger
The number of pages in items, which is the number you are billed for.
- reportedFailedinteger
The number of pages the crawl reported as failed, or null when the source does not report a complete count.
- reportedSkippedinteger
The number of pages the crawl reported as skipped, including those robots.txt disallows, or null when the source does not report a complete count.
- foundboolean
true
- reason"not_found"
Present only when `found` is false, and says why there is no result. `not_found`: the source states the target does not exist, or returned nothing for it. A `found: false` answer is a successful call, not an error, and `costUsd` is what it actually cost.
FAQ
About the Web Scraping Crawl website API
The AnyAPI Web Scraping Crawl website API returns Web Scraping crawl website data as normalized JSON from one POST call to /v1/run/web.crawl. Crawl a website and get clean text content from up to 10 pages in one normalized response - ideal for feeding sites into LLMs and search indexes. AnyAPI routes each request across 3 sources and falls back automatically when one fails. It costs $1.25 per 1,000 results, and never more than $0.0125 for a single request, in US dollars with no subscription and no monthly minimum. Over the last 30 days, 96.8% of Web Scraping crawl website calls through AnyAPI succeeded, with a median response time of 6.5 seconds across 2,315 measured calls.