> ## Documentation Index > Fetch the complete documentation index at: https://getanyapi.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Web Scrape > Scrape any web page and get its content back as clean Markdown (or HTML, or raw HTML) plus title and metadata. **Price:** \$0.50 per 1,000 requests (flat per request - same cost regardless of results returned). **Routing:** 3 independent lanes - a failed attempt fails over to the next lane automatically, and the same normalized schema comes back either way. ## OpenAPI ````yaml /openapi.json post /v1/run/web.scrape openapi: 3.1.0 info: contact: email: support@getanyapi.com description: >- Any API, one wallet, USD, no subscriptions. Each API is a single discovered operation: send the normalized input, get a normalized result. You are charged the winning provider's price in real dollars only on success; failed calls are never charged. title: AnyAPI version: 1.0.0 x-guidance: >- Use each operation's published method and path with its normalized JSON input (see the operation requestBody). To pay: send your AnyAPI key (Authorization: Bearer, billed from your USD wallet); or pay per call inline with no account - x402 (call with no key, get HTTP 402 with a PAYMENT-REQUIRED header, retry with the PAYMENT-SIGNATURE header, USDC on base), or MPP (retry with Authorization: Payment, tempo). You are charged only on success. servers: - url: https://api.getanyapi.com security: - bearerAuth: [] - apiKeyAuth: [] tags: - name: ahrefs x-group: Ahrefs - name: airbnb x-group: Airbnb - name: alibaba x-group: Alibaba - name: amazon x-group: Amazon - name: apollo x-group: Apollo - name: appstore x-group: Appstore - name: bluesky x-group: Bluesky - name: booking x-group: Booking - name: coinmarketcap x-group: Coinmarketcap - name: congress x-group: Congress - name: dexscreener x-group: Dexscreener - name: douyin x-group: Douyin - name: ebay x-group: Ebay - name: email x-group: Email - name: facebook x-group: Facebook - name: fiverr x-group: Fiverr - name: github x-group: Github - name: glassdoor x-group: Glassdoor - name: google x-group: Google - name: google-ads x-group: Google Ads - name: google-finance x-group: Google Finance - name: google-shopping x-group: Google Shopping - name: hackernews x-group: Hackernews - name: indeed x-group: Indeed - name: instagram x-group: Instagram - name: linkedin x-group: Linkedin - name: maps x-group: Maps - name: pandaexpress x-group: Pandaexpress - name: person x-group: Person - name: pinterest x-group: Pinterest - name: playstore x-group: Playstore - name: polymarket x-group: Polymarket - name: realtor x-group: Realtor - name: reddit x-group: Reddit - name: redfin x-group: Redfin - name: rednote x-group: Rednote - name: sec x-group: Sec - name: semrush x-group: Semrush - name: seo x-group: Seo - name: snapchat x-group: Snapchat - name: social x-group: Social - name: spotify x-group: Spotify - name: substack x-group: Substack - name: threads x-group: Threads - name: tiktok x-group: TikTok - name: tiktok-shop x-group: TikTok Shop - name: tripadvisor x-group: Tripadvisor - name: trustpilot x-group: Trustpilot - name: truthsocial x-group: Truthsocial - name: twitter x-group: X (Twitter) - name: upwork x-group: Upwork - name: walmart x-group: Walmart - name: web x-group: Web - name: weibo x-group: Weibo - name: yahoo-finance x-group: Yahoo Finance - name: yelp x-group: Yelp - name: youtube x-group: YouTube - name: zhihu x-group: Zhihu - name: zillow x-group: Zillow paths: /v1/run/web.scrape: post: tags: - web summary: Web Scrape description: >- Scrape any web page and get its content back as clean Markdown (or HTML, or raw HTML) plus title and metadata. **Price:** \$0.50 per 1,000 requests (flat per request - same cost regardless of results returned). **Routing:** 3 independent lanes - a failed attempt fails over to the next lane automatically, and the same normalized schema comes back either way. operationId: web_scrape parameters: - description: >- Optional. Comma-separated keys (dotted paths like `author.name` descend into nested objects) to keep on each result item. Keys are matched relative to each result item after the `data`/`items` envelope is unwrapped, not against the top-level response envelope, so use `jq` to reshape the whole envelope. Shrinks the response without changing cost. in: query name: fields required: false schema: type: string - description: >- Optional. Cap the number of result rows returned; a `_truncated` note reports how many were withheld so you can page via the API's own `limit`. Does not change cost. in: query name: max_items required: false schema: minimum: 0 type: integer - description: >- Optional. Return only a structural outline (top-level keys, item counts, and per-field byte sizes) instead of the full data. Does not change cost. in: query name: summary required: false schema: type: boolean - description: >- Optional. A jq expression applied to the result envelope; its output replaces `output` (multiple outputs collect into an array). Reshape freely, e.g. `jq=.data | {title, description, md: .markdown[:3500]}`. Runs sandboxed with a 250ms / 2MB budget; on failure the full result is returned with a `jqError`. Does not change cost. in: query name: jq required: false schema: type: string - description: >- Optional wallet idempotency key, scoped to this customer for 24 hours. When the gateway honors the key, this synchronous in-process execution can continue after the caller disconnects, bounded by its execution deadline. A completed replayable result charges normally exactly once and can be replayed without another provider run or charge. A pending duplicate returns 409 idempotency_in_progress; reuse with different request semantics returns 409 idempotency_conflict. in: header name: Idempotency-Key required: false schema: maxLength: 255 minLength: 1 type: string requestBody: content: application/json: example: formats: - markdown - rawHtml onlyMainContent: false url: https://example.com schema: additionalProperties: false example: formats: - markdown - rawHtml onlyMainContent: false url: https://example.com properties: blockAds: description: >- When true (upstream default), strip ad and cookie-consent elements before capture. Set false to keep them. type: boolean excludeTags: description: >- CSS selectors to drop before capture (for example ["nav", "footer", ".ads"]). Applied after includeTags. items: type: string type: array formats: default: - markdown - rawHtml description: >- Which representations of the page to return. Any combination of: markdown (page content as Markdown), html (cleaned HTML), rawHtml (verbatim page HTML). Each requested format is returned under the matching output field. Defaults to Markdown and raw HTML. items: enum: - markdown - html - rawHtml type: string type: array includeTags: description: >- CSS selectors to keep. When set, only content matching these selectors is captured (for example ["article", "main"] or ["#content"]). items: type: string type: array mobile: description: >- When true, render the page with a mobile viewport and user agent instead of desktop. Some sites serve materially different content to mobile. type: boolean onlyMainContent: default: false description: >- When true, return only the main article content, stripping navigation, headers, footers, and other boilerplate. Defaults to false to capture the full page. type: boolean stealth: default: false description: >- When true, fetch through a stealth proxy that gets past bot protection which refuses an ordinary request. This costs materially more per request (see the pricing ceiling), so leave it off unless a normal scrape of the site comes back blocked or empty. type: boolean url: description: The URL of the page to scrape. format: uri minLength: 1 type: string waitFor: description: >- Milliseconds to wait for the page to finish rendering before capture. Use this for JavaScript-heavy pages or single-page apps whose content loads after the initial paint. Capped at 15000 to stay within the request timeout. This wait is time you asked us to spend, so your response takes this much longer, and it is excluded from the latency published for this endpoint. maximum: 15000 minimum: 0 type: integer x-anyapi-caller-delay: unit: ms required: - url title: Web scrape input type: object required: true responses: '200': content: application/json: schema: properties: costUsd: description: >- USD charged on the original run. On a replay this value is echoed for parity; the replay itself is free. type: number items: description: >- Number of result rows returned. For per-result SKUs the per-item cost is charged against this count; for input-priced SKUs the charge is per submitted input, independent of this count. type: integer jqError: description: >- Present only when a `jq` expression failed; `output` then carries the full unshaped result and this explains why the reshape did not apply. type: string output: anyOf: - additionalProperties: false properties: data: oneOf: - type: 'null' - properties: description: description: The page meta description. type: string html: description: >- The cleaned page HTML. Present only when 'html' is among the requested formats. type: string markdown: description: >- The page content as clean Markdown. Present when 'markdown' is among the requested formats (the default). Populated whenever the provider has data for the entity. type: string x-anyapi-must-populate: true rawHtml: description: >- The verbatim page HTML before cleaning. Present only when 'rawHtml' is among the requested formats. type: string title: description: The page title from its metadata. type: string url: description: >- The canonical source URL of the scraped page. Populated whenever the provider has data for the entity. type: string x-anyapi-must-populate: true required: - url - title - description - markdown type: object found: type: boolean required: - found - data title: Web scrape output type: object x-anyapi-heavy: true - type: 'null' description: >- Normalized output, or null when the replay payload was not retained. provider: description: Always "AnyAPI". type: string replayed: description: >- True when this response replays the durable result of an earlier run without billing or upstream execution. type: boolean resultId: description: >- Opaque handle to the full unshaped result, cached ~15 min. Re-shape it for free (fields/max_items/summary/jq) via GET /v1/results/{id}, no re-billing. Absent when the result was too large to cache. type: string required: - output - provider - costUsd - replayed type: object description: Normalized result. headers: Idempotency-Replayed: description: >- Present only when the request carried an Idempotency-Key this gateway honored. "false" means the key was claimed and the run executed freshly; "true" means it replayed a prior identical run and was NOT charged again. Absent means either no key was sent or this gateway does not honor keys, so a retry could be charged twice. schema: enum: - 'false' - 'true' type: string '400': content: application/json: schema: $ref: '#/components/schemas/Error' description: >- Input did not match the schema, or the upstream rejected the request (no charge). '401': content: application/json: schema: $ref: '#/components/schemas/Error' description: Missing or invalid API key. '402': content: application/json: schema: oneOf: - $ref: '#/components/schemas/PaymentRequiredError' - $ref: '#/components/schemas/TrialCapReached' description: >- Insufficient wallet balance or spend cap (no charge). Agent trials receive a trial_cap_reached continuation with a live RFC 8628 device authorization. '404': content: application/json: schema: $ref: '#/components/schemas/Error' description: Unknown SKU, or the upstream had no matching resource (no charge). '409': content: application/json: schema: $ref: '#/components/schemas/Error' description: >- Idempotency collision. Codes: idempotency_in_progress means retry after the Retry-After delay; idempotency_conflict means use a new key for different request semantics; idempotency_needs_review means stop retrying and contact AnyAPI support with X-Anyapi-Request-Id. '422': content: application/json: schema: $ref: '#/components/schemas/Error' description: >- The request was well formed but this API cannot serve the requested target site; the body's `alternatives` array lists the AnyAPI APIs that can (no charge). '429': content: application/json: schema: $ref: '#/components/schemas/Error' description: The upstream was rate limited, retry shortly (no charge). '502': content: application/json: schema: $ref: '#/components/schemas/Error' description: >- Every provider failed with a server/transport error (charge released). components: schemas: Error: properties: code: description: Stable machine-readable error code when the endpoint defines one. type: string error: description: Customer-safe error message. type: string required: - error type: object PaymentRequiredError: allOf: - $ref: '#/components/schemas/Error' - properties: error: not: const: trial_cap_reached type: object TrialCapReached: properties: error: const: trial_cap_reached type: string message: description: >- Human-relayable continuation instructions, including an executable token poll. type: string upgrade: $ref: '#/components/schemas/AgentUpgrade' required: - error - message - upgrade type: object AgentUpgrade: properties: authorizationServer: format: uri type: string clientId: type: string device: $ref: '#/components/schemas/DeviceUpgrade' scope: type: string required: - authorizationServer - scope type: object DeviceUpgrade: properties: deviceCode: type: string expiresIn: minimum: 0 type: integer grantType: type: string interval: minimum: 1 type: integer tokenEndpoint: format: uri type: string userCode: type: string verificationUri: format: uri type: string verificationUriComplete: format: uri type: string required: - verificationUriComplete - verificationUri - userCode - deviceCode - tokenEndpoint - grantType - interval - expiresIn type: object securitySchemes: bearerAuth: description: Your AnyAPI key as a Bearer token. scheme: bearer type: http x-default: YOUR_ANYAPI_KEY apiKeyAuth: description: Your AnyAPI key. in: header name: X-API-Key type: apiKey x-default: YOUR_ANYAPI_KEY ````