Crawl a site starting from a root URL

Follow links from a starting URL and extract every page reached, up to max_depth and max_pages. Returns a job_id immediately; crawling is asynchronous.

Follow links from a starting URL and extract every page reached, up to max_depth and max_pages. Returns a job_id immediately; crawling is asynchronous.

Poll GET /crawl/{job_id} or supply webhook_url. The crawler stays on the starting host, honours robots.txt, and applies your key's domain policy to every link it follows — not just the root.

A crawl stops early rather than failing when it hits a ceiling: truncated is set with a truncated_reason of page_cap_reached or quota_exhausted, and the pages gathered so far are still returned. max_depth is capped by plan.

Metered per page extracted, not per page discovered.

Authorization

BearerAuth

AuthorizationBearer <token>

Your API key, sent as Authorization: Bearer wm_.... Keys are wm_ followed by 40 hex characters. Create one with POST /register (free, no auth) or POST /keys.

In: header

Request Body

application/json

TypeScript Definitions

Use the request body type in TypeScript.

allow_domains?array<string>|

Restrict the crawl to these domains. Narrowing only. Applied to every link followed, not just the starting URL.

deny_patterns?array<string>|

URL patterns the crawler must not follow. Narrowing only.

depth?Depth

How many links deep to follow. 0 extracts only the starting URL. Capped by plan; exceeding it returns 422 crawl_depth_exceeded.

Default1

format?Format

Output shape, applied to every page crawled. json and chunks are Pro+.

Default"markdown"

Value in

"markdown"
"json"
"chunks"
"html"
"links"

max_pages?|

Page budget for this crawl. The effective ceiling is the lower of this and your plan's cap, so it can only narrow. Null uses the plan cap alone. Hitting the ceiling truncates the crawl rather than failing it — check truncated on the result.

render_js?Render Js

Render every crawled page in a headless browser before extracting. Pro+ only.

Defaultfalse

respect_robots?|

Override robots.txt handling. lax→strict only.

retry?Retry

Extra attempts per page if the fetch times out.

Range0 <= value

Default0

url*Url

The starting page. The crawler stays on this host.

Formaturi

Length1 <= length <= 2083

webhook_include_results?Webhook Include Results

Inline the full results array in the webhook payload. Same ~5 MB cap as bulk.

Defaultfalse

webhook_url?|

Receive a signed POST when the crawl finishes, instead of polling. Same rules as BulkExtractRequest.webhook_url.

Response Body

`application/json`

curl -X POST "https://api.wellmarked.io/crawl" \  -H "Authorization: Bearer wm_your_api_key_here" \  -H "Content-Type: application/json" \  -d '{    "url": "http://example.com"  }'

{  "completed": 0,  "job_id": "453bd7d7-5355-4d6d-a38e-d9e7eb218c3f",  "job_token": "string",  "kind": "crawl",  "results": [    {      "blocks": [        {          "level": 0,          "text": "string",          "type": "heading"        }      ],      "chunks": [        {          "end_token": 0,          "start_token": 0,          "text": "string"        }      ],      "depth": 0,      "error": "string",      "html": "string",      "links": [        "string"      ],      "markdown": "string",      "metadata": {        "author": "string",        "date": "string",        "retrieved_at": "2019-08-24T14:15:22Z",        "title": "string",        "url": "string"      },      "metrics": {        "content_bytes": 0,        "input_tokens": 0,        "output_tokens": 0,        "reduction_pct": 0,        "tokens_saved": 0      },      "url": "string"    }  ],  "status": "string",  "total": 0,  "truncated": false,  "truncated_reason": "string",  "webhook_signing_secret": "string"}

Crawl a site starting from a root URL

Authorization

Request Body

Response Body

200application/json

401application/json

402application/json

403application/json

409application/json

422application/json

429application/json

`application/json`

`application/json`

`application/json`

`application/json`

`application/json`

`application/json`

`application/json`