Crawl a site starting from a root URL
Follow links from a starting URL and extract every page reached, up to `max_depth` and `max_pages`. Returns a `job_id` immediately; crawling is asynchronous. Poll `GET /crawl/{job_id}` or supply `webhook_url`. The crawler stays on the starting host, honours `robots.txt`, and applies your key's domain policy to every link it follows — not just the root. A crawl stops early rather than failing when it hits a ceiling: `truncated` is set with a `truncated_reason` of `page_cap_reached` or `quota_exhausted`, and the pages gathered so far are still returned. `max_depth` is capped by plan. Metered per page extracted, not per page discovered.
Authorization
BearerAuth Your API key, sent as Authorization: Bearer wm_.... Keys are wm_ followed by 40 hex characters. Create one with POST /register (free, no auth) or POST /keys.
In: header
Request Body
application/json
TypeScript Definitions
Use the request body type in TypeScript.
Response Body
application/json
application/json
application/json
application/json
application/json
application/json
application/json
curl -X POST "https://example.com/crawl" \ -H "Content-Type: application/json" \ -d '{ "url": "http://example.com" }'{ "completed": 0, "job_id": "453bd7d7-5355-4d6d-a38e-d9e7eb218c3f", "job_token": "string", "kind": "crawl", "results": [ { "blocks": [ { "level": 0, "text": "string", "type": "heading" } ], "chunks": [ { "end_token": 0, "start_token": 0, "text": "string" } ], "depth": 0, "error": "string", "html": "string", "links": [ "string" ], "markdown": "string", "metadata": { "author": "string", "date": "string", "retrieved_at": "2019-08-24T14:15:22Z", "title": "string", "url": "string" }, "metrics": { "content_bytes": 0, "input_tokens": 0, "output_tokens": 0, "reduction_pct": 0, "tokens_saved": 0 }, "url": "string" } ], "status": "string", "total": 0, "truncated": false, "truncated_reason": "string", "webhook_signing_secret": "string"}Poll the status of a bulk extraction job
Return the current state of a bulk job: `status`, how many URLs have `completed` out of `total`, and the `results` collected so far. Results stream in as they finish, so this is safe to poll while `status` is `processing`. Each result carries its own `status`, so check per item rather than assuming a finished job means every URL succeeded. Jobs are scoped to the key that created them; polling someone else's job returns 403. Unmetered.
Poll the status of a crawl job
Return the current state of a crawl job: `status`, `completed` out of `total` pages, and the `results` gathered so far. Safe to poll while `status` is `processing` — results stream in as pages finish. Check `truncated` and `truncated_reason` on a completed job to tell a full crawl from one that stopped at a page cap or an exhausted quota. Jobs are scoped to the key that created them. Unmetered.