Poll the status of a crawl job
Return the current state of a crawl job: `status`, `completed` out of `total` pages, and the `results` gathered so far. Safe to poll while `status` is `processing` — results stream in as pages finish. Check `truncated` and `truncated_reason` on a completed job to tell a full crawl from one that stopped at a page cap or an exhausted quota. Jobs are scoped to the key that created them. Unmetered.
Authorization
BearerAuth Your API key, sent as Authorization: Bearer wm_.... Keys are wm_ followed by 40 hex characters. Create one with POST /register (free, no auth) or POST /keys.
In: header
Path Parameters
uuidResponse Body
application/json
application/json
application/json
application/json
application/json
application/json
curl -X GET "https://example.com/crawl/497f6eca-6276-4993-bfeb-53cbbbba6f08"{ "completed": 0, "created_at": "2019-08-24T14:15:22Z", "finished_at": "2019-08-24T14:15:22Z", "job_id": "453bd7d7-5355-4d6d-a38e-d9e7eb218c3f", "kind": "crawl", "results": [ { "blocks": [ { "level": 0, "text": "string", "type": "heading" } ], "chunks": [ { "end_token": 0, "start_token": 0, "text": "string" } ], "depth": 0, "error": "string", "html": "string", "links": [ "string" ], "markdown": "string", "metadata": { "author": "string", "date": "string", "retrieved_at": "2019-08-24T14:15:22Z", "title": "string", "url": "string" }, "metrics": { "content_bytes": 0, "input_tokens": 0, "output_tokens": 0, "reduction_pct": 0, "tokens_saved": 0 }, "url": "string" } ], "status": "string", "total": 0, "truncated": false, "truncated_reason": "string"}Crawl a site starting from a root URL
Follow links from a starting URL and extract every page reached, up to `max_depth` and `max_pages`. Returns a `job_id` immediately; crawling is asynchronous. Poll `GET /crawl/{job_id}` or supply `webhook_url`. The crawler stays on the starting host, honours `robots.txt`, and applies your key's domain policy to every link it follows — not just the root. A crawl stops early rather than failing when it hits a ceiling: `truncated` is set with a `truncated_reason` of `page_cap_reached` or `quota_exhausted`, and the pages gathered so far are still returned. `max_depth` is capped by plan. Metered per page extracted, not per page discovered.
Search the web and extract the results to Markdown
Run a web search and return the result pages already extracted to clean Markdown — a query in, extracted content out, in one round trip. This endpoint is **synchronous**: there is no job to poll. Results are partial rather than all-or-nothing — each item carries its own `status`, so a page that times out is reported alongside the ones that succeeded instead of failing the whole call. `num_results` is clamped by your plan and by the search provider's own ceiling. Metered as 1 for the query plus 1 per page returned, so a 5-result search costs 6 requests.