Poll the status of a crawl job
Return the current state of a crawl job: status, completed out of total pages, and the results gathered so far.
Return the current state of a crawl job: status, completed out of total pages, and the results gathered so far.
Safe to poll while status is processing — results stream in as pages finish. Check truncated and truncated_reason on a completed job to tell a full crawl from one that stopped at a page cap or an exhausted quota.
Jobs are scoped to the key that created them. Unmetered.
Authorization
BearerAuth Your API key, sent as Authorization: Bearer wm_.... Keys are wm_ followed by 40 hex characters. Create one with POST /register (free, no auth) or POST /keys.
In: header
Path Parameters
uuidResponse Body
application/json
application/json
application/json
application/json
application/json
application/json
curl -X GET "https://api.wellmarked.io/crawl/497f6eca-6276-4993-bfeb-53cbbbba6f08" \ -H "Authorization: Bearer wm_your_api_key_here"{ "completed": 0, "created_at": "2019-08-24T14:15:22Z", "finished_at": "2019-08-24T14:15:22Z", "job_id": "453bd7d7-5355-4d6d-a38e-d9e7eb218c3f", "kind": "crawl", "results": [ { "blocks": [ { "level": 0, "text": "string", "type": "heading" } ], "chunks": [ { "end_token": 0, "start_token": 0, "text": "string" } ], "depth": 0, "error": "string", "html": "string", "links": [ "string" ], "markdown": "string", "metadata": { "author": "string", "date": "string", "retrieved_at": "2019-08-24T14:15:22Z", "title": "string", "url": "string" }, "metrics": { "content_bytes": 0, "input_tokens": 0, "output_tokens": 0, "reduction_pct": 0, "tokens_saved": 0 }, "url": "string" } ], "status": "string", "total": 0, "truncated": false, "truncated_reason": "string"}Crawl a site starting from a root URL
Follow links from a starting URL and extract every page reached, up to max_depth and max_pages. Returns a job_id immediately; crawling is asynchronous.
Search the web and extract the results to Markdown
Run a web search and return the result pages already extracted to clean Markdown — a query in, extracted content out, in one round trip.