Idempotency
Retry a bulk or crawl submission safely — the same key returns the original job instead of enqueuing a second one or charging twice.
POST /bulk and POST /crawl accept an Idempotency-Key header. Send the same key on a retry
and you get the original job back: the request is never enqueued twice, and your quota is
never charged twice.
Idempotency-Key: 6f9619ff-8b86-d011-b42d-00cf4fc964ffUse any unique string per logical submission. A UUID is ideal.
Keys are scoped to your account and live for 6 hours — the same TTL as the job they point at.
What a retry returns
| The retry sends | You get |
|---|---|
| An identical body | The original job_id, unchanged. No second job, no second charge. |
| A different body | 422 idempotency_key_reuse — a client bug, surfaced rather than swallowed. |
| Anything, while the first is still enqueuing | 409 idempotency_in_progress — retryable; retry shortly for the id. |
Why the 409 is retryable
It is the only 409 on the API, and it must be retryable. The same key and body arrived while
the first submission was still enqueuing, which resolves in milliseconds. An agent that treated
it as fatal would give up on a request that is about to succeed.
Example
curl -X POST https://api.wellmarked.io/bulk \
-H "Authorization: Bearer wm_your_api_key_here" \
-H "Idempotency-Key: 6f9619ff-8b86-d011-b42d-00cf4fc964ff" \
-H "Content-Type: application/json" \
-d '{"urls": ["https://a.example", "https://b.example"]}'Run that twice and you get one job.
The SDKs already do this
A submission that fails on a retryable error is retried internally reusing the same key, so
an SDK-level retry can never double-submit. Tune the attempt count with the max_retries
client option (default 2).
That is the reason to prefer the SDK over hand-rolled retry logic: getting this wrong costs real money, quietly.