Compliance and policy

Every key carries an enforced policy — allowed domains, denied patterns, robots.txt handling — that a request can tighten but never loosen.

Every API key carries a compliance policy constraining what it may fetch. Hand an agent a bounded key and it stays bounded: the policy is an enforced floor that a request can tighten but never loosen.

Policy fields

Field	Effect
`allow_domains`	If non-empty, only these hosts and their subdomains may be fetched.
`deny_patterns`	Glob patterns. A URL whose hostname or full URL matches any pattern is refused.
`respect_robots`	`"strict"` (default) honours `robots.txt` on `/extract` and `/bulk` too, not just crawl. `"lax"` limits robots enforcement to crawl.

Per-request overrides are narrow-only

/extract, /bulk, and /crawl accept the same three fields in the request body:

{
  "url": "https://docs.example.com/page",
  "allow_domains": ["example.com"],
  "deny_patterns": ["*/admin/*"],
  "respect_robots": "strict"
}

They can only make the key's policy more restrictive — add deny patterns, narrow to a subset of the key's allowed domains, or upgrade lax → strict.

Loosening attempts are ignored, not rejected

A request naming a domain the key does not allow, or attempting a strict → lax downgrade, is silently ignored rather than erroring. The key's floor holds. Do not use a 200 response as evidence that a broadening override took effect — it did not.

Where denials surface

On /extract, a policy denial fails the call with 403 — domain_not_allowed, domain_denied, or robots_disallowed.
On /bulk and /crawl, the denial appears per item in results[].error instead. The job itself succeeds.

Scoped keys

An account can hold many keys, each restricted to a subset of scopes. See Authentication for the scope table.

Operation	Endpoint
Create	`POST /keys` — requires the `keys` scope; a key can only grant scopes it holds
List	`GET /keys` — metadata only, never the raw key
Revoke	`DELETE /keys/{id}` — immediate

Audit log

GET /logs returns your own request history, newest first. Paginate with limit and offset; has_more signals another page.

Each row records which key ran the request (key_id) and how its policy decided (policy_decision: allowed, domain_not_allowed, domain_denied, or robots_disallowed).

It does not count toward your monthly quota.