crawl-preflight.postnov01.workers.dev

Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and how much is actually there. Returns per-bot robots.txt verdicts for 16 AI crawlers, crawlable page count from sitemaps, detected CDN and paid-access signals (x402, TollBit), plus a fetch/skip recommendation. Call before spending requests on an unknown domain.

Endpoint

https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain

Price

$0.02 / call

Settles via

cdp

Category

—

Last seen live

never

Last probed

Tue, 08 Sep 2026 05:24:04 UTC

First indexed

Tue, 08 Sep 2026 05:22:24 UTC

Sources

cdp-x402

Probe history

Last 1 probes, newest on the right. 0% answered with a payment challenge.

JSON: /api/v1/services/crawl-preflight.postnov01.workers.dev. Own this service? Update the listing.