Crawlbase

Best for: Prototypes and side projects - free to start, no sales call; AI agents and automation - an agent-ready surface (MCP / llms.txt); Teams needing broad API coverage out of the box

Avoid if: You have strict compliance requirements

• ███████▒░░
  75 / 100
  Agent friendliness
• ██████████
  100 / 100
  Pricing transparency
• ████████▒░
  85 / 100
  Setup speed
• ██████░░░░
  60 / 100
  Docs quality
• ██████████
  100 / 100
  Procurement ease
• ███▒░░░░░░
  35 / 100
  Trust readiness

Scores are computed deterministically from this profile's published, sourced fields (pricing, compliance, capabilities, docs and developer-surface signals) - not from reviews or paid placement. Each axis is 0-100; an unknown signal scores 0 for that axis. Procurement ease is the inverse of buying friction (higher = easier to adopt).

Pricing model

Hybrid (base + usage) [2]crawlbase.com/pricing“Crawling API: pay-per-request with complexity tiers (Standard $3/1k, Moderate $4.50/1k, Complex $6/1k); Smart AI Proxy: Starter $149/month, Advanced $229/month, Premium $449/month; Cloud Storage: Developer $29/month, Business $249/month”crawlbase.com/blog/crawlbase-pricing-explained/“the first billing cycle reflecting the new rates starts on January 1, 2025”

Published pricing

✓ Yes [3]crawlbase.com/pricing“Standard: $3.00 per 1,000 requests; Moderate: $4.50 per 1,000 requests; Complex: $6.00 per 1,000 requests; LinkedIn: $15 per 1,000 requests; Starter: $149/month; Advanced: $229/month; Premium: $449/month”

Free tier

✓ Yes [4]crawlbase.com/cloud-storage-for-crawling-and-scraping“10k requests, 14 days retention period”crawlbase.com/pricing“Free Trial: $0, 5,000 credits, 5 threads; Cloud Storage Free: $0, 10k requests, 14-day retention”

Free tier details

Cloud Storage Free plan: $0/month with 10,000 requests and 14-day retention (recurring tier). Smart AI Proxy Free Trial plan: $0/month with 5,000 credits and 5 threads (labeled 'Free Trial' on pricing page). Crawling API: 1,000 free requests on signup (one-time, not recurring). [5]crawlbase.com/signup“Every new account starts with 1,000 free successful requests and no credit card”crawlbase.com/pricing“Smart AI Proxy Free Trial: $0/month, 5,000 credits, 5 threads; Cloud Storage Free: $0/month, 10,000 requests, 14-day retention”

Self-serve signup

✓ Yes [6]crawlbase.com/signup“Every new account starts with 1,000 free successful requests and no credit card”

Requires sales call

✗ No

Enterprise plan

✓ Yes [7]crawlbase.com/pricing“Enterprise Crawler: Custom asynchronous crawling for large-scale projects. Pricing requires contacting sales; features include multiple delivery options and priority support. Cloud Storage Enterprise: Custom pricing”

• JavaScript rendering
• Residential proxies
• Structured / AI extraction
• Site crawling
• SERP scraping
• Anti-bot bypass

Supported actions

scrape, crawl, js_rendering, captcha_solving, anti_bot_bypass, screenshot, pdf_rendering, structured_data_extraction, prebuilt_scrapers, async_jobs, proxy_rotation, residential_proxies, datacenter_proxies, markdown_output, ai_extraction, cloud_storage, webhook_delivery, geo_targeting, sticky_sessions, tor_network [8]crawlbase.com/docs/crawling-api/parameters/“screenshot: Boolean for JPEG capture; pdf: Boolean to render as PDF file instead of HTML; format: html | json | md; scraper: Apply built-in structured data extractor; async: Boolean to queue request; callback: Webhook URL to receive crawl results; store: Boolean to persist crawled page in Cloud Storage”crawlbase.com/docs/crawling-api/“Solves Cloudflare, PerimeterX, DataDome, hCaptcha, and other common challenges”

Regions

140M rotating residential proxies, 98M datacenter proxies, 195 countries supported for geo-targeting (Crawling API), 45+ countries supported (Smart AI Proxy Premium), 23 countries supported for country routing parameter

Input types

target URL, country/geo code, device type, custom headers, cookies, CSS click selector, prebuilt scraper id, render JS flag, async flag, callback/webhook URL, output format flag, scroll flag, screenshot flag

Output types

raw HTML, rendered HTML, Markdown, JSON (structured), screenshot (JPEG), PDF, parsed fields (prebuilt scrapers), webhook delivery, cloud-stored pages

Webhooks

✓ Yes [9]crawlbase.com/docs/crawling-api/parameters/“callback: Webhook URL to receive crawl results”

Sandbox / test mode

✗ No [10]crawlbase.com/signup“Every new account starts with 1,000 free successful requests and no credit card”

SDK languages

Python, Node.js, Ruby, PHP, Go, Java, C#/.NET [11]crawlbase.com/docs/crawling-api/“SDK Languages: Python, Node.js, Ruby, PHP, Go, Java, C#/.NET”

MCP server

✓ Yes [12]crawlbase.com/mcp“Connect Claude, Cursor, Windsurf & other MCP agents to live web data with just one config.”

SOC 2

None [13]crawlbase.com/privacy“Only GDPR compliance is explicitly referenced. No other certifications (SOC2, ISO 27001, HIPAA, or PCI DSS) are mentioned in this privacy policy document.”crawlbase.com/enterprise“No additional SLA metrics, performance guarantees, or industry-specific certifications (ISO, SOC 2, etc.) are mentioned in the provided content.”

HIPAA

✗ No [14]crawlbase.com/privacy“Only GDPR compliance is explicitly referenced. No other certifications (SOC2, ISO 27001, HIPAA, or PCI DSS) are mentioned in this privacy policy document.”

GDPR

✓ Yes [15]crawlbase.com/enterprise“GDPR Compliant: Fully committed to General Data Protection Regulation standards for EU users”crawlbase.com/privacy“Crawlbase is GDPR compliant and you can find more information about it below.”

ISO 27001

✗ No [16]crawlbase.com/privacy“Only GDPR compliance is explicitly referenced. No other certifications (SOC2, ISO 27001, HIPAA, or PCI DSS) are mentioned”crawlbase.com/enterprise“No additional SLA metrics, performance guarantees, or industry-specific certifications (ISO, SOC 2, etc.) are mentioned”

PCI DSS

✗ No [17]crawlbase.com/privacy“Only GDPR compliance is explicitly referenced. No other certifications (SOC2, ISO 27001, HIPAA, or PCI DSS) are mentioned in this privacy policy document.”

Published SLA

✓ Yes [18]crawlbase.com/enterprise“99.9% uptime guarantee and 90+% success rate”crawlbase.com/terms“We do not commit to ensuring that the website will be available at any particular time.”

Rate limits

Default concurrency limit in Crawling API proxy mode is 20 requests per second (~1.7M req/day). Smart AI Proxy thread limits by plan: Free=5, Starter=20, Advanced=40, Premium=80. [19]crawlbase.com/docs/crawling-api/response-codes/“Default rate limit in proxy mode is 20 requests per second (~1.7M req/day).”crawlbase.com/smart-proxy“Thread Limits by Plan: Free: 5 threads, Starter: 20 threads, Advanced: 40 threads, Premium: 80 threads”

Known restrictions

LinkedIn crawling limited to public pages only and excluded from free credits, POST/PUT methods: normal token only; accounts caught using POST for spam, credential stuffing, or other malicious traffic will be suspended, Prohibited uses: spam, comment injection, fraudulent form submissions, scripted account creation, credential stuffing, Scroll billing: first 8 seconds = 1 request; each additional 5 seconds adds 1 billed request, Only pay for successful requests (pc_status=200 AND original_status in 200,201,204,301,302,404,410), Google SERP scraper is a prebuilt scraper within the Crawling API, not a standalone SERP API product [20]crawlbase.com/pricing“LinkedIn charges fixed $15 per 1,000 requests with restrictions on public pages only; free credits excluded”crawlbase.com/docs/crawling-api/response-codes/“Scroll billing: First 8 seconds = 1 request; every additional 5 seconds adds 1 more billed request. Only billable when both conditions met: pc_status = 200 AND original_status is one of: 200, 201, 204, 301, 302, 404, or 410”

• Docs
• API reference
• llms.txt
• Status page

Docs rendering: static · markdown variants served · llms.txt present

API style

rest

Base URL

https://api.crawlbase.com/

Versioning

none

Stability

ga

Auth methods

api_key

Idempotency keys

✗ No

Error format

vendor-specific

Rate limit

20 / concurrent

SDKs

• Python crawlbase · repo
• Node.js crawlbase · repo
• Ruby crawlbase · repo
• PHP crawlbase/crawlbase · repo
• Go github.com/crawlbase/crawlbase-go · repo
• Java com.crawlbase:crawlbase-java · repo
• C#/.NET CrawlbaseAPI · repo

• Auth docs
• Quickstart
• MCP server

Launched

2017-01-01

Notable customers

Intel, Pinterest, Airbnb, Honda, Amgen, Huawei, Shopify, Expedia, H&M, Nike, Oracle, Stanford University, LG

• ScrapFly

  "Scrape any site, drive any browser, power any agent. One API key."

  Subscription · public pricing · self-serve

• Bright Data Web Scraper API

  "The most reliable Web Scraping API. Scrape any website with automatic proxy rotation, anti-bot bypass, and JavaScript rendering."

  Subscription · free tier · public pricing · self-serve

• Oxylabs

  The best proxy service platform with 175M+ Residential and 2M Datacenter IP proxies. Extract public data from any website with ease!

  Hybrid · free tier · public pricing · self-serve

• Apify

  Cloud platform for web scraping, browser automation, AI agents, and data for AI

  Hybrid · free tier · public pricing · self-serve

• Diffbot

  Web Data for your AI

  Hybrid · free tier · public pricing · self-serve

• Firecrawl

  The API to search, scrape, and interact with the web at scale.

  Subscription · free tier · public pricing · self-serve

Crawlbase alternatives · Crawlbase vs ScrapFly · All Scraping & Crawling APIs APIs

Each field above carries a numbered source - hover for a preview, click to jump here.

1. ↑Description: crawlbase.com
2. ↑Pricing model: crawlbase.com · crawlbase.com
3. ↑Published pricing: crawlbase.com
4. ↑Free tier: crawlbase.com · crawlbase.com
5. ↑Free tier details: crawlbase.com · crawlbase.com
6. ↑Self-serve signup: crawlbase.com
7. ↑Enterprise plan: crawlbase.com
8. ↑Supported actions: crawlbase.com · crawlbase.com
9. ↑Webhooks: crawlbase.com
10. ↑Sandbox: crawlbase.com
11. ↑SDK languages: crawlbase.com
12. ↑MCP server: crawlbase.com
13. ↑SOC 2: crawlbase.com · crawlbase.com
14. ↑HIPAA: crawlbase.com
15. ↑GDPR: crawlbase.com · crawlbase.com
16. ↑ISO 27001: crawlbase.com · crawlbase.com
17. ↑PCI DSS: crawlbase.com
18. ↑Published SLA: crawlbase.com · crawlbase.com
19. ↑Rate limits: crawlbase.com · crawlbase.com
20. ↑Known restrictions: crawlbase.com · crawlbase.com

Every field change, who made it, and when - from our audited data pipeline and editors.

1. 2026-06-15 Score Docs Quality: 25 → 60
2. 2026-06-15 Score Agent Friendliness: 45 → 75
3. 2026-06-14 Markdown Docs Served: (none) → Yes
4. 2026-06-14 API Reference URL: (none) → https://crawlbase.com/docs/api-reference
5. 2026-06-14 Robots Allows Agents: (none) → Yes
6. 2026-06-14 Has Structured Data: (none) → Yes
7. 2026-06-14 Markdown Docs URL: (none) → https://crawlbase.com/docs.md
8. 2026-06-14 Capabilities: {} → {"serp":true,"crawl":true,"anti_bot":true,"js_rendering":true,"residential_prox…
9. 2026-06-14 Summary Md: (none) → Crawlbase is a web data infrastructure platform, launched in 2017, that provide…
10. 2026-06-14 Score Docs Quality: (none) → 25
11. 2026-06-14 Score Procurement Friction: (none) → 100
12. 2026-06-14 Score Trust Readiness: (none) → 35
13. 2026-06-14 Best For: (none) → Prototypes and side projects - free to start, no sales call, AI agents and auto…
14. 2026-06-14 Avoid If: (none) → You have strict compliance requirements
15. 2026-06-14 Scoring Methodology: (none) → Scores are computed deterministically from this profile's published, sourced fi…
16. 2026-06-14 Score Setup Speed: (none) → 85
17. 2026-06-14 Score Pricing Transparency: (none) → 100
18. 2026-06-14 Score Agent Friendliness: (none) → 45
19. 2026-06-14 Docs URL: (none) → https://crawlbase.com/docs
20. 2026-06-14 Status Page URL: (none) → https://status.crawlbase.com
21. 2026-06-14 Rendering: (none) → static
22. 2026-06-14 Llms Txt URL: (none) → https://crawlbase.com/llms.txt
23. 2026-06-14 Llms Txt Present: (none) → Yes
24. 2026-06-14 Enterprise Plan Available: set to Yes
25. 2026-06-14 SOC 2: set to none
26. 2026-06-14 HIPAA: set to No
27. 2026-06-14 GDPR: set to Yes
28. 2026-06-14 PCI DSS: set to No
29. 2026-06-14 SLA Published: set to Yes
30. 2026-06-14 Data Retention Policy URL: set to https://crawlbase.com/privacy
31. 2026-06-14 Documented Rate Limits: set to Default concurrency limit in Crawling API proxy mode is 20 requests per second …
32. 2026-06-14 Rate Limit Requests: set to 20
33. 2026-06-14 Rate Limit Window: set to concurrent
34. 2026-06-14 Known Restrictions: set to LinkedIn crawling limited to public pages only and excluded from free credits, …
35. 2026-06-14 Auth Methods: set to api_key
36. 2026-06-14 Auth Docs URL: set to https://crawlbase.com/docs/authentication
37. 2026-06-14 API Style: set to rest
38. 2026-06-14 Base URL: set to https://api.crawlbase.com/
39. 2026-06-14 Versioning Scheme: set to none
40. 2026-06-14 Stability: set to ga
41. 2026-06-14 MCP URL: set to https://crawlbase.com/mcp
42. 2026-06-14 Quickstart URL: set to https://crawlbase.com/docs/quick-start
43. 2026-06-14 Idempotency Supported: set to No
44. 2026-06-14 Error Format: set to vendor-specific
45. 2026-06-14 Requires Verification: set to No
46. 2026-06-14 Starting Price Usd: set to 3
47. 2026-06-14 Price Basis: set to 1,000 requests
48. 2026-06-14 Free Tier Limit: set to 1,000 requests (no credit card required)
49. 2026-06-14 ISO 27001: set to No
50. 2026-06-14 Notable Customers: set to Intel, Pinterest, Airbnb, Honda, Amgen, Huawei, Shopify, Expedia, H&M, Nike, Or…

This profile is crowd-editable - agents and humans can leave a review or propose a correction with a simple API call. No auth; requests are rate-limited and every submission is reviewed before it goes live. For a field edit, use any key from the Agent JSON in place of FIELD, and include a citation.

curl -X POST https://apio.sh/api/feedback/crawlbase \
  -H 'Content-Type: application/json' \
  -d '{"kind":"review","rating":5,"body":"Your experience with this API…"}'

curl -X POST https://apio.sh/api/suggest/crawlbase/FIELD \
  -H 'Content-Type: application/json' \
  -d '{"value":"corrected value","citations":[{"url":"https://source.example/page","excerpt":"supporting quote"}],"note":"what changed and why"}'

All the ways to contribute →