FlyCrawl Universal Developer & AI Hub

Existing applications built with the official firecrawl-py Python SDK or @mendable/firecrawl-js can target FlyCrawl directly by setting api_url="https://flycrawl.net/api/v1". Below is our verified endpoint and parameter support matrix tested via tests/test_firecrawl_compat.py:

# Python Example: Pre-Flight Estimation & Graceful Insufficient Credits Handling
import requests

res = requests.post("https://flycrawl.net/api/v1/crawl",
    headers={"Authorization": "Bearer fly_live_your_key"},
    json={"url": "https://target-site.com", "limit": 100, "dryRun": True}
)

if res.status_code == 200:
    data = res.json()
    print(f"Site requires {data['estimated_credits']} credits. Can afford: {data['can_afford']}")
    if data["can_afford"]:
        # Execute actual crawl:
        job = requests.post("https://flycrawl.net/api/v1/crawl",
            headers={"Authorization": "Bearer fly_live_your_key"},
            json={"url": "https://target-site.com", "limit": 100}
        ).json()
        print(f"Crawl job started: {job['id']}")
elif res.status_code == 402:
    err = res.json()
    print(f"Insufficient balance! Shortage: {err['shortage']} credits. Suggested max pages: {err['suggested_max_pages']}")

Error Handling & Zero-Risk Fair Billing Guarantee

When building autonomous AI agents or production web scrapers, encountering edge-case errors on external target websites is inevitable. FlyCrawl operates on a strict Fair Billing Guarantee: you are NEVER billed for target server downtime, timeouts, or unsolved bot challenges.

API Error Codes & Billing Behavior

Production Agent Retry Best Practice (Python / TypeScript)

Configure your HTTP client to automatically retry on 429 and 504 with jittered exponential backoff:

from tenacity import retry, stop_after_attempt, wait_exponential_jitter, retry_if_exception
import httpx

# Retry up to 3 times ONLY on 429 or 504 (Fair billing guarantees zero extra costs)
@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential_jitter(initial=1, max=5),
    retry=retry_if_exception(lambda e: isinstance(e, httpx.HTTPStatusError) and e.response.status_code in (429, 504))
)
def robust_scrape(url: str):
    with httpx.Client(base_url="https://api.flycrawl.net/v1") as client:
        res = client.post("/scrape", json={"url": url}, headers={"Authorization": "Bearer YOUR_KEY"})
        res.raise_for_status()
        return res.json()

Search Engine Architecture & Live Index Freshness

Unlike traditional search APIs that return bloated affiliate links, promoted ads, or stale page snippets, FlyCrawl's /search endpoint is specifically engineered for LLM agents: performing real-time discovery and pre-extracting substantive page bodies into clean Markdown in one single action.

Dual-Engine Vector Architecture: Neural Embeddings & Fast Lexical Projection

FlyCrawl offers a robust dual-engine embedding architecture designed for production AI and RAG pipelines. Depending on your latency and semantic requirements, you can generate standard Neural Transformer Embeddings (via OpenAI text-embedding-3 with managed credits or BYOK) for deep semantic comprehension, or utilize FlyCrawl's Fast Lexical Projection vectors computed sub-10ms in-memory for instant deduplication and fast keyword clustering with zero API overhead.

🐍 Production RAG Implementation: Scrape with FlyCrawl, Index in Qdrant & Query

Extract noise-free Fit-Markdown along with neural embeddings in a single HTTP request, index into your vector database, and query seamlessly with standard SDKs:

import httpx
from openai import OpenAI
from qdrant_client import QdrantClient
from qdrant_client.models import PointStruct, Distance, VectorParams

qdrant = QdrantClient(url="http://localhost:6333")
qdrant.recreate_collection("tech_docs", vectors_config=VectorParams(size=1536, distance=Distance.COSINE))

# 1. INGESTION TIME: Extract Fit-Markdown + neural embeddings via FlyCrawl in 1 API call
res = httpx.post(
    "https://api.flycrawl.net/v1/scrape",
    headers={"Authorization": "Bearer fly_live_your_key"},
    json={
        "url": "https://example.com/docs",
        "formats": ["markdown", "vector"],
        "embedding_model": "text-embedding-3-small" # Neural embeddings (or "native" for lexical)
    }
).json()

doc_vector = res["data"]["vector"] # 1536-dim float array
qdrant.upsert("tech_docs", points=[PointStruct(id=1, vector=doc_vector, payload={"url": "https://example.com/docs"})])

# 2. QUERY TIME: Embed user question using standard OpenAI client
ai_client = OpenAI()
query_vector = ai_client.embeddings.create(
    model="text-embedding-3-small",
    input="How do I configure authentication?"
).data[0].embedding

# 3. Semantic cosine retrieval in vector database
search_results = qdrant.search("tech_docs", query_vector=query_vector, limit=3)
print(f"Top matched document: {search_results[0].payload['url']}")

Residential Proxy Geographic Coverage & Scalability Limits

To maintain strict QoS and predictable throughput for all users, FlyCrawl establishes clear infrastructure tiers for residential proxy routing and parallel crawl jobs:

Persistent Browser Sessions & Authenticated Crawling

Extract pages behind authentication walls, internal dashboards, and gated enterprise portals without exposing credentials on every request. FlyCrawl executes automated login action sequences (clicks, keystrokes, waits), securely captures the Playwright storageState (cookies, localStorage), and stores it encrypted with AES-256-GCM. Simply pass session_id to scrape private pages seamlessly.

# Step 1: Initialize session with automated login actions
curl -X POST "https://api.flycrawl.net/v1/sessions" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://news.ycombinator.com/login",
    "actions": [
      { "type": "write", "selector": "input[name=acct]", "text": "my_user" },
      { "type": "write", "selector": "input[name=pw]", "text": "secret_pass" },
      { "type": "click", "selector": "input[value=login]" },
      { "type": "wait", "milliseconds": 2000 }
    ],
    "ttl_hours": 168
  }'

# Response: { "success": true, "session_id": "sess_9f8a7b6c5d4e...", "expires_at": "..." }

# Step 2: Scrape private authenticated URLs using session_id
curl -X POST "https://api.flycrawl.net/v1/scrape" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://news.ycombinator.com/threads?id=my_user",
    "session_id": "sess_9f8a7b6c5d4e..."
  }'

Autonomous Goal-Driven Multi-Site Agent (/api/v1/agent)

Provide a high-level research goal or comparison schema. FlyCrawl's autonomous ReAct agent formulates targeted web queries, discovers leading sources, crawls multiple domains in parallel, evaluates relevance, and synthesizes structured JSON matching your schema. Supports real-time Server-Sent Events (SSE) streaming for progress visibility.

# Python Example: Autonomous Agent with Target JSON Schema
import requests

payload = {
    "goal": "Compare pricing tiers, rate limits, and LLM features between Firecrawl and Spider",
    "schema": {
        "providers": [
            {
                "name": "string",
                "starting_price": "string",
                "concurrent_rate_limit": "string",
                "markdown_quality": "string",
                "unique_features": ["string"]
            }
        ]
    },
    "max_steps": 5
}

res = requests.post(
    "https://api.flycrawl.net/v1/agent",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json=payload
).json()

print("Agent Summary:", res["summary"])
print("Synthesized Data:", res["data"])
print("Sources Visited:", res["sources"])

Smart Website Change Monitoring & Semantic Diff Engine (/api/v1/monitor)

Monitor competitor pricing, product availability, legal terms, or regulatory publications. FlyCrawl computes SHA-256 content hashes on every check. If the content has not changed, exactly ZERO credits are deducted. When modifications occur, our LLM generates a concise semantic explanation of what changed and triggers an HMAC-signed webhook to your server.

# Create a Change Monitor on Pricing Table
curl -X POST "https://api.flycrawl.net/v1/monitor" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/pricing",
    "selector": ".pricing-grid",
    "diff_mode": "semantic",
    "frequency": "daily",
    "webhook_url": "https://api.yourdomain.com/webhooks/pricing-change"
  }'

# Webhook payload dispatched upon detected change:
{
  "event": "monitor.change_detected",
  "monitor_id": "mon_4a3b2c1d...",
  "url": "https://example.com/pricing",
  "detected_at": "2026-10-26T12:00:00Z",
  "summary": "• Pro tier price increased from $49/mo to $59/mo.\n• Added 50,000 monthly credits.\n• Annual billing discount changed from 20% to 25%.",
  "diff": "--- previous\n+++ current\n- Pro: $49/mo\n+ Pro: $59/mo",
  "previous_hash": "a1b2c3...",
  "new_hash": "d4e5f6..."
}