How does FlyCrawl empower Machine Learning (ML) and AI training workflows?

FlyCrawl provides high-throughput web scraping to construct noise-free, deduplicated, and token-optimized pretraining and fine-tuning datasets for Machine Learning (ML), open-weight LLMs, and domain-specific AI models.

Can FlyCrawl automate RAG pipelines and vector embeddings generation?

Yes! FlyCrawl automatically scrapes live web content, cleans HTML to Fit-Markdown, chunks text semantically, and generates native 1536-dimensional and 3072-dimensional vector embeddings ready for direct upsert into Pinecone, Qdrant, ChromaDB, and Milvus without requiring external API keys.

What makes FlyCrawl the ideal Web Scraping SaaS for modern developers?

FlyCrawl is a cloud-native scraping platform delivering 342ms static median latency, worker process recycling with bounded memory, resilient proxy handling, flexible developer billing, and native Model Context Protocol (MCP) server support.

What payment methods does FlyCrawl support?

FlyCrawl supports flexible developer billing including multi-currency digital assets (USDT, TON, TRX, BTC, ETH) and enterprise invoices, featuring automated instant credit top-ups without unnecessary verification hurdles.

Why is FlyCrawl a compelling alternative to Firecrawl?

FlyCrawl is engineered with a compiled Go and .NET core delivering fast extraction (342ms static p50), compatible API endpoints (/v1/scrape, /v1/crawl), native 1536/3072-dim embeddings, and transparent pay-as-you-go pricing without lock-in.

Can I connect FlyCrawl to Cursor IDE, Windsurf, or Claude Desktop?

Yes! FlyCrawl hosts a native Model Context Protocol (MCP) server at https://api.flycrawl.net/v1/mcp enabling AI coding agents to crawl, search, and ingest web knowledge seamlessly.

Product Changelog & Release Notes

v1.5.0

Enterprise llms.txt Generator & Dual-Engine Vector Architecture

Late October 2026

Enterprise llms.txt & llms-full.txt Generator: Automated domain scanning, semantic section hierarchy, and free hosted CDN URLs (/llms/{slug}.txt) for Shopify, Wix, and WordPress.
Dual-Engine Vector Architecture: Fully separated Neural Transformer Embeddings (OpenAI BYOK / Managed) for deep RAG from Fast Lexical Projections for sub-10ms in-memory deduplication.
Google Page 1 Schema Triple-Stack: Comprehensive WebApplication, HowTo, and FAQPage JSON-LD structured data.

v1.4.0

Verifiable Benchmarks & Unified Pricing Engine

Mid October 2026

Reproducible Benchmark Suite: Published open methodology in /benchmarks measuring static p50 latency (342ms), token reduction (83.2% avg), and 24-hour continuous memory soak profiling.
Transparent Cost Calculator: Added interactive client-side cost calculator comparing FlyCrawl ($0.40–$0.97/1k) with competitor rates based on published pricing.
Refund & Credit Protection Policy: Formally published clear refund terms and zero-deduction guarantee on failed/blocked responses at /RefundPolicy.

v1.3.0

Model Context Protocol (MCP) Server

Early October 2026

Native MCP Server: Introduced /api/v1/mcp endpoint directly usable in Cursor IDE, Claude Desktop, and Windsurf without intermediate adapters.
Tool Schemas: Added MCP tools: scrape_url, crawl_site, and search_web with full input validation.
SSE Live Streaming: Streaming feedback for MCP clients executing long-running crawl jobs.

v1.2.0

Fit-Markdown & Multilingual Bidi Engine

Late September 2026

Bidi & CJK Layout Reconstruction: Complete unicode support preserving RTL reading order for Persian and Arabic text and ideographic line breaking for Chinese, Japanese, and Korean.
Fit-Markdown Parser: Proprietary semantic DOM reduction algorithm achieving over 80% reduction in raw prompt tokens while preserving structural headings, tables, and lists.

v1.1.0

Batch Scraping & Vector Embeddings

Mid September 2026

Batch Scraping (/v1/batch/scrape): Submit up to 50 URLs in one request with concurrent asynchronous worker distribution.
Direct Vector Generation (/v1/vector): Output 1536-dim or 3072-dim embeddings ready for direct upsert to Pinecone, Qdrant, or Milvus.

v1.0.0

Initial Public API Release

Early September 2026

Core REST Endpoints: /v1/scrape, /v1/crawl, and /v1/map.
Wire Compatibility: Full drop-in compatibility allowing api_url override with official firecrawl-py and @mendable/firecrawl-js clients.
Developer Billing: Instant crypto checkout for USDT (TRC20, TON) and TON native coin without mandatory KYC.

Product Changelog