Home / Integrations / Dify.ai
🤖 LLMOPS & AGENT TOOL • DIFY.AI INTEGRATION

Live Web Scraping & RAG for Dify.ai Agents

Empower your Dify ChatFlows, Workflows, and Autonomous Agents with real-time web browsing. Ingest clean Fit-Markdown directly into DeepSeek-R1, Qwen 2.5, Claude, or GPT-4o with zero proxy friction.

Get Free API Key →
OpenAPI 3.0
1-Click Custom Tool
DeepSeek-R1
Native reasoning fit
95%
Context token reduction
Dify Custom Tool Schema (YAML) 
openapi: 3.0.1
info:
  title: FlyCrawl Scraping Engine
  description: Extract clean Markdown from any website for Dify LLM reasoning
  version: 1.0.0
servers:
  - url: https://api.flycrawl.net/v1
paths:
  /scrape:
    post:
      operationId: flycrawl_scrape
      summary: Scrape web page content
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              required: [url]
              properties:
                url:
                  type: string
                  description: Full URL to scrape
                formats:
                  type: array
                  items: { type: string }
                  default: ["markdown"]
🔄 DIFY AGENTIC ARCHITECTURE

How Dify.ai Interacts with FlyCrawl in Real Time

End-to-end flow from user query to autonomous tool calling and structured reasoning.

💬
1. User Query
User asks for live data or site summary
🧠
2. Dify Tool Call
LLM invokes flycrawl_scrape with URL
⚡
3. Sub-350ms FlyCrawl
Bypasses Cloudflare & returns Fit-Markdown
🎯
4. Accurate Synthesis
DeepSeek / Claude delivers factual answer

3 Steps to Set Up FlyCrawl in Dify.ai

Configure once, reuse across all your Dify apps, agents, and knowledge pipelines.

01

Create Custom Tool

In your Dify workspace, click on Tools in the left sidebar, click '+ Create Custom Tool', and choose OpenAPI / Swagger schema.

02

Paste OpenAPI Specification

Paste the YAML schema from this page. Dify will automatically recognize the flycrawl_scrape endpoint and its parameters.

03

Add Bearer Authorization

In Tool Credentials, set Auth Type to API Key, Header Name to 'Authorization', and enter 'Bearer YOUR_KEY'. Attach the tool to any agent!

📄 READY-TO-IMPORT SPECIFICATION

Dify.ai OpenAPI 3.0 Tool Schema (Full Specification) 

openapi: 3.0.1
info:
  title: FlyCrawl Scraping Engine
  description: High-performance web scraping and markdown conversion tool for Dify agents
  version: 1.0.0
servers:
  - url: https://api.flycrawl.net/api/v1
paths:
  /scrape:
    post:
      operationId: flycrawl_scrape
      summary: Scrape dynamic web content into clean Markdown
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              required:
                - url
              properties:
                url:
                  type: string
                  description: The target website URL to browse and scrape
                formats:
                  type: array
                  items:
                    type: string
                  default: ["markdown"]
                onlyMainContent:
                  type: boolean
                  default: true
                  description: Strip headers, navbars, ads, and footers
      responses:
        '200':
          description: Successful scrape
          content:
            application/json:
              schema:
                type: object
                properties:
                  success:
                    type: boolean
                  data:
                    type: object
                    properties:
                      markdown:
                        type: string
                      title:
                        type: string
                      url:
                        type: string

Frequently Asked Questions

Got questions about deploying FlyCrawl in Dify? We've got answers.

Can I use FlyCrawl with Dify Cloud and Dify Community Edition (Docker)? +
How does Fit-Markdown prevent Dify LLM context window overflows? +
What happens if a website uses heavy client-side JavaScript (React/Vue/Angular)? +

Power Up Your Dify AI Agents Today

Get 100 free scraping credits immediately. Connect in under 2 minutes with OpenAPI 3.0.

Create Free Account → Explore Documentation