Firecrawl

Firecrawl is a developer API that converts any website into clean, LLM-ready markdown. It handles JavaScript rendering, dynamic content, and complex site structures — returning structured data ideal for AI pipelines, RAG systems, and content extraction workflows.

Features

Core Capabilities

• Scrape: Fetch a single URL and return clean markdown or structured HTML
• Crawl: Recursively crawl an entire website and return all pages as markdown
• Map: Quickly generate a sitemap of all URLs on a domain
• Extract: Use AI to extract structured JSON from pages using a defined schema

Content Processing

• Full JavaScript rendering via headless browser
• Automatic removal of navigation, ads, headers, and footers
• Clean markdown output preserving headings, tables, lists, and code blocks
• Returns page metadata (title, description, og:image)

LLM Integration

• Output format designed for RAG pipelines and vector databases
• Works out-of-the-box with LangChain, LlamaIndex, and CrewAI
• Batch crawling for building knowledge bases from documentation sites

Quick Start

using System.Net.Http.Json;
using System.Text.Json.Nodes;

using var client = new HttpClient();
client.DefaultRequestHeaders.Add("Authorization", "Bearer fc-your-api-key");

// Scrape a single page
var scrapeResponse = await client.PostAsJsonAsync(
    "https://api.firecrawl.dev/v1/scrape",
    new { url = "https://docs.example.com/intro", formats = new[] { "markdown" } }
);
var scrapeResult = await scrapeResponse.Content.ReadFromJsonAsync<JsonNode>();
Console.WriteLine(scrapeResult!["data"]!["markdown"]);

// Crawl an entire site
var crawlResponse = await client.PostAsJsonAsync(
    "https://api.firecrawl.dev/v1/crawl",
    new { url = "https://docs.example.com", limit = 50 }
);
var crawlResult = await crawlResponse.Content.ReadFromJsonAsync<JsonNode>();
foreach (var page in crawlResult!["data"]!.AsArray())
    Console.WriteLine(page!["markdown"]);

Use Cases

• Building RAG knowledge bases from documentation sites
• Extracting product data for price monitoring
• Feeding website content into LLM-powered chatbots
• Competitive research and content analysis at scale
• Automated content ingestion for AI workflows

Pricing

• Free: 500 credits/month
• Hobby: $16/month — 3,000 credits
• Standard: $83/month — 100,000 credits
• Growth: $333/month — 500,000 credits