ScrapeforLLM Docs
ScrapeforLLM Docs
Getting StartedScrape a PageScreenshotCrawl a SiteMap, Search & ExtractList & Get ScrapesError Codes

Map, Search & Extract

Discover URLs, search the web, and extract structured data with AI.

Map, Search & Extract

Three powerful operations beyond basic scraping. All return results immediately (no polling needed).

Map a Site

Discover all URLs on a website without scraping their content. Useful for finding pages before crawling.

curl -X POST https://scrapeforllm.com/api/app/scrapes \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "url": "https://docs.example.com",
    "type": "map"
  }'
const response = await fetch("https://scrapeforllm.com/api/app/scrapes", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    Authorization: "Bearer YOUR_API_KEY",
  },
  body: JSON.stringify({
    url: "https://docs.example.com",
    type: "map",
  }),
});

const data = await response.json();
console.log(data.scrape.result.links);
import requests

response = requests.post(
    "https://scrapeforllm.com/api/app/scrapes",
    headers={
        "Content-Type": "application/json",
        "Authorization": "Bearer YOUR_API_KEY",
    },
    json={
        "url": "https://docs.example.com",
        "type": "map",
    },
)

data = response.json()
print(data["scrape"]["result"]["links"])

Response:

{
  "scrape": {
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "status": "completed",
    "creditsUsed": 1,
    "result": {
      "links": [
        "https://docs.example.com",
        "https://docs.example.com/getting-started",
        "https://docs.example.com/api-reference",
        "https://docs.example.com/guides/auth"
      ]
    }
  }
}

Cost: 1 credit.

Search the Web

Search the web for a query and get the results as structured data with page content.

Note

For search, the url field is used as the search query (not a URL).

curl -X POST https://scrapeforllm.com/api/app/scrapes \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "url": "latest Next.js 15 features",
    "type": "search"
  }'
const response = await fetch("https://scrapeforllm.com/api/app/scrapes", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    Authorization: "Bearer YOUR_API_KEY",
  },
  body: JSON.stringify({
    url: "latest Next.js 15 features",
    type: "search",
  }),
});

const data = await response.json();
for (const result of data.scrape.result.data) {
console.log(result.title, result.url);
}
import requests

response = requests.post(
    "https://scrapeforllm.com/api/app/scrapes",
    headers={
        "Content-Type": "application/json",
        "Authorization": "Bearer YOUR_API_KEY",
    },
    json={
        "url": "latest Next.js 15 features",
        "type": "search",
    },
)

data = response.json()
for result in data["scrape"]["result"]["data"]:
    print(result["title"], result["url"])

Response:

{
  "scrape": {
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "status": "completed",
    "creditsUsed": 2,
    "result": {
      "data": [
        {
          "markdown": "# Next.js 15 Release Notes\n\nContent from the page...",
          "url": "https://nextjs.org/blog/next-15",
          "title": "Next.js 15"
        }
      ]
    }
  }
}

Cost: 2 credits.

Extract Structured Data

Use AI to extract specific information from a page. Returns structured JSON based on your prompt.

curl -X POST https://scrapeforllm.com/api/app/scrapes \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "url": "https://example.com/product",
    "type": "extract",
    "options": {
      "prompt": "Extract the product name, price, rating, and availability"
    }
  }'
const response = await fetch("https://scrapeforllm.com/api/app/scrapes", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    Authorization: "Bearer YOUR_API_KEY",
  },
  body: JSON.stringify({
    url: "https://example.com/product",
    type: "extract",
    options: {
      prompt: "Extract the product name, price, rating, and availability",
    },
  }),
});

const data = await response.json();
console.log(data.scrape.result.data);
import requests

response = requests.post(
    "https://scrapeforllm.com/api/app/scrapes",
    headers={
        "Content-Type": "application/json",
        "Authorization": "Bearer YOUR_API_KEY",
    },
    json={
        "url": "https://example.com/product",
        "type": "extract",
        "options": {
            "prompt": "Extract the product name, price, rating, and availability"
        },
    },
)

data = response.json()
print(data["scrape"]["result"]["data"])

Response:

{
  "scrape": {
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "status": "completed",
    "creditsUsed": 5,
    "result": {
      "data": {
        "product_name": "Wireless Headphones Pro",
        "price": "$129.99",
        "rating": "4.5/5",
        "availability": "In Stock"
      }
    }
  }
}

If you don't provide a prompt, a default one is used that extracts the main content and key information.

Cost: 5 credits (uses LLM extraction).

Quick Comparison

OperationInputReturnsCreditsUse Case
MapURLList of URLs1Find all pages before crawling
SearchQuery stringWeb results with content2Research a topic
ExtractURL + promptStructured JSON5Pull specific data points

Crawl a Site

Crawl an entire website and get every page as clean markdown.

List & Get Scrapes

Retrieve your scrape history and check on individual scrape results.

On this page

Map, Search & ExtractMap a SiteSearch the WebExtract Structured DataQuick Comparison