FREE TIER — 200 CREDITS

Web Scraping API

Extract clean text, markdown, or structured data from any URL. JavaScript rendering included. No proxies, no browser setup, no headless Chrome. Just send a URL.

Try It Live API Docs

Live Scraping Demo

// Click "Scrape" to extract content

Features

JavaScript Rendering

Headless browser renders the full page before extraction. SPAs, dynamic content, and lazy-loaded elements are captured.

Multiple Output Formats

Get results as clean text, Markdown (preserving headers, lists, links), raw HTML, or structured JSON with metadata.

Metadata Extraction

Automatically extracts page title, meta description, Open Graph tags, canonical URL, and all links and images.

No Proxy Management

No rotating proxies, no IP bans to manage, no CAPTCHA solving. We handle all the infrastructure.

LLM-Ready Output

Markdown format is perfect for feeding into LLMs, RAG pipelines, and AI agents. Clean content without HTML noise.

Fast Response

Most pages scraped in under 3 seconds. Results are cached for repeated lookups of the same URL.

API Endpoints

MethodEndpointDescription
GET/api/scrape?url={url}Scrape URL (default: markdown)
GET/api/scrape?url={url}&format=textPlain text extraction
GET/api/scrape?url={url}&format=htmlRaw HTML source
GET/api/scrape?url={url}&format=jsonStructured JSON with metadata

Code Examples

# Scrape a URL as markdown
curl "https://agent-gateway-kappa.vercel.app/api/scrape?url=https://example.com" \
  -H "Authorization: Bearer YOUR_API_KEY"

# Get structured JSON with metadata
curl "https://agent-gateway-kappa.vercel.app/api/scrape?url=https://example.com&format=json" \
  -H "Authorization: Bearer YOUR_API_KEY"
const API_KEY = 'YOUR_API_KEY';
const BASE = 'https://agent-gateway-kappa.vercel.app';

// Scrape URL and get markdown
const target = encodeURIComponent('https://example.com');
const res = await fetch(
  `${BASE}/api/scrape?url=${target}`,
  { headers: { 'Authorization': `Bearer ${API_KEY}` } }
);
const data = await res.json();

console.log(data.title);    // Page title
console.log(data.content);  // Markdown content
import requests

API_KEY = "YOUR_API_KEY"
BASE = "https://agent-gateway-kappa.vercel.app"

# Scrape URL and get markdown
r = requests.get(
    f"{BASE}/api/scrape",
    params={"url": "https://example.com"},
    headers={"Authorization": f"Bearer {API_KEY}"}
)
data = r.json()
print(data["content"])  # Clean markdown

Use Cases

AI & RAG Pipelines

Feed web content into LLMs for question answering, summarization, and retrieval-augmented generation (RAG).

Market Research

Extract competitor pricing, product details, reviews, and news articles for analysis and reporting.

Content Aggregation

Build news feeds, content curators, and monitoring dashboards by scraping multiple sources programmatically.

SEO Analysis

Extract meta tags, headers, links, and structured data from competitor pages for SEO auditing and benchmarking.

Frequently Asked Questions

Is the web scraping API free?
Yes. 200 free credits on signup. Each scrape request costs 2 credits. No credit card or email required to get started.
Does it handle JavaScript-rendered pages?
Yes. The API uses a headless browser to render JavaScript before extracting content. SPAs, dynamic content, and lazy-loaded elements are all captured.
What output formats are available?
Text (clean plain text), Markdown (preserves formatting and structure), HTML (raw source), and JSON (structured metadata including title, description, links, and images).
Do I need to manage proxies?
No. The API handles all request routing. You send a URL, we return the content. No proxy setup, rotation, or IP management needed.
Is this good for feeding content to LLMs?
Yes. The Markdown output format is specifically designed for LLM consumption. Clean, structured text without HTML noise, perfect for RAG pipelines and AI agents.

Get Your Free API Key

200 free credits. No proxies. No browser setup. Just clean content from any URL.

Get Started Free