Scrape and crawl web pages through an API
Fetch public web pages through an API and get them back as Markdown, HTML, screenshots or structured JSON, with crawling, JavaScript rendering, proxies and retries handled for you -- not browsing the web yourself (browse-with-an-ai-browser) and not searching your own documents. 21 products across 18 companies in the Ludbee catalogue carry this task — across 2 of the 4 layers of the stack: 2 Application, 16 Infrastructure.
Filter the whole directory by this job to sort and compare these products.
Where these run
- SaaS
- 21 of 21 products that record it
- Self-hosted
- 1 of 21 products that record it
One product can run more than one way — the counts above are of records, not of products.
What this page does not claim
This list carries no score and no recommendation, and one order only: inside each layer, a company whose product for this task is marked AI feature — sold for another job, with AI as one function its own page names — comes after the rest, A–Z within each. The standing rule that applies to the whole directory applies here too: no payment changes a figure, a ranking or the order of any list, this one included. Nor can it yet say what governs any of these products once deployed — what data they can reach, what they are certified against, whether a vendor trains on what you send it. That field is defined in the ontology and is populated on none of the 5,815 products in the catalogue yet, so a page that leaned on it would be promising an answer the data cannot back. It will say so here the day that changes.
The companies
Application (2)
Infrastructure (16)
- Context.dev — Context.dev
- Firecrawl — Interact, Monitor (AI feature)
- HasData — HasData Google AI Mode API, HasData Web Scraping API
- Hyperbrowser — Hyperbrowser Web Scraping
- Import.io — Import.io
- Linkup — Linkup Fetch
- Parse — Parse
- Pomerado — Pomerado
- Scrape.do — ChatGPT Scraper API, Gemini Scraper API
- ScrapingAnt — AI Data Scraper
- SearchApi — Google AI Overview API
- Skyvern — Skyvern
- TinyFish — TinyFetch
- Exa — Contents API (AI feature)
- ScrapeOps — Web Scraping API Aggregator (AI feature)
- Spider — Spider Web Crawling & Scraping API (AI feature)