Website Content Crawler
by apify
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. Pricing: Pay for platform usage. Apify records 167.8k users with a 4.6/5 rating.
Actor pages summarize saved Apify platform data. Listings are sorted by recorded user counts; inclusion is not a test result. Affiliate links may earn us a commission. Read our methodology or send a correction to hello@tryapify.com.
TL;DR: Website Content Crawler by apify is built for ai & llm workflows. It has 167.8k recorded actor users. Pricing: Pay for platform usage. 4.6/5 stars.
What It Does
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
Key Stats
- Users
- 167,833
- Total Runs
- 45,495,744
- Pricing
- Pay for platform usage
- Rating
- View on Apify →
Quick Answers
Teams that need a ready-made ai & llm actor and want to compare pricing, reviews, and actor activity before running it.
Test a small run first, export the sample output, then decide whether the pricing model works for your volume.
apify/website-content-crawler
Pay for platform usage
Related Docs
What is Apify? Plain-English review of actors, pricing, and pros
Apify is a cloud platform with pre-built web scrapers (actors). Free tier included. Here is what it does, how pricing works, and who it suits.
Read guide →Apify API: get your token and run actors from Python or JavaScript
How to use the Apify API: get your token, install the Python or JS client, start actor runs, and pull datasets into your app. Code examples included.
Read guide →Scraping Dynamic Websites
How to scrape JavaScript-rendered websites using Playwright and Puppeteer. Learn browser automation, wait strategies, and handling single-page apps for dynamic scraping.
Read guide →Common Questions
What is Website Content Crawler?
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
How much does Website Content Crawler cost?
Pricing: Pay for platform usage. Saved event rates may exclude other events and platform charges. See the actor pricing on Apify before running it.
Is Website Content Crawler free?
Apify platform usage or actor charges may apply. Pricing: Pay for platform usage. Check the current actor terms and your plan before running it.
How to use Website Content Crawler?
Open Website Content Crawler on Apify, configure the input for the data you want to scrape, run the actor, and export the results when the run completes.
We earn commission from qualifying purchases. This does not affect our ratings.
Related Tools
RAG Web Browser
Web search and fetch tool for AI agents and RAG pipelines. It queries Google Search, scrapes the top N pages using a full web browser, and returns their content as clean Markdown for further processing by an LLM. Can also fetch individual URLs.
Manga Colorizer
AI-powered manga page colorizer. Takes black-and-white manga image URLs and returns fully colored pages with the original line art preserved. Supports batch processing and multiple original languages for accurate text region handling.
Website Content Crawler - Markdown, 45 Fields
Crawls websites from start URLs with depth, domain and URL-pattern limits. 45 fields per page: Markdown and plain text, word count, title and meta tags, headings, Open Graph and JSON-LD, author and dates, emails, phones and social links, detected technologies, links and HTTP status. Ready for LLMs.
Audio Transcriber - Speech to Text
Transcribes audio files from URL or upload into text with timestamps, language, and confidence. Supports WAV, MP3, AIFF, AAC, OGG, FLAC, and more.