Common Crawl URL Index Lookup Scraper
by parseforge
Queries the Common Crawl URL index for a domain or URL and returns each crawl record as a flat row with timestamp, status code, MIME type, and crawl ID. Pricing: $0.011/event. Apify records 18 users.
Actor pages summarize saved Apify platform data. Listings are sorted by recorded user counts; inclusion is not a test result. Affiliate links may earn us a commission. Read our methodology or send a correction to hello@tryapify.com.
TL;DR: Common Crawl URL Index Lookup Scraper by parseforge is built for data extraction workflows. It has 18 recorded actor users. Pricing: $0.011/event.
What It Does
Queries the Common Crawl URL index for a domain or URL and returns each crawl record as a flat row with timestamp, status code, MIME type, and crawl ID. Supports filters for date, status, and MIME type.
Key Stats
- Users
- 18
- Total Runs
- 229
- Pricing
- $0.011/event
- Rating
- View on Apify →
Quick Answers
Teams that need a ready-made data extraction actor and want to compare pricing, reviews, and actor activity before running it.
Test a small run first, export the sample output, then decide whether the pricing model works for your volume.
parseforge/common-crawl-index-scraper
$0.011/event
Related Docs
Apify tutorial: sign up and run your first scraper in 5 minutes
Step-by-step Apify tutorial for beginners. Create an account, open the Console, run your first actor, and export clean data. No code required.
Read guide →Scraping Dynamic Websites
How to scrape JavaScript-rendered websites using Playwright and Puppeteer. Learn browser automation, wait strategies, and handling single-page apps for dynamic scraping.
Read guide →Apify API: get your token and run actors from Python or JavaScript
How to use the Apify API: get your token, install the Python or JS client, start actor runs, and pull datasets into your app. Code examples included.
Read guide →Common Questions
What is Common Crawl URL Index Lookup Scraper?
Queries the Common Crawl URL index for a domain or URL and returns each crawl record as a flat row with timestamp, status code, MIME type, and crawl ID. Supports filters for date, status, and MIME type.
How much does Common Crawl URL Index Lookup Scraper cost?
Pricing: $0.011/event. Saved event rates may exclude other events and platform charges. See the actor pricing on Apify before running it.
Is Common Crawl URL Index Lookup Scraper free?
Apify platform usage or actor charges may apply. Pricing: $0.011/event. Check the current actor terms and your plan before running it.
How to use Common Crawl URL Index Lookup Scraper?
Open Common Crawl URL Index Lookup Scraper on Apify, configure the input for the data you want to scrape, run the actor, and export the results when the run completes.
Saved event rates may exclude other events and platform charges. See current pricing on Apify.
We earn commission from qualifying purchases. This does not affect our ratings.
Related Tools
Web Scraper
Crawls arbitrary websites using a web browser and extracts structured data from web pages using a provided JavaScript function. The Actor supports both recursive crawling and lists of URLs, and automatically manages concurrency for maximum performance.
Cheerio Scraper
Crawls websites using raw HTTP requests, parses the HTML with the Cheerio library, and extracts data from the pages using a Node.js code. Supports both recursive crawling and lists of URLs. This actor is a high-performance alternative to apify/web-scraper for websites that do not require JavaScript.
Playwright Scraper
Crawls websites with the headless Chromium, Chrome, or Firefox browser and Playwright library using a provided server-side Node.js code. Supports both recursive crawling and a list of URLs. Supports login to a website.
Puppeteer Scraper
Crawls websites with the headless Chrome and Puppeteer library using a provided server-side Node.js code. This crawler is an alternative to apify/web-scraper that gives you finer control over the process. Supports both recursive crawling and list of URLs. Supports login to website.