Common Crawl URL Index Lookup Scraper
by parseforge
Queries the Common Crawl URL index for a domain or URL and returns each crawl record as a flat row with timestamp, status code, MIME type, and crawl ID. It runs on Apify at $0.0110/result and has 15 users.
Actor pages combine Apify platform data with editorial review. We keep affiliate links separate from the ranking and update or remove pages when the actor changes. Read our methodology or send a correction to hello@tryapify.com.
TL;DR: Common Crawl URL Index Lookup Scraper by parseforge is built for data extraction workflows. It has 15 users and runs on $0.0110/result.
What It Does
Queries the Common Crawl URL index for a domain or URL and returns each crawl record as a flat row with timestamp, status code, MIME type, and crawl ID. Supports filters for date, status, and MIME type.
Key Stats
- Users
- 15
- Total Runs
- 175
- Pricing
- $0.0110/result
- Rating
- View on Apify →
Quick Answers
Teams that need a ready-made data extraction actor and want to compare pricing, reviews, and actor activity before running it.
Test a small run first, export the sample output, then decide whether the pricing model works for your volume.
parseforge/common-crawl-index-scraper
$0.0110/result
Related Docs
Apify tutorial: sign up and run your first scraper in 5 minutes
Step-by-step Apify tutorial for beginners. Create an account, open the Console, run your first actor, and export clean data. No code required.
Read guide →Scraping Dynamic Websites
How to scrape JavaScript-rendered websites using Playwright and Puppeteer. Learn browser automation, wait strategies, and handling single-page apps for dynamic scraping.
Read guide →Apify API: get your token and run actors from Python or JavaScript
How to use the Apify API: get your token, install the Python or JS client, start actor runs, and pull datasets into your app. Code examples included.
Read guide →Common Questions
What is Common Crawl URL Index Lookup Scraper?
Queries the Common Crawl URL index for a domain or URL and returns each crawl record as a flat row with timestamp, status code, MIME type, and crawl ID. Supports filters for date, status, and MIME type.
How much does Common Crawl URL Index Lookup Scraper cost?
Common Crawl URL Index Lookup Scraper costs $0.0110/result on Apify, and the free tier lets you test it before upgrading.
Is Common Crawl URL Index Lookup Scraper free?
Common Crawl URL Index Lookup Scraper is not fully free, but Apify provides a free tier so you can test it before paying for usage.
How to use Common Crawl URL Index Lookup Scraper?
Open Common Crawl URL Index Lookup Scraper on Apify, configure the input for the data you want to scrape, run the actor, and export the results when the run completes.
Free tier available. No credit card needed.
We earn commission from qualifying purchases. This does not affect our ratings.
Related Tools
Web Scraper
Crawls arbitrary websites using a web browser and extracts structured data from web pages using a provided JavaScript function. The Actor supports both recursive crawling and lists of URLs, and automatically manages concurrency for maximum performance.
Cheerio Scraper
Crawls websites using raw HTTP requests, parses the HTML with the Cheerio library, and extracts data from the pages using a Node.js code. Supports both recursive crawling and lists of URLs. This actor is a high-performance alternative to apify/web-scraper for websites that do not require JavaScript.
Playwright Scraper
Crawls websites with the headless Chromium, Chrome, or Firefox browser and Playwright library using a provided server-side Node.js code. Supports both recursive crawling and a list of URLs. Supports login to a website.
Puppeteer Scraper
Crawls websites with the headless Chrome and Puppeteer library using a provided server-side Node.js code. This crawler is an alternative to apify/web-scraper that gives you finer control over the process. Supports both recursive crawling and list of URLs. Supports login to website.