Web Crawler Resource Collection

TitleCategoryRemarks
chromedpchromeGo-written Chrome supports CDP, high concurrency efficiency
agent-browserai crawlerOpen-source by Vercel, automated crawler with skills plugins
watercrawlcrawler serviceCommercial crawler
sync-your-cookiecrawlerCookie synchronization across multiple devices
ProxyCatproxyAutomatic proxy switching, IP pool
magentic-uicrawlerIntegrates Microsoft's AI models for automated crawling and script execution
ScraplingcrawlerCan adapt to CSS changes and still capture content
MediaCrawlercrawlerE-commerce and video crawler solution based on Playwright, mainly for crawling comments
camoufoxanti-bot crawlerHeadless browser crawler with strong anti-bot capabilities, a customized Firefox
scrapydwebcrawlerCentralized management of crawlers
playwright_stealthpluginBypasses common web verifications
InfoSpiderpython crawlerPersonal data
scrawlercrawlerCrawls images from foreign social networks, only works on Windows
scrapydcrawlerCrawler to daemon process
solverecaptchascrawlerCAPTCHA Playwright eCAPTCHA v2
DrissionPagecrawlerDomestic crawler plugin combining requests and headless browser, uses browser fingerprint erasure technology, supports deep hiding in Linux environments
grabcrawlerSuper concise crawler framework
maxunframeworkSimulates human operations, Docker deployment, avoids being blocked