This repo contains a prototype that analyzes crawl output, looking to quantify how much blocking of CCBot is due to bot defenses, vs. disallows in robots.txt files.
Please see OPEN-ATHENA-PILOT.md for an example analysis.
commoncrawl/ccbot-blocking-analysis
Folders and files
| Name | Name | Last commit date | ||
|---|---|---|---|---|