Amazonbot is finally respecting robots.txt | Hacker News

Hacker News users say Amazonbot has begun honoring robots.txt rules.

The change suggests Amazon’s crawler is behaving more like a compliant web bot.

Commenters also discussed blocking bot traffic with WAFs and Cloudflare-based controls.

The debate reflects wider concerns about managing automated scraping and AI crawler access.