about our crawler

BataooBot

Bataoo operates a polite, identifying web crawler that helps us index Indian web content for search. Here's exactly how it behaves and how to control it.

User-Agent

BataooBot/0.1 (+https://bataoo.com/bot)

What it does

  • · Fetches /robots.txt first on every host and caches it. Honored strictly.
  • · Reads sitemap.xml to discover URLs.
  • · Default 1 request per second per host; respects Crawl-Delay in robots.txt if higher.
  • · Truncates fetches at 64 KB per page — we read the structure, not the whole asset.
  • · Stores a snippet of at most 280 characters per page; never republishes full content.
  • · Does not bypass paywalls, login walls, or geographic restrictions.

Block us

To exclude BataooBot from your site, add to robots.txt: 

User-agent: BataooBot
Disallow: /

Throttle us

To slow our crawl rate on your host: 

User-agent: BataooBot
Crawl-Delay: 10

Contact

Crawler problems, abuse reports, or takedown requests: contact@bataoo.com. See also /takedown.