RabbitblazeBot
The crawler that reads your docs, politely
RabbitblazeBot fetches public pages that Rabbitblaze customers add as knowledge sources, so their AI agent can answer questions from them with links back to your site.
How to recognize it
- User agent
-
RabbitblazeBot/1.0 (+https://rabbitblaze.com/bot) - robots.txt token
-
RabbitblazeBot - Fetches
- Public HTML pages, robots.txt and sitemaps
-
Only sites our customers add
It visits a website only when a Rabbitblaze customer adds it as a knowledge source for their AI agent, usually their own docs or help site.
-
Follows robots.txt
It reads robots.txt before anything else and obeys the rules for RabbitblazeBot or *. If robots.txt answers with a server error, it doesn’t crawl at all.
-
One request at a time
Requests to a site are sequential, at least a second apart. A Crawl-delay in robots.txt slows it down further, up to 10 seconds between pages.
-
Sitemaps first, then links
It starts from the sitemaps listed in robots.txt (or /sitemap.xml), then follows links on the same host, under the starting path and at most three levels deep.
-
Public HTML only
It fetches public HTML pages, skips nofollow links, honors noindex and canonical URLs, and never logs in, submits forms or runs JavaScript.
-
Small, scheduled syncs
Each source is capped at 1,000 pages, and 100 by default. When auto-sync is on, it re-reads the site once a week.
Block or limit it
Your site, your rules
Add a group for RabbitblazeBot to your robots.txt. It applies from the next sync, and if the starting page is off limits, the customer who added your site is told that robots.txt doesn’t allow it.
Questions, or a crawler misbehaving in our name? Email [email protected].
robots.txt
# Keep RabbitblazeBot out of the whole siteUser-agent: RabbitblazeBotDisallow: /robots.txt
# Allow your docs, keep everything else private, and slow it downUser-agent: RabbitblazeBotAllow: /docs/Disallow: /Crawl-delay: 5FAQ
Questions, answered
Something else about RabbitblazeBot? Write to us and a person will answer.
Contact usWhy is RabbitblazeBot visiting my site?
Someone using Rabbitblaze added a page on your site as a knowledge source, so their AI agent can answer questions from it. Usually that’s your own team connecting your docs or help center.
Does it use my content to train AI models?
No. Pages are indexed so one workspace’s AI agent can quote and link them in answers. Rabbitblaze never trains AI models on the content it reads.
How do I verify a request really came from RabbitblazeBot?
Every request carries the user agent RabbitblazeBot/1.0 (+https://rabbitblaze.com/bot). If something claiming to be RabbitblazeBot ignores your robots.txt, write to us with the details and we’ll look into it.
How quickly does a robots.txt change take effect?
RabbitblazeBot reads robots.txt at the start of every sync, so a change applies from the next one. Pages it may no longer read drop out of that source, and if the starting page itself is blocked, the sync stops and the customer is told why.