HelpBrainBot: how we read your website

Every request carries this user-agent string:

Mozilla/5.0 (compatible; HelpBrainBot/1.0; +https://helpbrain.net/docs/crawler)

The +https://… part is the convention search engines use: it links back to this page so you can check what the bot is before deciding anything about it.

Only the website a HelpBrain customer entered when training their bot — nobody can point it at a site they do not own without that site appearing in their own bot's knowledge, visibly, under their name. It reads your sitemap first (from robots.txt, then /sitemap.xml), falls back to following links from the homepage, and fetches at most one page per second per site after a short initial burst. After the first read it re-checks the site once a night and re-learns only the pages whose content changed.

PDFs count as pages. When a page links a PDF — a price list, a job spec, a brochure, a policy — the bot fetches it and reads it exactly as it would an uploaded one: its text layer where there is one, and a transcription where the file is a scan. This works whether the PDF is listed in your sitemap or only linked from a page, and it follows documents onto your own sibling subdomains (a file on docs.example.com linked from www.example.com), which is where most sites keep them. Those hosts' own robots.txt rules are checked before anything there is fetched. Files over 25 MB are skipped, and a run transcribes only a handful of scanned PDFs before leaving the rest for the next night.

Pages that disappear are forgotten. When a URL the bot has learned starts answering 404, or drops off the site altogether, that page is removed from the bot's knowledge on the next nightly check — so a job spec for a filled role stops being quoted back to your visitors. If a whole site becomes unreachable, or a sitemap breaks and returns far fewer pages than usual, nothing is removed and the source says so instead.

Disallow rules in the User-agent: * group are honoured: disallowed paths are never fetched, not even to discover links. To keep HelpBrainBot out of a site entirely, add:

User-agent: HelpBrainBot
Disallow: /

If training fails with a message like “blocked our reader”, your firewall or bot protection is refusing automated visitors — most often Cloudflare's “Block AI bots” or Bot Fight Mode, which is on by default for many sites. You have three options:

1. Allow the bot. In Cloudflare: Security → WAF → Custom rules → create a rule with the expression (http.user_agent contains "HelpBrainBot") and the action Skip (skip all remaining custom rules, Bot Fight Mode and managed challenges). Other firewalls have an equivalent user-agent allowlist.
2. Add specific pages instead. Some protections only guard the homepage or the sitemap; “Specific pages” in the Knowledge screen reads just the URLs you list.
3. Skip the website. Upload PDFs, Word or Excel files, or paste the text directly. The bot answers from those exactly as it would from crawled pages.

Seeing HelpBrainBot on a site you run and do not want it there? Email [email protected] with the domain and we will stop reading it.