Knowledge sources
Everything your bot can answer from, and how each kind stays current.
The five kinds
All five go through the same pipeline and produce the same kind of knowledge — the difference is only where the text comes from and whether there is anything to re-read later.
| Kind | Best for | Refreshes |
|---|---|---|
| Website | Your main source. Give it a domain and it follows links and your sitemap to read everything it can reach. | Nightly |
| Sitemap | A site where you want exactly the URLs you publish, and nothing the crawler might wander into. | Nightly |
| Single page | One important URL — a pricing page, a policy — including pages the crawler could not find by following links. | Nightly |
| File | Content that is not on the web at all: a manual, a price list, an internal FAQ. | Never — re-upload to change it |
| Text | An answer you can write in a paragraph and do not want to publish anywhere. | Never — edit it to change it |
Reading a website
The reader starts at the URL you give it, follows links within the same site, and reads every page it can reach. It skips images, video, archives and other assets rather than fetching them to find out they are not text.
Sitemaps
If your site publishes a sitemap, it is used — including a sitemap index that points at other sitemaps, which is the normal shape for anything larger than a brochure site. Child sitemaps are walked together under one shared budget, duplicates across them are read once, and one unreachable child does not lose you the rest.
Linked PDFs
A PDF your site links to is read like a page. The reader looks at the actual bytes rather than trusting the content type, because servers routinely mislabel downloads, and hands it to the same extractor an upload would use — so a linked price list and an uploaded one produce identical knowledge.
Files and pasted text
Upload a file, or paste text directly. Both are read once, at the moment you add them.
PDF, Word (.docx), Excel (.xlsx), CSV, plain text and Markdown are supported, up to 25 MB each. Scanned PDFs — the ones that are really images of pages — are transcribed rather than skipped.
Heads up
Files and pasted text are never re-synced, because there is nothing to re-read: the file lives with us, not on a URL we can check. If the underlying document changes, upload the new version. This is also why they do not slow down or interfere with your nightly website crawl.
Staying up to date
Website, sitemap and single-page sources are re-read every night at 05:00 UTC. Pages that changed are re-learned; pages that did not are left alone.
You can also re-sync any source by hand from the sources page, which is the right move after you publish something you want answerable today rather than tomorrow.
Note
A manual re-sync is queued rather than run inside the request, and the page polls for the result. A full-site crawl cannot finish inside a single HTTP response, and pretending otherwise would just mean a timeout instead of an answer.
Removing one page
You do not have to delete a whole source to get rid of one bad page. Remove the individual page from the source and its content stops being used immediately.
Removing a page marks it excluded rather than deleting the record, and later crawls skip it. That is deliberate: a deleted record would simply be rediscovered by tonight's crawl and quietly re-learned, so the page you removed would come back without anyone touching anything.
When there is nothing to answer from
The bot will not invent an answer that no page supports. If a real visitor asks something your content does not cover, three things happen:
- The visitor is told the bot is not sure and offered a person, rather than being given a plausible guess.
- The question is recorded in your inbox as a gap.
- Answering it once — either by adding content, or by replying to the email we send you — makes it answerable for everyone who asks it afterwards.