Choosing what to train a website assistant on

When people set up a website assistant, the first instinct is to point it at the whole domain and upload every document they have. It feels thorough. It reliably produces worse answers than a carefully chosen subset.

The reason is mechanical. Retrieval works by finding the passages most similar to the question. Add more content and you add more candidates that are superficially similar but wrong. A blog post from two years ago describing a feature you removed will compete with your current documentation, and similarity does not know which one is true.

The three questions for every source

Is it current?

Outdated content is worse than absent content, because absent content produces "I do not know" and outdated content produces a confident wrong answer. Old pricing pages, deprecated feature docs and superseded policies should be excluded, not included and hoped over.

Is it authoritative?

For any given question there should be one place that answers it. If three pages describe your refund policy slightly differently, pick one and exclude the others. Contradictory sources produce inconsistent answers and there is no way to reason your way out of it at query time.

Is it prose?

Navigation menus, footers, cookie banners and legal boilerplate repeat on every page and add noise to every chunk. Content that is mostly structure rather than sentences is a poor retrieval source.

A starting set that works

For most marketing sites, this is the right initial scope:

• Product and feature pages, current versions only.
• Pricing page.
• Documentation or help centre, if it exists.
• Policy pages that visitors ask about: privacy, terms, refund, security.
• A hand written FAQ covering things not documented elsewhere.
• About page, for company questions.

Notably absent: the blog. Blog archives are the single most common source of bad answers, because they are large, old, exploratory and often contain positions the company no longer holds. Add specific evergreen posts by hand if they answer real questions. Do not add the archive.

The hand written FAQ is the highest value source

Every product has facts that live in someone's head and appear on no page. Whether you support a particular integration, what happens at a plan limit, whether a feature works on mobile. These generate questions constantly and there is nothing to retrieve.

Writing thirty of these as explicit question and answer pairs takes an afternoon and usually improves answer quality more than any other single change. It also has a useful side effect: the act of writing them tells you which ones should have been on your website all along.

Test with the questions you already have

If you have any record of what visitors ask, use those as your test set rather than inventing questions. Inventing test questions produces questions phrased the way you think, which is not how visitors think.

Twenty real questions, run against the assistant, read by hand. Count how many are answered correctly, how many are answered wrongly, and how many correctly say they do not know. The middle number is the one that matters, because a wrong confident answer costs more trust than a refusal.

Retraining cadence

Content changes and the index needs to follow. The rule that avoids both extremes: retrain when a source page materially changes, not on a fixed schedule. A weekly retrain on a site that changes monthly is wasted work. A quarterly retrain on a site that ships weekly means three months of wrong answers about pricing.

The pages worth watching most closely are pricing, plan limits and anything with a date in it.