Keeping an assistant in sync with a changing site

When you train an assistant on your website, you create a snapshot of that content at a point in time. The website keeps changing. The snapshot does not, until you retrain. Everything about staleness follows from that one fact.

The cost of staleness is not uniform

This is the part most retraining schedules ignore. A stale answer about your company story costs almost nothing. A stale answer about your pricing costs a customer and possibly a complaint.

So a single global retrain interval is the wrong shape. What you want is a tiering of sources by how expensive it is to be wrong:

• High cost: pricing, plan limits, availability, legal and policy pages. Retrain on change, same day.
• Medium cost: feature descriptions, integrations, documentation. Retrain weekly or on significant change.
• Low cost: about page, blog posts, general marketing copy. Retrain monthly or when convenient.

Change detection beats scheduling

A fixed schedule is either too frequent, wasting compute on unchanged content, or too infrequent, leaving known-wrong answers live. Detecting change is more efficient than guessing at an interval.

The cheap version, which works for most sites, is comparing a hash of the extracted text per URL. If the hash changed, that URL needs reindexing. If it did not, skip it. Content hashing rather than HTTP headers, because last-modified headers on many CMS platforms are unreliable and reflect template changes rather than content changes.

for url in sources:
    text = extract_text(fetch(url))
    h = sha256(normalise(text))
    if h != stored_hash[url]:
        reindex(url, text)
        stored_hash[url] = h

Normalising before hashing matters. Strip navigation, footer, and anything that changes on every page load such as a year in a copyright line, or every page will look changed on every check.

The deletion problem

Adding new content is the easy direction. Removing it is where systems commonly fail. If you delete a page describing a discontinued feature and only run an additive reindex, the old chunks stay in the index and the assistant keeps describing a feature that no longer exists.

Any sync process needs an explicit removal step: fetch the current source list, compare against what is indexed, and delete chunks whose source URL is no longer present or no longer returns a success status. Treat a 404 as a deletion signal, and treat a redirect as a move rather than a deletion.

What to do during a reindex

Reindexing is not instantaneous. During it, the index is in a mixed state. For a marketing site the honest answer is that this rarely matters, because a few minutes of mixed content is not a business risk.

For a pricing change it does matter. If you are changing prices, the sequence should be: publish the new page, retrain that specific source, verify with a test question, then announce. Not the reverse.

A verification habit

Keep a short list of test questions whose correct answers you know, covering your highest cost sources. After any retrain, run them. Five questions, thirty seconds, and it catches the case where a reindex silently failed and left you serving last quarter's prices.