Questions to ask any AI chat vendor before you sign

Evaluating this category is hard because the demos all look similar. Every product answers a question about your website convincingly in a controlled demo. The differences show up later, in the areas nobody demonstrates.

These are the questions worth asking. We have written them so they apply to us too, and where we do not yet have an answer, we say so.

Answer quality

• What happens when the assistant does not know? Ask them to demonstrate a refusal, not just a success.
• Can I see which source an answer came from?
• What happens if two of my pages contradict each other?
• How do I test it against my own real questions before committing?

That first one is the most revealing question in the list. A vendor who cannot readily show you a refusal either has not designed one or does not want to.

Content and control

• Can I exclude specific pages or sections?
• How does retraining work when my site changes, and is it automatic or manual?
• What happens to indexed content when I delete a page?
• Can I add answers by hand for things not on my site?

Data handling

• Which model providers do you use?
• Is provider training on my data disabled, and is that contractual or a setting?
• How long do you retain conversations, and how long do your providers?
• Where is data processed geographically?
• What is the deletion process and how long does it take end to end?
• Can I export everything?

Ask for these in writing. A confident verbal answer about retention is not the same as a documented one, and the difference matters if you are describing it on your own privacy page.

Escalation and operations

• Where does a handoff go, and what context travels with it?
• What happens outside working hours?
• What does the visitor see if your service is unavailable?

Commercial

• What is the unit, and what does it correspond to in real usage?
• What happens at the limit: stop, throttle, or overage billing?
• Can I move between plans in both directions?
• What is the notice period?

Compliance, and how to read the answer

If a vendor claims a certification, ask which auditor, which scope and when it was issued. A certification is a specific document about a specific scope, and "we are SOC 2 compliant" without those details is a marketing sentence rather than a fact you can rely on.

Equally, absence of a certification is not disqualifying for an early stage product. What matters is whether they are honest about where they are.

Our own answers, as of this writing

Applying the list to ourselves, in the interest of not asking others for a standard we do not meet:

• Creobot is in development, with a self serve launch targeted for the next month. There is no signup URL yet.
• Pricing shown on this site is a launch direction, not final production pricing.
• Model providers are not finalised. Creobot is designed for multi model routing so teams can route conversations by cost, quality and task type.
• Retention periods, subprocessors and data residency are not finalised and will be published before general availability.
• We hold no security certification and are not claiming one.

If a vendor cannot produce an equivalent list, that is itself information.