How to evaluate a docs chatbot before you ship it
Use this manual pre-launch acceptance test to decide whether a docs chatbot gives grounded, reviewable answers for tasks your team is responsible for.
A polished demo can still miss the question a real reader asks when they need to act. Before launch, decide whether your docs chatbot is ready to handle the tasks your team is responsible for, especially where a wrong answer could cause harm or leave a reader stuck. Use a pass, needs-review, or fail record for each task so the team can make that decision from evidence.
| Test question | Reader task or risk | Source to check | Result | Failure label | Owner and next check |
|---|---|---|---|---|---|
| Example only: rotate an API key | A developer needs a safe credential-replacement procedure | Current API-key rotation page | Needs review | Missing prerequisite | Docs owner confirms required permissions, then reruns the question |
Copy this record for each task you test. Keep fictional examples separate from your product's actual procedures.
Launch only when a reviewer can trace every answer for a high-risk task to current documentation and explain every exception.
Choose questions that could block launch
Test the questions for which your team would need to explain a bad answer. Treat access control, data deletion, security settings, billing changes, and destructive actions as high-risk tasks when they apply to your product. Include a common task, an alternate wording, a recent change, a combined task, an unsupported request, and a request where the documentation lacks a necessary detail.
| Question type | What the reviewer checks | Example |
|---|---|---|
| Common task | The core procedure is available | "How do I rotate an API key?" |
| Alternate wording | The answer recognizes reader language | "My credential leaked. What is the replacement process?" |
| Recent change | The answer follows the current procedure | "Which key-rotation flow applies after the latest release?" |
| Combined task | A necessary condition survives a multi-part question | "How do I rotate a key without breaking the deployment that uses it?" |
| Unsupported request | The answer does not invent a capability | "Can I restore a deleted key?" |
| Missing-information request | The answer names the documentation boundary | "What is the recovery process for an undocumented account state?" |
Write questions in the language a reader would use. Do not write a polished expected answer and reward the chatbot for repeating it.
Review the answer and its source together
Check the answer and the source that supports it. A source link lets a reviewer inspect the evidence, but it does not prove that the answer is correct.
| Criterion | Pass evidence | Hold-the-launch result |
|---|---|---|
| Groundedness | The material claim matches the canonical source | The answer contradicts the source or adds an unsupported product detail |
| Source usefulness | The page lets the reader verify or continue the task | The source is missing, unrelated, or too broad to check the claim |
| Completeness | The answer includes the condition needed to act safely | It omits a prerequisite that can cause a failed or risky action |
| Freshness | The answer reflects the current documented procedure | It returns a retired workflow or superseded detail |
| Boundary handling | The answer identifies what the documentation does not establish | It fills a known gap with a confident procedure |
| Reviewability | A reviewer can record why the result passed or failed | The outcome is only an unexplained aggregate score |
Record pass when the answer and source meet the check. Record needs review when an important detail cannot yet be confirmed. Record fail when the answer is unsupported, contradictory, or unsafe.
Label the failure so the owner can repair it
Choose the first repair from the failure you found. A thin documentation page needs a documentation repair, while an available page that the assistant does not use needs a source-coverage investigation.
| Failure label | First repair to investigate | Recheck |
|---|---|---|
| Missing or thin source content | Improve the canonical documentation page | Ask the same question after the source is available |
| Stale source content | Update the canonical page | Refresh the relevant source, then rerun the question |
| Relevant source not retrieved | Inspect source coverage, structure, wording, and indexing | Confirm the intended source is available and repeat the test |
| Answer exceeds source evidence | Review assistant guidance and supporting documentation | Retest the boundary case and the normal task |
| Unclear support handoff | Define the human route and its ownership | Test that a reader can understand the next step |
Keep the transcript, source link, result, failure label, owner, and recheck in the same record. “The answer omitted the required permission” gives the next reviewer a usable reason to verify the repair.
Decide the launch rule before testing
Hold launch when a high-risk task produces an unsupported, contradictory, or unsafe answer. Set that rule before testing so a convincing demo cannot redefine what is acceptable.
Every failure needs an owner and a recheck. A wording weakness can remain open when the answer is grounded, the source is usable, and an owner is assigned. A confident guess about an action that can damage a deployment or lock a user out should block launch.
Check the sources available to a Biel.ai project
Before treating a missing answer as an assistant problem, confirm that the canonical documentation is connected to the project. Biel.ai documents supported source types and the steps for managing them in its source documentation. Use that page to verify the project setup, save the intended source, and rerun the affected question after indexing completes.
This test covers pre-launch review only. After launch, use a separate method to measure chatbot performance and identify recurring documentation gaps, such as how to tell if your documentation chatbot is actually working. For background on the retrieval layer that supplies source material, see how RAG makes technical documentation useful. For a post-launch workflow that turns recurring questions into documentation work, see chatbot analytics to improve documentation quality.
Frequently asked questions
How many questions should the first acceptance test include?
Start with one common task and one boundary case for each high-risk task your chatbot must handle. Add alternate wording or a recent-change question where that task has changed or readers describe it differently. Keep the set small enough for a reviewer to inspect every answer and source.
Who should review a high-risk test result?
Use a reviewer who can verify the canonical source and name the owner of the repair. The answer should remain needs review until that person can confirm the missing condition, source, or support route.
If you want to run this acceptance test against your own documentation sources, start a Biel.ai project.