How to build an evaluation harness for a documentation chatbot

A documentation change can improve one chatbot answer and accidentally make another worse.

You can catch those regressions with a small evaluation harness: a fixed set of real documentation questions that you run again after important source, product, or configuration changes.

The same approach works whether you call the product a documentation chatbot or an AI documentation assistant. The goal is the same: test the questions readers actually ask and check that the answers still match the current documentation.

You do not need a large benchmark. Start with a handful of questions that represent tasks your readers actually need to complete, send them through the Biel API, save the responses, and compare them with the current documentation.

Start with a small set of real questions

Choose questions that could expose a meaningful documentation problem.

For example:

• a common setup task
• a question about something that recently changed
• an alternate way readers ask an important question
• a task with an important prerequisite
• a question the documentation does not support

Save those questions in a file such as eval-cases.json.

[
  {
    "id": "rotate-key",
    "question": "How do I rotate an API key without breaking the deployment that uses it?",
    "expected_evidence": [
      "Current API-key rotation guide",
      "Required permissions section"
    ]
  },
  {
    "id": "unsupported-recovery",
    "question": "Can I restore a deleted key?",
    "expected_evidence": [
      "Current API-key lifecycle documentation"
    ]
  }
]

These are fictional examples. Replace them with questions and source pages from your own documentation.

Keep the set small enough that someone can review every result. Ten useful questions are better than hundreds of generic prompts nobody inspects.

Send each question through the Biel API

Biel API v2 lets you send a message to a project with:

POST /api/v2/projects/{slug}/chats/

Create an API key with the required chat permission and keep it server-side. See the Biel API reference and send-message reference for the current authentication and request details.

Save the fixture as eval-cases.json, then create a small runner:

# run_biel_eval.py
 
import json
import os
import urllib.error
import urllib.request
from datetime import datetime, timezone
from pathlib import Path
 
base_url = "https://app.biel.ai"
project_slug = os.environ["BIEL_PROJECT_SLUG"]
api_key = os.environ["BIEL_API_KEY"]
 
cases = json.loads(Path("eval-cases.json").read_text())
output_path = Path("evaluation-results.jsonl")
 
 
def new_record(case):
    return {
        "run_at": datetime.now(timezone.utc).isoformat(),
        "id": case["id"],
        "question": case["question"],
        "expected_evidence": case["expected_evidence"],
    }
 
 
with output_path.open("w") as output:
    for case in cases:
        record = new_record(case)
 
        request = urllib.request.Request(
            f"{base_url}/api/v2/projects/{project_slug}/chats/",
            data=json.dumps({
                "message": case["question"]
            }).encode(),
            headers={
                "Authorization": f"Api-Key {api_key}",
                "Content-Type": "application/json",
            },
            method="POST",
        )
 
        try:
            with urllib.request.urlopen(request) as response:
                record["http_status"] = response.getcode()
                body = response.read().decode("utf-8", errors="replace")
 
                try:
                    record["response"] = json.loads(body)
                    record["run_status"] = "ok"
                except json.JSONDecodeError:
                    record["run_status"] = "fail"
                    record["error"] = "Response was not valid JSON"
 
        except urllib.error.HTTPError as error:
            record["run_status"] = "fail"
            record["http_status"] = error.code
            record["error"] = "HTTP request failed"
 
        except urllib.error.URLError as error:
            record["run_status"] = "fail"
            record["http_status"] = None
            record["error"] = f"Network error: {error.reason}"
 
        output.write(json.dumps(record) + "\n")
 
        print(f"{case['id']}: {record['run_status']}")

Set the project slug and API key as environment variables before running the script.

export BIEL_PROJECT_SLUG="your-project"
export BIEL_API_KEY="your-api-key"
 
python run_biel_eval.py

The runner creates a new chat for each question so one answer does not affect the next one.

It saves each result to evaluation-results.jsonl, giving you a record you can compare across runs.

Do not commit the API key to the repository.

Review the answer against the documentation

A successful API request only tells you that the request worked. It does not tell you whether the chatbot or documentation assistant gave a good answer.

For each question, check:

1. Does it answer the task?
2. Are the important claims supported by the current documentation?
3. Does it include the conditions or prerequisites the reader needs?
4. Does it avoid inventing behavior the documentation does not support?
5. Would the returned sources help a reader verify the answer?

For the API-key example, an answer might sound plausible while missing a required permission or an important step in the rotation process.

That should count as a regression even if the response is fluent.

Keep a simple regression log

You do not need a complicated scoring system.

Record whether each important case passed, failed, or needs review.

Keep the saved answer with the result so another reviewer can see why the decision was made.

If an important task fails, fix the documentation or source configuration and run that question again.

Run the harness after changes that could affect answers

You do not need to run the entire suite after every typo fix.

Rerun relevant questions when you change something that could affect what the chatbot retrieves or explains, such as:

• an important procedure
• an API reference
• authentication or permissions
• a product behavior
• a source included in the Biel project
• chatbot configuration
• terminology readers use in their questions

For a large documentation release, running the full set can give you a quick regression check before publishing.

For a smaller change, run the cases related to that part of the documentation.

Add automated checks only where they help

Once you know the response fields you want to test, you can add simple automated checks.

For example, you might flag:

• an empty answer
• a missing expected term
• a phrase that describes known unsupported behavior
• an API or network failure

But avoid turning exact wording into the definition of a good answer.

Two correct answers can use different language, and a response can contain the expected keyword while still giving the wrong procedure.

Use automation to catch obvious failures. Use the documentation to decide whether the answer is actually correct.

Grow the test set when you find real regressions

Your evaluation set should come from the tasks that matter to readers.

Add a case when:

• a documentation change breaks an important answer
• support repeatedly sees the same question
• analytics exposes a recurring Content Gap
• a product release changes an important workflow
• a reader asks the same task using wording your original test did not cover

Remove cases when the underlying task is no longer supported.

The goal is not to maximize the number of questions. It is to maintain a short set that catches problems you would care about shipping.

Frequently asked questions

How many questions should an evaluation harness include?

Start small. A set of 5 to 20 questions covering important reader tasks is enough to make the process useful. Add cases when real changes or regressions show you something important is missing.

Is a documentation chatbot the same as a documentation AI assistant?

The terms are often used for similar products. In this guide, "documentation chatbot" means an AI interface that answers questions from your documentation. The same evaluation approach can also be used for an AI documentation assistant exposed through chat, an API, or another interface.

Should every answer be checked automatically?

No. Automated checks are useful for obvious failures, but they cannot reliably determine whether every technical claim is supported by your current documentation.

Should each test start a new chat?

Usually, yes. Starting a new chat keeps previous messages from changing the result. If you want to test a multi-turn workflow, create a separate test case for that conversation.

When should we rerun the harness?

Run the relevant cases after documentation, product, source, or configuration changes that could affect the answers. Run the full set before larger releases when you want a broader regression check.

What should we do when a test fails?

Read the answer and its source documentation first. Fix the documentation, source configuration, or chatbot setup that caused the problem, then run the same case again.

Start with the questions you do not want to break

Pick five or ten important documentation questions, save them in a fixture, and run them through the Biel API.

Keep the responses with your documentation change, review them against the current sources, and rerun the same questions when those sources change.

That is enough to turn a few manual chatbot checks into a repeatable regression test.

If you want to build and test a documentation chatbot or AI documentation assistant with your own content, create a Biel.ai project and follow the Quickstart to connect your documentation.