Evaluation

One conversation tells you how one answer went. It cannot tell you whether uploading a new price list, rewriting the system prompt or switching models made your assistant better overall — answers vary, and you remember the memorable ones. Evaluation replaces that impression with a measurement: a fixed set of questions, answered the way a visitor would be answered, scored the same way every time, and plotted run after run.

The Evaluation page in the portal has two views:

• Set & runs — your question set, its runs and the trend. This is the part you come back to after every change.
• Ask one question — answers and scores a single question on the spot, with the retrieved content and the judge's reasoning. Use it to debug one answer. Nothing is stored and it never appears in the trend.

Each chatbot has one question set: the handful of questions you most need answered well. Start by generating questions from your own content, write your own, or both.

• Generate from your content reads your knowledge base and proposes up to ten questions. It keeps only questions your content can actually answer, so it sometimes returns fewer than ten — that is the filter doing its job.
• Expected answer (optional) is an example of a good answer, written by you. The judge uses it to decide what a relevant, complete answer covers. It is not treated as ground truth: different wording or extra correct detail is not penalised, and faithfulness is always judged against your retrieved content.
• Expected sources (optional) are the documents the answer should come from, comma separated, as they appear in your Knowledge Base — for example refund-policy.pdf, faq.md. Matching ignores case and accepts part of a name. Naming sources turns on the Right source retrieved check.
• Enabled controls whether a question is included in runs. Your plan sets how many questions can be enabled; if you move to a smaller plan, extra questions are disabled, never deleted.

1. Nubo records the configuration the run used: chat model, response style (temperature), reranking, retrieval depth, a fingerprint of your system prompt and a fingerprint of your knowledge base. That record is what lets the trend say what changed between two runs.
2. Each enabled question is answered through the same pipeline as a real conversation: search your knowledge base, rerank the results if reranking is on, and answer with your chosen model.
3. Each answer is scored by the judge, described below. Progress updates live on the page.
4. When every question is done the run is marked complete and joins the trend. You can cancel a run while it is in progress; a cancelled run is kept but not plotted.

If an answer cannot be produced, it is retried a few times. A question that still has no answer is marked Failed.

Every answer is read by an AI judge — a language model given a fixed scoring rubric. For each answer the judge sees:

• the question;
• the content that was retrieved and handed to the assistant, with the source of each passage;
• the assistant's answer;
• your expected answer, if the question has one;
• a note naming any tools the assistant asked for but did not get.

It is told to judge only against the retrieved content — not against the web, and not against what it happens to know — and not to reward or punish style or length. It runs with randomness turned down to zero, so the same answer scores consistently from run to run. Before choosing any number it writes down its reasoning and lists the claims it could not find support for; both are shown on each result as Why it scored that way and Unsupported claims.

Each answer gets four scores from 1 to 5, and an overall yes/no on whether it is grounded in the content. If the judge itself fails on an answer, that answer is shown as Not scored rather than given a score it did not earn.

Read them in this order. Context sufficiency describes what the assistant had to work with; the other three describe what it did with it.

Grounded, shown when you ask one question, is the judge's overall call on whether the answer is built on the retrieved content.

The summary at the top of a run:

Each question in the run carries one status:

• Passed — all four scores reached the threshold.
• Below threshold — at least one score fell short. Open the result to see which, and why.
• Failed — no answer could be produced.
• Not scored — an answer was produced, but the judge could not score it.

The trend plots every completed run, oldest first, spaced evenly rather than by date — runs are events, not a schedule. The four average scores share one 1-to-5 axis; the pass rate sits beneath them as a percentage of scored answers. You need two completed runs before there is anything to compare.

Wherever the configuration differs from the run before, a marker names the change, so a drop explains itself:

• Knowledge base changed — documents or pages were added, removed or updated.
• System prompt changed
• Model A → B
• Reranking on / Reranking off
• Top-K 5 → 8 — how many passages are retrieved per question.
• Temperature 0.3 → 0.7 — the response style.

Full per-question detail is kept for your most recent runs; how many depends on your plan. Older runs keep their summary scores, so the trend stays complete.

• The judge is a language model, and it may be from the same family as the model that wrote the answer, which can flatter the answer slightly. That bias is the same in every run, so treat a single score as a signal and the comparison between runs as the measurement.
• A run uses your live configuration and knowledge base, so run it after a change is saved, not while you are still editing.
• Ask one question is for investigating a single answer. Its results are not stored and never enter the trend.
• Your plan sets how many questions you can enable, how often you can run the set, and how many runs keep full detail.