A documentation assistant can give a fluent answer for the wrong software version. It might retrieve an obsolete setup guide, combine it with a current troubleshooting note, and present both as one procedure. Checking whether the answer sounds reasonable will not reveal which component failed. The retriever may have selected unsuitable evidence, the generator may have misread useful evidence, or both may have contributed.
For a knowledge base application, evaluate these stages separately before judging the complete system. Retrieval testing asks whether the right material was available. Generation testing asks whether the answer accurately uses a fixed set of material. End-to-end testing then checks what happens when those stages interact under realistic application conditions.
When considering an API integration through GPTZZZ, keep these responsibilities explicit. Confirm the interface and account capabilities separately; the application team still needs its own retrieval checks, evidence records, and answer review. A successful API response is not, by itself, evidence that an answer is correct.
Define a useful answer before building a scorecard
Start with the questions the assistant is supposed to handle. A developer documentation tool might need to explain configuration, identify required permissions, or distinguish a supported operation from a deprecated one. Each task has different conditions for success. An answer about permissions needs the relevant restriction, not just a matching function name.
Choose the authoritative source for each case and record the applicable product, version, and effective date. The newest document is not automatically the right one when the question concerns an older release. If the available documentation conflicts, resolve that conflict or label the case as requiring clarification rather than silently choosing a preferred answer.
Specify acceptable outcomes beyond answering. Asking the user which version they use, stating that the available material is insufficient, or directing a sensitive case to review can be correct behavior. Keep these application decisions separate from authentication failures, network errors, and model safety refusals. They describe different events and require different fixes.
Build a reusable question and evidence set
Select questions from representative workflows or suitably redacted support records. Include short questions, informal wording, abbreviations, and questions containing several requirements. Also include cases with no supported answer, misleading terminology, and similar features from different product versions. A collection consisting only of straightforward examples can conceal important weaknesses.
For each question, save the expected evidence and the reason it matters. Some questions can be answered from one paragraph; others require a prerequisite, an action, and an exception from separate sections. Mark these pieces separately. Otherwise, finding a relevant paragraph could be mistaken for finding everything needed to answer.
In a hypothetical documentation example, a question asks how to rotate an application credential. The expected evidence might include the authorized role, the rotation procedure, and the effect on existing connections. A page describing only how to create a credential is related, but does not cover the task. Labeling that distinction makes the test useful for debugging.
Reserve some representative questions for evaluation rather than repeatedly tuning against the same set. Keep a separate set of previously discovered failures for regression checks. When reference documents change, review the affected labels instead of assuming that yesterday’s expected answer remains valid.
Evaluate retrieval without generating an answer
Run the questions through retrieval alone. Record the query, index revision, filters, ranking configuration, and the number of results requested. Preserve each returned document or passage identifier, its revision, and its position. A link to a page that changes later may not be enough to reproduce the evidence the application actually supplied.
Review relevance and coverage separately. A result can be closely related to the question while omitting the condition needed for a safe answer. Check whether the selected material matches the requested version, whether important exceptions are present, and whether duplicate passages are crowding out distinct evidence.
Define every metric’s denominator. For example, Hit@K can mean the share of answerable test questions with at least one labeled relevant result among the first K results. That measures an initial match, not complete support for a multi-part answer. Add an evidence coverage measure for cases that require several pieces, and report unanswerable questions separately.
Review failures by category, not only through an overall average. A good aggregate result can conceal a consistent problem with an older release or a particular document type. Establish acceptance thresholds from the task’s consequences and the available test set; a convenient number copied from another application is not an acceptance argument.
Test generation against fixed evidence packets
Once retrieval behavior is understood, give the generation step fixed packets of evidence. Keep the question, passages, passage order, and application instructions recorded. This makes a comparison interpretable: an answer changed while its supplied evidence remained the same. It does not guarantee identical outputs across repeated calls.
Define required and forbidden claims instead of demanding an exact reference sentence. In the hypothetical credential example, the answer might need to preserve the authorized role and explain the documented connection behavior. If the evidence provides no expiration interval, a specific interval invented by the answer is unsupported even if it sounds plausible.
Check citations at the claim level. A citation identifier must point to a passage that actually supports the adjacent statement. The mere presence of a source title or a working link does not establish support. Review whether the answer keeps qualifications such as “only when,” “unless,” and “not supported” rather than retaining the main action and dropping its limits.
Include packets with missing evidence and conflicting passages. Define whether the application should ask a clarification question, explain the gap, or request review. For documents containing instruction-like text, also check that quoted source material is treated as evidence rather than as permission to change the task or perform an action.
Reconnect the stages and check application boundaries
Run the complete path using the same evaluation questions and additional realistic cases. Examine whether passage selection, context assembly, or length limits remove evidence that was present in the retrieval results. A retriever can pass its own test while a later context builder drops the exception that made the answer accurate.
Keep authorization outside the model’s judgment. The application should retrieve only material the requesting user is allowed to access, using enforced permissions rather than asking the model to hide restricted passages after retrieval. Test access boundaries with approved synthetic accounts or fixtures, without putting confidential material into an unrestricted test environment.
Measure operational behavior separately from answer quality. Record timeouts, incomplete responses, and retries as such. Do not score an empty response as an unsupported factual claim or count a recovered request twice as two successfully completed user tasks. If retries are enabled, track attempts and applicable usage costs so that a quality improvement does not hide additional work.
Turn each failure into an inspectable record
A useful failure record includes the question identifier, applicable document versions, retrieved passages, the final evidence packet, relevant request configuration, the answer, and reviewer notes. Distinguish observed metadata from assumptions about what happened inside a provider. Do not infer an internal model revision merely from a change in writing style.
Store only what is needed for diagnosis. Redact credentials and personal information, control access to retained passages, and define retention periods. If full text cannot be retained, record an approved reference or digest and acknowledge the resulting limit on reproducibility. Debugging convenience does not justify unrestricted prompt and response logging.
Assign the repair to the stage supported by the evidence. Missing passages may call for indexing or retrieval work; correct passages followed by an unsupported claim point toward answer generation or validation. Fixing one stage does not remove the need to rerun the complete path, because a new ranking rule or shorter evidence packet can affect later behavior.
Make the release decision explicit
Before release, document which question types passed, which cases need clarification, and which operations remain subject to human review. Include unresolved failures and the conditions that would stop or limit deployment. Passing a narrow evaluation should authorize only the scope that was examined, not every future use of the assistant.
Repeat the relevant checks when documents, retrieval settings, application instructions, or API configuration change. Keep the earlier records so that reviewers can distinguish a regression from a change in the expected answer. The result is a practical acceptance process: locate the evidence, evaluate the answer against it, and verify that the combined application preserves both accuracy and access boundaries.
