A convincing demonstration shows that an AI application can complete one scenario. Everyday use requires a broader assessment: factual accuracy, integrations, access permissions, incomplete requests and behavior when something fails. The subject of evaluation is the complete software process, including the quality of its text.
Consider an illustrative assistant that answers questions from internal procedures and prepares request drafts. This example shows how to organize evaluation, what to include in a test collection and why checks continue after the initial release.
Define a correct result first
One request may have several acceptable phrasings. Requiring an exact match with a reference sentence can therefore misrepresent the task. It is often more useful to specify the result’s properties: the right source, preserved conditions, required fields and no unsupported factual claims.
For our assistant, success means finding the applicable procedure, explaining its steps and providing a source link. When information is absent, reporting that gap or asking a question can be correct. A well-written answer containing invented instructions does not meet the same criteria.
These expectations need an owner from the relevant business process. Developers can check structure and execution, while a person familiar with the procedure helps determine whether the explanation preserves what employees actually need to do.
Build a varied set of cases
Evaluation examples should represent intended use. Some can come from prepared real requests. Difficult or uncommon cases should be included deliberately so that a large volume of easy questions does not conceal their failures. A process specialist should explain the acceptable outcome.
- An ordinary question with one current supporting document.
- The same question phrased differently or with spelling mistakes.
- An incomplete request requiring clarification.
- Contradictory or outdated documents.
- A question not answered by the accessible material.
- A request for a document or record outside the user’s permissions.
- An unavailable service or an interrupted operation.
Keep the case description with its relevant sources and evaluation criteria. The team can then repeat it after a change and establish whether a problem was corrected, rather than simply absent from the next demonstration.
Separate retrieval, explanation and action
A wrong answer can have several causes. Retrieval might miss the applicable document, the model might omit an important condition, or an integration might return the wrong record. Looking only at the final text makes these different defects harder to distinguish.
For our example, first inspect the selected procedure, then compare the explanation with its content, and separately review the request draft’s fields. If the application can create a request, check confirmation from the receiving system. A success statement in chat needs to match the actual record.
This separation also helps assign fixes. Adjusting the model’s instructions may not solve a missing document, and changing the search configuration may not repair an incorrect field mapping in the integration service.
What can be checked automatically?
Programmatic checks are useful for data formats, required fields, permitted operations and responses to API failures. They can detect duplicate records after repeated requests and verify that users do not receive records outside their permissions. These properties should not depend on whether the model happened to phrase its response well.
For response content, use agreed criteria and review by a specialist. A second model can help organize or score outputs, but its judgments also need checking against examples with known evaluations. Its assessment is an additional signal rather than unconditional evidence of correctness.
The balance depends on the task. A draft’s tone may need human judgment, while whether a reference number exists in the returned record is a more direct check. Treating both as one general quality score can make the results difficult to interpret.
Why one accuracy percentage is insufficient
Imagine a collection containing many easy questions and a few cases with outdated procedures. An overall score can look encouraging even if the system consistently chooses the wrong version in that smaller group. Review results by task type and failure reason as well as in aggregate.
Measure the properties that matter to the process: source correctness, preservation of essential information, required revisions and time to complete the whole task. For drafts, also consider whether an employee can understand the basis of the answer and verify important facts efficiently.
Repeated runs can help assess variation in generation, but they should not conceal unsuccessful outputs. Preserve failed cases and check them after corrections. Over time, this becomes a regression collection that helps detect the return of known problems.
Organizing a pilot
Begin with an agreed user group and a defined task scope. Employees should review results and record concrete issues: the request, source material, output and expected behavior. This provides more useful evidence than a general report that the assistant is unreliable.
Decide who investigates cases and how the team responds to a significant defect. Sometimes the appropriate fix is to correct a document or temporarily remove a capability rather than replace the model. The pilot also needs a clear way to return a task to a person.
Include the effort required to use the application. If employees need to copy information between several interfaces to review every draft, the complete workflow may require improvement even when individual answers are accurate.
Checks after changes
Behavior can change with model versions, instructions, retrieval settings, source documents and external APIs. Relevant examples need to be run again after those changes. Previous failure cases and important permission boundaries are particularly useful parts of that review.
Associate evaluation results with a specific application version and data collection. Otherwise a team may struggle to explain why an answer differs from the one shown during acceptance. The appropriate scope of re-evaluation depends on what changed and how the application is used.
Evidence a team needs before release
Before release, retain agreed criteria, results grouped by case type, known limitations and named maintenance owners. This gives the team a basis for deciding how to use the application and which improvements to prioritize.
Testing makes behavior observable and understandable. The development preparation guide explains how to establish these checks early. For applications that act through external services, the article on business-system integrations covers the additional execution requirements.
