Build Tests in Metadata API

To define tests, use the AiEvaluationDefinition Metadata API type. To learn how to use Metadata API, see Quick Start: Metadata API.

To use Salesforce CLI to create agent tests instead of directly using Metadata API, see Test an Agent with Agentforce DX.

The AiEvaluationDefinition metadata type contains a set of test cases. Each test case takes inputs (including an utterance) and contains a set of expectations (such as an expected action sequence) for the response.

In addition to an utterance, a test case input can contain context variables. These variables allow you to create more nuanced tests on how agents behave in different contexts, and determine the overall robustness of an agent in scenarios that better simulate a production environment.

For more information on context variables, see the Standard Variable Reference.

By default, context variables are immutable and are only set at the beginning of an agent session. The only context variable that is editable after a session begins is EndUserLangauge.

‌This sample XML AiEvaluationDefinition has two test cases for the Agentforce_for_Salesforce agent. The first test case provides an utterance (“Summarize the Global Media account”) and defines multiple expectations for the response.

  1. The first expectation verifies that the OOTBSingleRecordSummary topic is used.
  2. The second expectation verifies that the IdentifyRecordByName action is used.
  3. The third expectation includes a string that's expected in the test response.
  4. The fourth expectation uses the conciseness quality metric to gauge whether the generated answer is brief but comprehensive. Shorter is better.

See AiEvaluationDefinition.

To deploy metadata components with Salesforce CLI, Deploy and Run Tests in the Command Line.