Metadata API Reference for Testing API

To define tests, use the AiEvaluationDefinition Metadata API type.

Represents an evaluation's purpose, the agent being tested, and the test sets to be executed.

In Metadata API, you can create test definitions, including specifying inputs and expected outcomes, and deploy them to different orgs. In Connect API, you can execute test scenarios, poll for results, and retrieve test outcomes.
This type extends the Metadata metadata type and inherits its fullName field.

AiEvaluationDefinition components have the suffix .aiEvaluationDefinition and are stored in the aiEvaluationDefinitions folder.

AiEvaluationDefinition components are available in API version 63.0 and later.

AiEvaluationDefinition is available only if Agentforce is enabled. See Set Up Agents in Salesforce Help.

Field Name	Type	Description
description	string	The purpose of the test.
name	string	Required. The API name of the test. Can contain only underscores and alphanumeric characters and must be unique in your org. It must begin with a letter, not include spaces, not end with an underscore, and not contain two consecutive underscores.
subjectName	string	Required. A unique identifier for the agent being tested. Make sure that this identifier matches the API name of the agent, which you can find on the agent details page in Setup.
subjectType	string	Required. The type of subject being tested. The only currently supported value is AGENT.
subjectVersion	string	The agent version to test. If not provided, the latest active version is used by default. You can find the version in the BotVersion metadata type.
testCase	AiEvaluationTestCase[]	A list of test cases.

Represents a test case.

Field Name	Type	Description
expectation	AiEvaluationExpectation[]	The criteria used to test the artifact's responses.
inputs	AiEvaluationAgentTestCaseInput[]	The specific input provided to the artifact being tested.
number	int	The unique number for the test case. If not provided, the value is automatically calculated.

Represents the expected outcome for a test case.

Field Name	Type	Description
expectedValue	string	The expected outcome of the test. The format of this field depends on the value of the name field. The expected outcome is compared against the response generated when you run the test using Connect REST API.
label	string	An optional label for an expectation. Typically added when using the same custom expectation name multiple times in a test case. If provided, this label appears in the test results; otherwise, the expectation name appears.
name	string	Required. The expectation name. Valid values are: topic_sequence_match—The expectedValue field value is a string representing the topic that the agent is expected to use, such as OOTBSingleRecordSummary. For a list of agent topics, see Standard Agent Topic Reference in Salesforce Help. action_sequence_match—The expectedValue field value is a string[] representing a list of actions that you expect the artifact to take during the test, such as ['IdentifyRecordByName', 'action2']. For a list of agent actions, see Standard Agent Action Reference in Salesforce Help. This option was previously called action_sequence_match. bot_response_rating—The expectedValue field value is a string representing the expected response generated by the artifact, such as Summarization of the Global Media account. coherence—A generated answer is coherent if it’s easy to understand and has no grammatical errors. If you use this quality check, you don't need an expectedValue field value. completeness—A generated answer is complete if it includes all the essential information. If you use this quality check, you don't need an expectedValue field value. conciseness—A generated answer is concise if it's brief but comprehensive. Shorter is better. If you use this quality check, you don't need an expectedValue field value. output_latency_milliseconds—Latency in milliseconds from sending a request until a response is received. If you use this quality check, you don't need an expectedValue field value. string_comparison—A custom evaluation criteria that tests a response for a specified string value. numeric_comparison—A custom evaluation criteria that tests a response for a specified numeric value.
parameter	parameter[]	Required for custom test criteria. An array of parameters for the specific custom criteria defined by expectation.name. This field replaces expectedValue for custom test criteria.

Field Name	Type	Description
name	string	Required for custom evaluation criteria. The name of the parameter required by the evaluation. Valid values are: operator—Specifies the type of string or numeric comparison. equals—Checks if the actual value directly matches the expected value. contains—Checks if the actual value contains the expected value. startswith—Checks if Checks if the actual value begins with the expected value. endswith—Checks if the actual value ends with the expected value. equals—Checks for numerical equality. For numerical comparison only. greater_than_or_equal—Checks if the actual value is greater than or equal to (>=) the expected value. For numerical comparison only. greater_than—Checks if the actual value is greater than (>) the expected value. For numerical comparison only. less_than—Checks if the actual value is less than (<) the expected value. For numerical comparison only. less_than_or_equal—Checks if the actual value is less than or equal to (<=) the expected value. For numerical comparison only. actual—Represents the string or numeric value obtained during the test run that you want to evaluate. expected—The target string or numeric value to compare against the actual value. This is the same as expectedValue for out-of-the-box metrics.
value	string	Required for custom evaluation criteria. The value for the parameter. This field can be a literal value or a JSONPath expression if isReference is true. Typically, JSONPath expressions are used to dynamically retrieve actual parameters.
isReference	boolean	If true, indicates that value is a JSONPath expression referencing runtime data from the generatedData object returned by the Get Test Results resource. If true, the value must be a JSONPath string. The default value is false.

Represents the input for a test case.

Field Name	Type	Description
utterance	string	Required. The request sent to the agent.
contextVariable	contextVariable[]	An XML array of context variables sent to the agent. Each context variable contains these required fields: variableName: Required. A string that represents the name of the context variable. variableValue: Required. A string that represents the value of the context variable.

Here's an example of an AiEvaluationDefinition component.

This metadata type supports the wildcard character * (asterisk) in the package.xml manifest file. For information about using the manifest file, see Deploying and Retrieving Metadata with the Zip File.