Generate a Test Spec File

A test spec is a YAML file in your DX project that lists the test cases for a particular agent. The test spec file is local to your DX project and is an easy-to-read equivalent of the AiEvaluationDefinition metadata component that defines an agent test in your development org.

Run this command from the VS Code integrated terminal to generate a test spec:

As with all Salesforce CLI commands, you can also generate a test spec by opening a terminal (macOS, Linux) or command prompt (Windows), changing to your DX project directory, and running the agent generate test-spec CLI command.

The agent generate test-spec command asks you what test you want to make (currently only agent is supported), the API name of the agent you want to test, and the name and description of your new test. The command doesn’t require an org. Instead, it looks in your local DX project to compile the list of agent API names, topics, actions, and so on. The command then prompts you to create one or more test cases by asking you for this information for each test case:

  • Utterance - A natural-language statement, question, or command to test the agent. Enter utterances that you expect the users of your agent to use. For example: “Who is working the front desk today at noon?” Or “Send me the list of customer complaints from yesterday.”

  • Expected topic - The API name of the topic that you expect the agent to use when it responds to the utterance. The command gets the list of topics from the local metadata components associated with your agent, specifically GenAiPlugin.

  • Expected actions - The API name of one or more actions associated with the topic that you expect the agent to use. The command gets the list of actions from the local metadata components associated with the agent topic, specifically GenAiFunction.

  • Expected outcome - A natural-language description of the expected outcome of the test case. For example, “Anna is working the front desk.” Or “The list of customer complaints is provided.”

This is an example of a generated test spec file for the Resort_Manager agent. By default, the test spec is generated in the specs directory and its name is {Agent_API_name}-testSpec.yaml. This sample spec file defines one test case:

The agent generate test-spec command automatically adds the metrics: section; see Use Metrics to Assess Agent Responses for more information about these useful metrics.

Now that you understand the structure of the agent spec YAML file, you can manually edit it in the VS Code editor by adding new test cases.

If you have an AiEvaluationDefinition metadata XML file in your DX project, you can make a YAML test spec file from it. You can do this by passing the file path of the XML file to the --from-definition flag. This example also shows how to use the --output-file flag to specify the name of the spec file: