Swagger Petstore - OpenAPI 3.0
POST/evals

Create an evaluation

Creates a new evaluation structure that defines testing criteria and data source configuration for model performance testing. Supply data_source_config and testing_criteria at minimum — name and metadata are optional. The created evaluation can then be run against different models and parameters.

  • IdempotentThe SDK sends Idempotency-Key, so a retried request is only applied once.

4 body fields

Evaluation structure with testing criteria and data source configuration.

namestringoptional
The name of the evaluation.
metadataobjectoptional
Optional set of up to 16 key-value pairs for storing additional structured information. Keys are strings with a maximum length of 64 characters.
data_source_configobjectrequired
The configuration for the data source used for the evaluation runs. Dictates the schema of the data used in the evaluation.
testing_criteriaarray<LabelModelGrader>required
A list of graders for all eval runs in this group. Graders can reference variables in the data source using double curly braces notation, like `{{item.variable_name}}`. To reference the model's output, use the `sample` namespace (ie, `{{sample.output_text}}`).

2 status codes
201Returns the created evaluation object with its unique identifier, name, data source configuration, testing criteria, creation timestamp, and metadata.
objectstringrequired
The object type.
Allowed:evalDefault:eval
idstringrequired
Unique identifier for the evaluation.
namestringrequired
The name of the evaluation.
data_source_configobjectrequired
Configuration of data sources used in runs of the evaluation.
testing_criteriaarray<LabelModelGrader>required
A list of testing criteria.
created_atintegerrequired
The Unix timestamp (in seconds) for when the eval was created.
metadataobjectrequired
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
429Returned when the request rate limit is exceeded.
errorobjectrequired

Error handling

data_source_config and testing_criteria are required — a request omitting either is rejected. data_source_config must be one of: CustomDataSourceConfig, LogsDataSourceConfig, or StoredCompletionsDataSourceConfig. testing_criteria must be a non-empty array of grader objects. A 429 is returned when the request rate limit is exceeded.