Swagger Petstore - OpenAPI 3.0
POST/evals/{eval_id}/runs

Create an evaluation run

Triggers a new evaluation run for the specified evaluation against a data source and model configuration. The data source is validated against the evaluation's schema. A 400 is returned if the data source does not match the evaluation's schema or required fields are missing.

  • IdempotentThe SDK sends Idempotency-Key, so a retried request is only applied once.

1 parameter · 3 body fields
eval_idstringrequired
The unique identifier of the evaluation to create a run for.

Configuration for the evaluation run including data source and optional metadata.

namestringoptional
The name of the run.
metadataobjectoptional
Optional set of up to 16 key-value pairs for storing additional structured information. Keys are strings with a maximum length of 64 characters.
data_sourceobjectrequired
Details about the run's data source.

3 status codes
201Returns the created evaluation run object with its unique identifier, status, model, name, creation timestamp, report URL, result counts, per-model usage statistics, per-testing-criteria results, data source information, and metadata.
objectstringrequired
The type of the object. Always "eval.run".
Allowed:eval.runDefault:eval.run
idstringrequired
Unique identifier for the evaluation run.
eval_idstringrequired
The identifier of the associated evaluation.
statusstringrequired
The status of the evaluation run.
modelstringrequired
The model that is evaluated, if applicable.
namestringrequired
The name of the evaluation run.
created_atintegerrequired
Unix timestamp (in seconds) when the evaluation run was created.
report_urlstringrequired
The URL to the rendered evaluation run report on the UI dashboard.
result_countsobjectrequired
Counters summarizing the outcomes of the evaluation run.
per_model_usagearray<object>required
Usage statistics for each model during the evaluation run.
per_testing_criteria_resultsarray<object>required
Results per testing criteria applied during the evaluation run.
data_sourceobjectrequired
Information about the run's data source.
metadataobjectrequired
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
errorobjectrequired
An object representing an error response from the Eval API.
400Returned when the data source is missing, invalid, or does not conform to the evaluation's schema.
codestringrequired
messagestringrequired
paramstringrequired
typestringrequired
429Returned when the request rate limit is exceeded.
errorobjectrequired

Error handling

eval_id must identify an existing evaluation. data_source is required and must be one of: JsonlRunDataSource, CompletionsRunDataSource, or ResponsesRunDataSource. The data source must conform to the evaluation's data_source_config schema. A 400 is returned when the data source is invalid or missing. A 429 is returned when the request rate limit is exceeded.