Evaluations API

Browse held-out LLM benchmarks such as AIME, GPQA and MMLU-Pro. Evaluate a pinned model with workspace approval and an evaluations:write key.

Read the evaluation catalog

GET /api/v1/evaluations/catalog is public. It describes qualified environments, held-out sets, available modes, ordering state, and limits. Use its set identities, counts, thinking support, and token guidance to prepare a request. The catalog currently lists AIME 2025, AIME 2026, GPQA Diamond, IFBench and a fixed 500-question sample of MMLU-Pro; read the response for the current sets.

Authenticated routes require evaluations:write. Dataset-order keys do not grant evaluation access. To evaluate a model on your own questions, use a Batch file with references instead.

Evaluation catalog · public read

curl --fail-with-body --silent --show-error \
  'https://api.reliqua.ai/api/v1/evaluations/catalog'

Evaluations on your file — Use your own reference answers through the Batch API.

Teutonic-I-Reliquary benchmarks — Read the documented protocol and results for a published model.

Routes

Method and pathPurpose
POST /api/v1/evaluations/quotesValidate and price the evaluation configuration.
POST /api/v1/evaluationsConfirm quote_id with Idempotency-Key; optional client_ref and webhook_url.
GET /api/v1/evaluationsRead your last 100 evaluations; optional exact quote_id and client_ref filters.
GET /api/v1/evaluations/:idRead owned status, progress, completeness, summary, and delivery metadata.
POST /api/v1/evaluations/:id/cancelCancel when the current state permits it.
GET /api/v1/evaluations/:id/files/:nameDownload an owned delivery file (graded.parquet, report.json, manifest.json, or part-<k>- files for several sets).
POST /api/v1/evaluations/webhook-secretCreate or rotate the account’s webhook secret.

Prepare the quote configuration

The quote selects model, revision, envs, max_new_tokens, and optional thinking, sampling, mode, model_access and episode settings. model is a Hugging Face repository; revision is its full 40-character commit. Public access is the default. Private access requires direct mode, current account approval and an operator-configured repository allowlist and read credential; a client cannot supply or expand that credential. Each envs entry includes env, problems, samples, and optionally set_id.

An omitted set_id selects the newest qualified set for that environment. Use an explicit set_id when comparing experiments. The server validates set size, model support, context limits, pricing, and the maximum feasible generation window.

Subnet mode is the default. Direct mode must be explicitly enabled on the deployment. A successful quote provides a price ceiling and estimates; confirmation reserves the quoted amount.

Quote one reviewed environment; no confirmation or reservation

: "${EVALUATIONS_API_KEY:?}" "${MODEL_REPO:?}" "${MODEL_REVISION:?Full 40-character commit}" "${EVALUATION_ENV:?}" "${EVALUATION_SET_ID:?}"
jq -n --arg model "$MODEL_REPO" --arg revision "$MODEL_REVISION" \
  --arg env "$EVALUATION_ENV" --arg set_id "$EVALUATION_SET_ID" \
  '{model:$model,revision:$revision,envs:[{env:$env,set_id:$set_id,problems:1,samples:1}],max_new_tokens:1024,thinking:false,mode:"subnet"}' > evaluation-quote.json
curl --fail-with-body --silent --show-error --max-time 20 \
  -H "Authorization: Bearer $EVALUATIONS_API_KEY" \
  -H 'Content-Type: application/json' --data-binary @evaluation-quote.json \
  'https://api.reliqua.ai/api/v1/evaluations/quotes'
FieldCurrent HTTP contract
envs1–8 entries, each set_id once; one sampling, max_new_tokens and thinking apply to all. Each problems count is 1–5,000 and samples 1–64; at most 10,000 problems total, bounded by each frozen set.
max_new_tokensInteger 16–131,072, subject to model context and execution feasibility.
samplingOptional temperature 0–2 (strictly positive in subnet mode), top_p 0.01–1, top_k=-1 or 1–1,000, seed 0–2^31-1. Defaults: 0.6, 0.95, 20 and seed 0. The seed contributes to the forced draw for each job. The job ID and checkpoint fingerprint also affect the draw, so two orders with the same settings can draw different samples.
modesubnet by default; direct requires explicit deployment support.
model_accesspublic by default; private requires approved direct execution and repository access.
harness / max_turns / max_episode_tokens / episode_timeout_secondsExplicit episode settings require direct mode, a maintained qualified harness, an explicit sampling seed, and frozen protocol support from every selected set. Server limits include 1–48 turns, at most 65,536 episode tokens and 3,600 seconds; context and set bounds can be lower.
revision / set_idFull model commit and exact qualified set identity; record both for comparisons.

Example: quote a catalog set and order it

This script reads the catalog and quotes two AIME 2025 problems on Qwen/Qwen3-8B with one sample each. A quote reserves nothing. To order it, POST {"quote_id"} to /api/v1/evaluations with an Idempotency-Key header (16–80 letters, digits, underscores or hyphens): that reserves price_usd. Then follow GET /api/v1/evaluations/:id until it is delivered and download its files.

eval_catalog_curl.sh · quote only

# Evaluation on a catalog set with curl: read the catalog and request a quote. QUOTE ONLY: nothing
# is ordered. The commented step 3 shows how to order the quote.
#
# Needs: curl, jq, and RELIQUARY_API_KEY set to an API key with the evaluations:write scope
# (an orders:write key is refused on /api/v1/evaluations).
set -euo pipefail
: "${RELIQUARY_API_KEY:?Set RELIQUARY_API_KEY to an evaluations:write API key}"

APP=https://app.reliqua.ai
AUTH="Authorization: Bearer $RELIQUARY_API_KEY"
# Catalog evaluations take a full 40-character commit (there is no main pinning on this route).
REVISION=b968826d9c46dd6066d109eabc6255188de91218

call() {
  local out
  if ! out=$(curl --fail-with-body --silent --show-error "$@"); then
    echo "$out" >&2
    exit 1
  fi
  echo "$out"
}

# 1. The public catalog: environments, their sets and the limits.
CATALOG=$(call "$APP/api/v1/evaluations/catalog")
echo "$CATALOG" | jq '{ordering, max_problems_per_env, max_problems, envs: [.envs[] | {env, sets: [.sets[] | {set_id, name, count, thinking}]}]}'

# 2. Quote 2 problems of AIME 2025, one sample each, 64 new tokens, thinking off.
QUOTE=$(call -H "$AUTH" -H 'Content-Type: application/json' \
  --data "$(jq -n --arg rev "$REVISION" '{
    model: "Qwen/Qwen3-8B",
    revision: $rev,
    envs: [{env: "verifiers:aime25", set_id: "verifiers-aime25-r0-n30", problems: 2, samples: 1}],
    max_new_tokens: 64,
    thinking: false,
    sampling: {temperature: 0.6, top_p: 0.95, top_k: 20, seed: 0}
  }')" \
  "$APP/api/v1/evaluations/quotes")
echo "$QUOTE" | jq .

# 3. To order it (reserves price_usd from your balance), uncomment:
# QUOTE_ID=$(echo "$QUOTE" | jq -r .quote_id)
# call -H "$AUTH" -H 'Content-Type: application/json' \
#   -H "Idempotency-Key: doc-eval-catalog-$(date +%s)-$RANDOM" \
#   --data "$(jq -n --arg q "$QUOTE_ID" '{quote_id: $q}')" \
#   "$APP/api/v1/evaluations" | jq .

Use the installed evaluation client

Use createEvaluationTransport with origin, a server-side evaluations:write key and an optional injected fetch, then pass its request function to createEvaluationClient. The bounded transport does not manage webhook secrets or retry mutations automatically.

Review the quote before confirmation. Persist the original request key, quote ID and client reference together. A timeout, HTTP 408 or server failure can mean the mutation succeeded: use the exact owned filters and original request identity to reconcile. client_ref is a correlation label, not a uniqueness constraint or a replacement for Idempotency-Key. A conflict is not permission to submit a different order.

Read and quote with an existing bounded transport

import { createEvaluationClient } from '@reliquary/api/evaluations';

// request is createEvaluationTransport({ origin, apiKey }).request.
// quoteInput uses the pinned model and qualified set selected above.
const evaluations = createEvaluationClient(request);
const catalog = await evaluations.catalog();
const quote = await evaluations.quote(quoteInput);
// Show quote.price_usd, quote.expires_at and configuration for review.

Confirm a reviewed quote and reconcile its receipt

import { createEvaluationClient } from '@reliquary/api/evaluations';

// Persist reviewedQuoteId, requestKey and clientRef before sending.
const evaluations = createEvaluationClient(request);
const order = await evaluations.create(reviewedQuoteId, requestKey, clientRef);
const receipt = await evaluations.detail(order.id);
const matches = await evaluations.orders({ quote_id: reviewedQuoteId, client_ref: clientRef });
if (!matches.evaluations.some((item) => item.id === receipt.id)) {
  throw new Error('The owned receipt could not be reconciled.');
}
// If confirmation throws outcome_unknown, perform the filtered read with
// these same saved values before deciding whether to retry the same action.

Evaluation OpenAPI — Exact installed request and response schemas.

Read a result in context

complete remains null before grading and can be false on a delivered evaluation. Inspect both the overall flag and per-environment completeness in summary. Keep report.json and manifest.json with the model revision and generation provenance.

Reports can include pass@1, confidence intervals, and pass@k by environment. Compare results only when the model revision, set identity, sample count, and generation settings are comparable.

In subnet mode every sample follows a public forced draw seeded by sampling.seed, which the subnet can check. Outputs are audited before grading, and tokens that were not verified are refunded at delivery. An evaluation over several sets runs one job per set and delivers a combined report.json and manifest.json with each set’s part-<k>-graded.parquet, part-<k>-report.json and part-<k>-manifest.json; a single set delivers graded.parquet, report.json and manifest.json. A set without a report leaves complete false and its share of the grading fee is refunded.

Use webhooks carefully

Create a webhook secret before supplying a public HTTPS webhook_url on confirmation. The API sends evaluation.state_changed events and evaluation.delivered with the result summary. Delivery failures are retried without blocking the evaluation.

X-Reliquary-Signature is t=<Unix seconds>,v1=<hex HMAC-SHA256>. Verify v1 over the exact timestamp string, a period and the raw request-body bytes with your webhook secret; do not parse and reserialize JSON first. Compare signatures in constant time, enforce a local timestamp tolerance and deduplicate X-Reliquary-Delivery before processing.

X-Reliquary-Event identifies evaluation.state_changed or evaluation.delivered. The JSON contains id, event, evaluation_id, status, status_reason, needs_attention and occurred_at (ISO8601), plus event-specific result data. Acknowledge valid persisted deliveries with 2xx. Treat events as notifications and reconcile the owned evaluation state; duplicated or delayed delivery does not authorize a new order.

Evaluations on your file — Your own questions and references, graded after a Batch run.

Authentication — Create the correctly scoped evaluation key.

Errors & retries — Handle unavailable models, expired quotes, and state conflicts.