Evaluations on your file

How it works

An evaluation on your file is a batch whose lines also carry the expected answer. The completions are generated and verified as for any batch, then each one is graded against its line’s reference and a report is delivered beside the completions.

It uses the Batch API routes and an orders:write key with dataset access. Evaluations on catalog sets use the Evaluations API and an evaluations:write key instead.

Batch API — File format, models, states, limits and pricing.

Evaluations API — Catalog sets: AIME, GPQA, IFBench, MMLU-Pro.

Evaluations on your file with an AI agent — The same workflow through the Reliquary MCP tools and official skill.

Add a reference to each line

Each line is a Batch line plus a top-level reference: a non-empty string of at most 4,096 characters. A line without one goes to the error file (reference_required or invalid_reference). A plain batch, without a grader, refuses lines that carry a reference (invalid_request).

An evaluation line

{"custom_id":"m1","method":"POST","url":"/v1/chat/completions","body":{"model":"Qwen/Qwen3-8B","messages":[{"role":"user","content":"What is 7 times 8? Put the final answer in \\boxed{}."}],"max_tokens":64,"temperature":0.6,"top_p":0.95,"top_k":20,"seed":7,"chat_template_kwargs":{"enable_thinking":false}},"reference":"56"}

Choose a grader

The answer is the part of the completion after the reasoning block. Another grader name is refused with 400 grader_unknown.

Order the evaluation

• OpenAI-compatible: POST /api/openai/v1/batches with metadata.reliquary_grader set to the grader. The batch object keeps OpenAI’s fields; the scores are on the order.
• Native: POST /api/v1/dataset-orders/quotes with grader next to model, revision and input_file_id, then POST /api/v1/dataset-orders {"quote_id"}. The quote and the order show purpose: "eval" and the grader.
• Price: the batch price plus a grading fee, both included in the reserved price (the quote shows the fee before the margin as pricing.grading_fee_usd). Unverified tokens are refunded as for a batch.
• The batch stays finalizing (order status exporting) while it is graded, and becomes completed once the report is stored.
• Capacity: like any batch, it starts at once on a model already running, otherwise it waits its turn while validating (reliquary_queue on the batch, capacity on the quote); after 24 hours in line it is released in full. See Capacity and queueing in the Batch API reference and GET /api/v1/capacity.

Read the results

GET /api/v1/dataset-orders/{order id} (the batch id without batch_) returns order.evaluation: {grader, status, complete, summary}. status is pending until the order ends, graded with a summary, or ungraded when the completions were delivered without a report; then the grading fee is refunded with the unverified tokens. complete is false when some samples are missing from the report.

summary.envs holds one entry named after the grader, with pass@1 and its 95 % confidence interval, pass@k, mean_score and row counts; summary.macro repeats pass@1 and pass@k over the entries.

The order’s deliveries hold the graded files under delivery evaluation (graded.parquet with one row per completion and its score, report.json, manifest.json) and the completions under dl-{order id}. Download each from its url with the same key; byte ranges are supported.

order.evaluation once graded (abridged)

{
  "grader": "boxed_math_v1",
  "status": "graded",
  "complete": true,
  "summary": {
    "complete": true,
    "envs": {
      "boxed_math_v1": {
        "n_problems": 2,
        "samples": 1,
        "graded_rows": 2,
        "mean_score": 1,
        "complete": true,
        "pass@1": { "value": 1, "ci95": [0.34, 1] },
        "pass@k": { "1": 1 }
      }
    },
    "macro": { "pass@1": 1, "pass@k": { "1": 1 } }
  }
}

Example with curl

This script uploads two questions with references, orders an evaluation graded by boxed_math_v1 on Qwen/Qwen3-8B, waits for it and prints order.evaluation. It places a real order: the batch price and the grading fee are reserved, and metadata.max_price_usd refuses it above 5 USD.

eval_file_curl.sh · places a real order

# Evaluation on your own file with curl: 2 questions with references, graded by boxed_math_v1.
#
# THIS PLACES A REAL ORDER and charges your Reliquary balance: the batch price (about 2 USD, mostly
# a fixed share of GPU time) plus the grading fee. metadata.max_price_usd below refuses the order
# if its worst-case price is above 5 USD. Unverified tokens are refunded at delivery.
#
# Needs: curl, jq, and RELIQUARY_API_KEY set to an API key with the orders:write scope.
set -euo pipefail
: "${RELIQUARY_API_KEY:?Set RELIQUARY_API_KEY to an orders:write API key}"

APP=https://app.reliqua.ai
BASE=$APP/api/openai/v1
AUTH="Authorization: Bearer $RELIQUARY_API_KEY"
# Qwen/Qwen3-8B at a fixed commit. Omit metadata.revision to pin the repo's current main commit.
REVISION=b968826d9c46dd6066d109eabc6255188de91218
WORK=$(mktemp -d)

call() {
  local out
  if ! out=$(curl --fail-with-body --silent --show-error "$@"); then
    echo "$out" >&2
    exit 1
  fi
  echo "$out"
}

# 1. Batch lines plus a top-level "reference": the expected answer for the grader.
cat > "$WORK/eval.jsonl" <<'EOF'
{"custom_id":"m1","method":"POST","url":"/v1/chat/completions","body":{"model":"Qwen/Qwen3-8B","messages":[{"role":"user","content":"What is 7 times 8? Put the final answer in \\boxed{}."}],"max_tokens":64,"temperature":0.6,"top_p":0.95,"top_k":20,"seed":7,"chat_template_kwargs":{"enable_thinking":false}},"reference":"56"}
{"custom_id":"m2","method":"POST","url":"/v1/chat/completions","body":{"model":"Qwen/Qwen3-8B","messages":[{"role":"user","content":"What is 1/2 + 1/3? Put the final answer in \\boxed{}."}],"max_tokens":64,"temperature":0.6,"top_p":0.95,"top_k":20,"seed":7,"chat_template_kwargs":{"enable_thinking":false}},"reference":"\\frac{5}{6}"}
EOF

FILE=$(call -H "$AUTH" -F purpose=batch -F "file=@$WORK/eval.jsonl" "$BASE/files")
echo "$FILE" | jq .
FILE_ID=$(echo "$FILE" | jq -r .id)

# 2. metadata.reliquary_grader turns the batch into an evaluation.
BATCH=$(call -H "$AUTH" -H 'Content-Type: application/json' \
  -H "Idempotency-Key: doc-eval-$(date +%s)-$RANDOM" \
  --data "$(jq -n --arg file "$FILE_ID" --arg rev "$REVISION" '{
    input_file_id: $file,
    endpoint: "/v1/chat/completions",
    completion_window: "24h",
    metadata: {revision: $rev, reliquary_grader: "boxed_math_v1", max_price_usd: "5"}
  }')" \
  "$BASE/batches")
echo "$BATCH" | jq .
BATCH_ID=$(echo "$BATCH" | jq -r .id)
ORDER_ID=${BATCH_ID#batch_}

# 3. Poll: an evaluation stays "finalizing" while it is graded.
while :; do
  BATCH=$(call -H "$AUTH" "$BASE/batches/$BATCH_ID")
  STATUS=$(echo "$BATCH" | jq -r .status)
  echo "$(date -u +%H:%M:%S) $STATUS $(echo "$BATCH" | jq -c .request_counts)"
  case "$STATUS" in completed | expired | cancelled | failed) break ;; esac
  sleep 60
done

# 4. The scores are on the native order: evaluation.summary, and the delivered files.
ORDER=$(call -H "$AUTH" "$APP/api/v1/dataset-orders/$ORDER_ID")
echo "$ORDER" | jq '.order | {status, status_reason, price_usd, refunded_usd, evaluation, deliveries}'
REPORT_URL=$(echo "$ORDER" | jq -r '.order.deliveries[] | select(.delivery_id == "evaluation") | .files[] | select(.name == "report.json") | .url')
if [ -n "$REPORT_URL" ]; then
  call -H "$AUTH" "$APP$REPORT_URL" > report.json
  echo "report.json saved"
fi