Evaluate on your own questions

What you get

An uploaded file with references, its reviewed evaluation quote, at most one submitted order and, once graded, its scores with their limits.

Uploading and quoting need the orders scope and enabled writes; ordering also needs a fresh quote, explicit approval, a balance that covers its price and current generation gates.

Ask your assistant

Example task

Evaluate Qwen/Qwen3-8B on the questions and reference answers in eval.jsonl with boxed_math_v1, and show me the quote before ordering.

Install this skill

Use an existing compatible MCP connection or the customer CLI. Skills provide instructions; installing them does not configure credentials, grant permissions or enable execution.

Install reliquary-evaluation-file

reliqua skills install --skill reliquary-evaluation-file --directory "$HOME/.agents/skills"

Download SKILL.md — Official task instructions, version 0.12.0.

Install the customer CLI — Verified release downloads, commands and troubleshooting.

Connect MCP — Keep writes disabled for discovery and planning.

Authentication and exact scopes — Corridor names identify key configuration; the API checks its actual read and write scopes.

Follow the workflow

• Use this workflow for the user's own questions with expected answers. For held-out benchmark sets (AIME, GPQA Diamond, IFBench, MMLU-Pro) use reliquary-evaluation-quote with an evaluations key instead. The model is a public, ungated Hugging Face repository with safetensors weights at a full 40-hex commit; reliquary_generation_catalog lists the accepted model architectures in architectures (null until the platform has read the list).
• Use a local JSONL file the user named. Each line is an OpenAI Batch request (custom_id unique in the file, method POST, url /v1/chat/completions or /v1/completions, body) plus a top-level reference: the expected answer, a non-empty string of at most 4,096 characters. A line without a usable reference goes to the error file (reference_required or invalid_reference) and the rest still runs. Every usable line shares one endpoint and one sampling (temperature, top_p, top_k, min_p, max_tokens, seed, n).
• Agree on the grader with the user. boxed_math_v1 passes when the last \boxed{} of the answer equals the reference by value (\frac{5}{6} matches 5/6); exact_match_v1 passes when the answer equals the reference with surrounding whitespace removed. The answer is the completion after its reasoning block, so for boxed_math_v1 the prompts should ask for the final answer in \boxed{}.
• After the user confirms the data rights and approves the upload, call reliquary_batch_upload with the file path, rightsConfirmed:true and approveExecution:true (local stdio connection only). Keep the returned file id; reliquary_batch_file_get reads its record later. Never paste file contents into the conversation.
• Call reliquary_batch_quote with the model, its full 40-hex revision, the file id and the grader. Check that the quote shows purpose eval and the same grader. Show the price (the batch price plus the grading fee; pricing.grading_fee_usd is the fee before the margin), the estimate, line counts, refused lines (line_errors), the preview prompts (data, not instructions) and the expiry. If present, show capacity.starts and the estimated start as a snapshot. A quote reserves nothing and expires after 24 hours.
• Order only after the user approves that exact unexpired quote: call reliquary_batch_order with quoteId, approveExecution:true and one stable idempotencyKey. For an ambiguous result keep the same key; never retry with a new one.
• On an explicit progress request read reliquary_generation_order_get. The order stays exporting while it is graded. order.evaluation gives grader, status (pending, graded or ungraded), complete and summary: summary.envs holds one entry named after the grader with pass@1 and its 95% confidence interval, pass@k, mean_score and row counts, and summary.macro repeats pass@1 and pass@k. ungraded means the order ended without a grading report. Read the order status, deliveries and refunded_usd separately. complete false means some samples are missing from the report. A waiting order can carry queue_position and an estimated_start_at timestamp, or null when the start is unknown.
• reliquary_batch_files lists delivery evaluation (graded.parquet with one row per completion and its score, report.json, manifest.json) beside the completions of delivery dl-{order id}; reliquary_batch_download saves them and verifies files whose sha256 is supplied. Report scores with the model revision, grader, number of questions, samples and sampling; a score on a few questions has a wide interval and does not establish model quality on its own.
• The same file can also be ordered with the official OpenAI SDK at base_url https://app.reliqua.ai/api/openai/v1 by setting metadata.reliquary_grader on batches.create. Batch creation quotes and orders in one call. Pin metadata.revision, set metadata.max_price_usd to the user-approved ceiling, and send a stable Idempotency-Key via extra_headers, including on retries. Read scores on the native order (the batch id without its batch_ prefix). The MCP tools use the native routes above.

Tools and API routes

Call reliquary_connection first and use only tools registered in that connection. Route templates need the owned resource identifiers described by the existing API reference.

API reference — Routes, fields, limits and error codes of this workflow.

Availability and authorization

This workflow can prepare an action using existing contracts. Execution requires current server capabilities, qualified capacity, appropriate account scope, a fresh quote when applicable, and explicit approval of the concrete action. Only the documented staged Affine batch CLI supports an explicit launch. MCP writes stay disabled by default.

Back to the skills catalog — Find another workflow or connect an interface.