Choose a Batch or LLM evaluation workflow

Compare JSONL batch inference, LLM evaluation on your own questions, and held-out benchmark evaluation. Check inputs, outputs and access before starting.

Choose by the result you need

GoalInputResultAccess
Generate completions for your promptsOpenAI Batch JSONL and a pinned public modelVerified completions and per-request errorsApproved dataset access and orders:write
Grade answers to your own questionsBatch JSONL with a reference answer per request and a supported graderVerified completions and a grading reportApproved dataset access and orders:write
Compare a checkpoint on a held-out benchmarkA pinned model and a suite from the evaluation catalogBenchmark scores and evaluation receiptsPublic catalog reads; ordering requires pilot access and evaluations:write

Batch API — Prepare, submit and download a JSONL batch.

Evaluations on your file — Add reference answers and choose exact match or boxed math.

Evaluations API — Check which benchmark suites can be ordered.

Compare two checkpoints on the same task

Keep the question file or benchmark suite, grader, sampling settings and output limits fixed. Record each model revision and check delivery completeness before comparing scores.

Compare two LLM checkpoints on your own questions — Reuse one evaluation file and report scores alongside delivery coverage.

Turn JSONL prompts into LLM training data — Match completions to prompts and review them before exporting a dataset.

Teutonic-I-Reliquary benchmark report — See a published comparison with task counts, settings and limitations.