Choose a Batch or LLM evaluation workflow
Compare JSONL batch inference, LLM evaluation on your own questions, and held-out benchmark evaluation. Check inputs, outputs and access before starting.
Choose by the result you need
| Goal | Input | Result | Access |
|---|---|---|---|
| Generate completions for your prompts | OpenAI Batch JSONL and a pinned public model | Verified completions and per-request errors | Approved dataset access and orders:write |
| Grade answers to your own questions | Batch JSONL with a reference answer per request and a supported grader | Verified completions and a grading report | Approved dataset access and orders:write |
| Compare a checkpoint on a held-out benchmark | A pinned model and a suite from the evaluation catalog | Benchmark scores and evaluation receipts | Public catalog reads; ordering requires pilot access and evaluations:write |
Batch API — Prepare, submit and download a JSONL batch.
Evaluations on your file — Add reference answers and choose exact match or boxed math.
Evaluations API — Check which benchmark suites can be ordered.
Compare two checkpoints on the same task
Keep the question file or benchmark suite, grader, sampling settings and output limits fixed. Record each model revision and check delivery completeness before comparing scores.
Compare two LLM checkpoints on your own questions — Reuse one evaluation file and report scores alongside delivery coverage.
Turn JSONL prompts into LLM training data — Match completions to prompts and review them before exporting a dataset.
Teutonic-I-Reliquary benchmark report — See a published comparison with task counts, settings and limitations.