How to compare two LLM checkpoints on your own questions
Follow a Batch JSONL example from candidate training data to a held-out evaluation of two pinned LLM checkpoints. Reproduce the score and coverage check offline.
Create one evaluation file
Start with the Batch JSONL mini-lot for a support ticket classifier. It asks a pinned public model to label requests as billing or access. Join the delivered completions and errors by custom_id to retain one prompt and response as a candidate for supervised fine-tuning.
For evaluation, keep four different tickets out of the training data and label each one independently. Use a new Batch-format JSONL file with unique custom_id values and a top-level reference string on each line. Compare against these labels instead of deriving the test answers from Batch completions.
checkpoint-eval.jsonl · first of four held-out tickets
{"custom_id":"eval-billing-01","method":"POST","url":"/v1/chat/completions","body":{"model":"Qwen/Qwen3-8B","messages":[{"role":"user","content":"Classify this support request as billing or access. Reply with the category only. Request: When will my refund appear on my card?"}],"max_tokens":64,"temperature":0.6,"top_p":0.95,"top_k":20,"seed":7,"chat_template_kwargs":{"enable_thinking":false}},"reference":"billing"}Prepare and inspect the Batch delivery — See the input, completions, error and candidate row that start this example.
Download the Batch and evaluation example — ZIP with the Batch mini-lot, a held-out JSONL file, synthetic grades and errors, and local scripts.
Run the same questions against both revisions
On the OpenAI-compatible API, upload checkpoint-eval.jsonl with POST /api/openai/v1/files and purpose=batch. Create each evaluation with POST /api/openai/v1/batches and poll GET /api/openai/v1/batches/{id}. The file evaluation walkthrough has the request and download commands.
- Upload checkpoint-eval.jsonl once. Use the same input_file_id in two Batch creation requests with distinct idempotency keys. Set metadata.revision to the full 40-character commit for each supported revision of the same model repository.
- Set metadata.reliquary_grader to exact_match_v1 for both orders. Keep the prompts, references, output-token limits, sampling settings and one sample per question fixed. Record the batch ID and pinned revision for each order.
- Batch inference and file evaluation require approved dataset access and an orders:write key. Review the price and access requirements before placing an order. The offline archive uses example revisions and synthetic results.
File evaluation walkthrough — See how to upload the file, create an evaluation and download its report.
Report pass rates alongside coverage
Read GET /api/v1/dataset-orders/{id}, using the batch ID without batch_. Check order.evaluation.complete and download report.json, graded.parquet and manifest.json from its evaluation delivery. Keep the Batch output and error files when present. Record input validation errors against the original holdout and review delivery errors separately.
In this illustrative delivery, A has four grades and B has three. B reports missing_samples for eval-access-02. The offline grade files are JSONL projections of the relevant graded.parquet columns.
| Measure | Checkpoint A | Checkpoint B |
|---|---|---|
| Questions requested | 4 | 4 |
| Questions graded | 4 | 3 |
| Grades missing | 0 | 1 |
| Passes | 2 | 3 |
| Pass rate among graded | 2/4 = 50% | 3/3 = 100% |
| Grading coverage | 4/4 = 100% | 3/4 = 75% |
| Pass rate with missing grades counted as failures | 2/4 = 50% | 3/4 = 75% |
Compare the questions both checkpoints answered
Pair graded rows on problem_id and sample_index. The simple identifiers in this file are preserved in problem_id. When imports use line-based IDs, reconcile the rows to the original file before pairing. Use only rows with a grade on both checkpoints.
On the three common questions, A passes 1/3 and B passes 3/3. This paired result and the full-set rates above use different denominators. Record the denominator and missing-grade rule alongside the score, and read the errors before interpreting missing grades.
Read file evaluation results — Find the grading summary and delivered report fields.
Reproduce the decision before RL
Extract the example ZIP and run these commands from its directory. The scripts use Python’s standard library and work offline with synthetic files.
For this four-question illustration, the comparison advances only if both runs grade all four questions and B improves on the paired set. B improves on the three shared questions but grades only three of four, so resolve the delivery gap and repeat the comparison. Use a larger representative holdout and set your coverage and quality criteria before a real evaluation.
This workflow prepares evidence for a training decision. Customer reinforcement learning is planned and execution access is separate.
Run the offline example
python3 join.py
python3 compare.pyPrepare tasks and rewards for reinforcement learning — Plan a checkpoint, a verifiable grader and a representative holdout.