PAYERBENCH ENVIRONMENT · TRAINING & EVALUATION METRICS
PayerBench training results.
De-identified outpatient claims, nine evidence tools, and a deterministic remittance-based verifier. Metrics reflect the current evidence, including measurements still to be established.
| Frontier model baseline | measured59.7% pass@1 750 claims; equal paid, partial, and denied weight; 95 payers. |
|---|---|
| Held-out capability gain after RL | measured+0.8 pp overall; +7.6 pp on denied claims Seed 2 − SFT. Overall CI [−1.2, 2.8]; denied CI [4.4, 11.2]. Seed 1 overall: −4.5 pp. |
| Unseen task-family gain | unmeasuredNot measured All six controlled families were represented in training; no held-out-family experiment. |
| Production-derived holdout gain | measured+0.8 pp incremental RL; +7.5 pp for SFT + RL Same holdout as above. SFT + RL vs base CI [4.7, 10.1]; no independent replication. |
| Sim2Real transfer efficiency | not applicableNot applicable Production-derived training and evaluation; no separate simulator-to-production stage. |
| Human / reward agreement | unmeasuredNot measured No independent expert agreement study. Labels are observed payer responses. |
| Reward false-positive rate | unmeasuredNot established No independently adjudicated reward-correctness set. |
| Reward exploit rate after red-teaming | scoped0 / 2,247 fixed attacks rewarded (0%) Three fixed mutation classes × 749 accepted references; no open-ended adversarial search. |
| Effective environment scale | scoped18,445 training task IDs / 6 controlled families 12,618 observed + 5,827 constructed tasks. Distinct IDs; semantic redundancy unaudited. |
| Saturation point | scopedNot established; 7,752 rollouts / seed observed Ten updates per seed; dev peaks at step 15, then falls at 20. No saturation study. |
| Throughput | scoped2.83 rollout attempts/sec 8B / H100: 7,752 attempts in 2,738 rollout seconds; 0.42/sec with updates. Excludes validation/setup. |
| Environment reliability | scoped99.6% valid evaluated completions (1,550 / 1,556) Final seed 2 results after retries; prediction + process + integrity pass. Not uptime. |
| Cost per 1M valid episodes | scopedNot measured; ~$93 total experiment spend Reported total: this version’s SFT, RL, control, and evaluation (29 GPU-hours). Unit cost unmeasured. |
| Learning efficiency per $10K training | scopedNot established; +0.8 pp at the selected RL checkpoint Seed 2: five updates, 3,876 rollouts, 2.72 GPU-hours excluding SFT/validation/setup. No dollar extrapolation. |
| Synthetic substitution gap | unmeasuredNot measured No matched training comparison against a strongest cheap synthetic replica. |
Scope: Pass@1 requires the full claim, line, and reason-code prediction. Observed-claim test: 750 claims, 250 per outcome, 95 payers; 558 allergy/immunology, 165 dermatology, 27 orthopedics. The full corpus is 24,302 claims, 486 payers, five specialties.
Audit: Three fixed mutation classes × 749 accepted canonical references. One additional reference has a source-label inconsistency. Zero accepted mutations does not establish zero general exploit risk. Throughput and completion are reference-policy measurements, not service guarantees.
Evidence: Saved model predictions rescored without new inference; saved trainer timing logs; paper-reported compute estimate; targeted verifier audit. Measured = direct result; scoped = bounded evidence; unmeasured = no supporting experiment; not applicable = outside the evaluation design.