PayerBench Training

01Summary

Supervised fine-tuning raises Qwen3 8B’s class-balanced full-prediction score from 48.7% to 55.3%: +6.7 percentage points, with a paired 95% interval of [4.4, 9.1]. Denied-claim performance rises from 10.8% to 27.6%; paid-claim performance stays at 94.8%.

RL seed 2 reaches 56.1% overall, +0.8 points over SFT [−1.2, 2.8]. Its denied-claim score increases to 35.2%, while its paid-claim score falls to 86.0%. RL seed 1 reaches 50.8% overall, 38.4% on denied claims, and 66.0% on paid claims. Both runs start independently from the same SFT checkpoint.

All headline results use 750 held-out claims, balanced across paid, partial, and denied outcomes. Passing requires the correct claim class, graded service-line outcomes, and reason-code numbers.

02What training changed

All results below use the same 750 held-out claims and the prediction-integrity scorer revision of October 22, 2026. Saved predictions were replayed through the revised grader; no new model inference was used. The benchmark score requires the complete claim, service-line, and reason-code prediction.

All completed runs, ranked by overall score

16 RUNS · SAME 750 CLAIMS

FIGURE 1. All 16 completed comparable runs, ranked by class-balanced full-prediction pass@1, with 95% class-stratified bootstrap intervals from 4,000 resamples. Gold marks the highest observed score. SFT, both RL seeds, and the random-reward control carry an explicit “(Trained)” label. The checklist prompt variant and statistical baseline are separately identified.

The comparison includes every completed model configuration in this evaluation and the class-balanced tree baseline. The Opus checklist attempt was stopped amid API errors; earlier training experiments used different evaluation setups. Those runs are excluded from this comparison.

Swipe charts horizontally to read the full figures.

All runs by actual claim outcome

250 CLAIMS PER OUTCOME

FIGURE 2. Full-prediction pass within each actual outcome class, using the same overall ranking as Figure 1. Gold raised cells mark the highest observed score in each column, including ties; other cells use the green 0–100% scale. The tree leads on paid claims, GPT 6.1 on partial claims, and RL seed 1 on denied claims. Highlights indicate point estimates, not statistically established superiority.

Catching more denials has a visible cost

ALL 16 OPERATING POINTS

FIGURE 3. Higher and left is preferable. Numbers identify the same runs in the adjacent key, with exact false-alarm and denied-pass rates. DeepSeek and Qwen3 235B share an operating point. The vertical metric is complete denied-claim prediction; the horizontal metric counts actually paid claims classified as partial or denied. Each point is one evaluated policy, not a threshold sweep or a matched-false-alarm-rate comparison.

RL’s denied-claim point estimates exceed GPT 6.1: 38.4% and 35.2%, versus 34.0%. The paired differences are +4.4 points [−0.4, 8.8] and +1.2 [−3.2, 5.2], so these results do not establish superiority. Paid false alarms rise to 34.0% and 14.0%, versus GPT 6.1’s 6.4%.

Which gains survive a paired comparison?

DIFFERENCE · 95% INTERVALS

FIGURE 4. Paired bootstrap differences retain which claims each policy got right. Point differences are computed before rounding; they may differ by 0.1 from subtracting the displayed headline scores. Overall and denied intervals are resampled within their respective evaluation populations.

What this evidence supports

• Learnability through supervised training. The held-out SFT gain is clear for this model and recipe, particularly on denied claims.
• Denial gains and paid-claim false alarms. Both RL seeds improve denied-claim full pass over SFT while increasing false alarms on paid claims.
• RL remains an open optimization problem. Two short runs are insufficient to establish robust gains across seeds or larger models. The random-reward control scores 54.7%, close to SFT’s 55.3%.
• Scope is specific. One platform, a three-specialty evaluation dominated by allergy / immunology, one run per policy, and no measured human-expert baseline. Missing evidence can limit how well the payer response is predictable.

The shown RL policies were trained before the later prediction-integrity rescore. Rescoring measures their saved behavior under the tighter verifier; it does not mean they were retrained with that revision.

03Evaluation metrics

The primary comparison is RL seed 2 against the same supervised checkpoint, paired on the same 750 observed claims. Each result includes its evaluation scope and any measurement limits.

The fixed attack suite was run on October 22 against the current verifier, without model inference. One of 750 canonical reference answers fails because of the previously identified claim/line label inconsistency; the 2,247 mutations use the other 749 references. This audit does not establish human agreement or open-ended adversarial robustness.

04A dataset with the workflow attached

The corpus contains 24,302 de-identified outpatient claims and 47,613 service lines from Arrow’s revenue-cycle platform, spanning five specialties and 486 payers. The records preserve claim structure and actual payer outcomes, together with the evidence exposed by the environment.

The later period begins June 1, 2026, after all training and development service dates. Training and development are separated by patient; test patients may have earlier training claims. The 750-claim evaluation is balanced: 250 paid, 250 partial, and 250 denied.

Corpus specialties

Allergy / immunology, dermatology, orthopedics, primary care, and urgent care

Evaluated population

558 allergy / immunology, 165 dermatology, 27 orthopedics · 95 payers

Controlled rule tasks

5,827 constructed training tasks · 806 test tasks, reported separately

Six controlled families

Authorization, duplicates, diagnosis coverage, provider identifiers, timely filing, and visit limits

De-identification. Names and direct identifiers are removed or replaced; dates and demographic details are transformed while preserving the relationships needed for the task. The pipeline uses the HIPAA Safe Harbor identifier list as its basis, with additional checks for rare combinations.

Corpus counts describe the complete research snapshot. Training runs and the held-out evaluation use the subsets specified above.

05Evidence in. A verifiable prediction out.

An episode starts with a submitted claim. The agent chooses which evidence to inspect, then calls predict_adjudication once. The payer’s response is withheld from the evidence tools and used as the grading target.

{
  "claim": "fictional-example",
  "serviceLines": ["office_visit", "allergy_injection"],
  "payerOutcome": "withheld"
}

Nine tools, a 12-call budget

The budget includes the terminal prediction. Payer rules and similar claims are derived from the training split and exclude the current patient.

The benchmark’s full-pass metric does not score paid dollar amounts. The target is the observed insurer response; it is not an expert determination that the payer’s decision was correct.

06From supervised evidence use to online RL

We tested the environment with Qwen3 8B in non-thinking mode, using rank-16 LoRA adapters. The same tool interface serves training and evaluation. The supervised stage establishes an evidence-reading policy; the RL stage samples alternative trajectories and updates from verifier feedback.

Supervised learning: teach an evidence procedure

Training examples open the evidence sources, build a checklist from returned facts, and submit the prediction. Paid claims remain in the pool, including cases where a warning cue does not lead to a denial. Partial and denied training examples are retained only when the visible evidence supports every adverse line’s reason. This avoids teaching the model to invent an explanation for an unsupported label.

The supervised recipe uses a learning rate of 10⁻⁴ across 16 chunks, with checkpoint selection on dev150. The selected checkpoint is chunk 10. Test labels are not used for training or checkpoint selection.

Reinforcement learning: optimize sampled trajectories

The reference trainer uses ART with GRPO-style group-relative updates. Dynamic sampling seeks groups whose attempts receive different rewards, with 16 attempts for adverse claims and 8 for other tasks. Updates reweight outcome classes, with a target 15% share for constructed tasks. Reward standard-deviation scaling is disabled so the reward magnitudes retain their effect.

RL learning rate: 5 × 10⁻⁶. Each run has a six-hour cap and a maximum schedule of 60 additional steps; only about ten updates fit per RL seed. Checkpoints are evaluated on dev1000 every five steps. Both seeds select step 15, starting from SFT step 10. The control replaces the shaped reward with a seeded random binary reward.