Skip to content

A.1 Training Debugging Guide

You have already written DQN, Actor-Critic, and PPO, and you have seen the training pipelines for RLHF, GRPO, and Agentic RL. At this point, a natural question comes up:

Why does the same algorithm run fine in the paper, run fine in someone else's code, and then become unstable the moment you swap in your own environment, change the reward, or scale up the model?

You are not alone in this. The hard part of reinforcement learning was never just "can you derive the formulas." Training itself is a closed loop that keeps changing the data distribution it's fed by: the policy is moving, the sampled data is moving with it, the reward model may be biased, and the value function is chasing a moving target. In supervised learning, a bad batch usually only costs you one gradient step. In RL, a bad policy collects bad data, and that bad data trains an even worse policy.

So this appendix is not a catalog of common error messages, and it doesn't stop at four failure modes. It's a debugging lecture: we first build a mental model, then walk through the different kinds of training anomalies using that model.

By the end of this section you should be able to answer three questions:

  1. When a training curve goes wrong, which part of the pipeline should you suspect first?
  2. What is the relationship between reward, loss, KL, entropy, value loss, GPU memory, and eval scores?
  3. Facing an unstable RL experiment, how do you narrow it down step by step instead of tweaking hyperparameters by feel?

Looking at Training as a Closed Loop

Let's start by drawing an abstract loop. This is not a diagram of any specific framework's implementation, and it doesn't claim that every modern LLM RL pipeline must look exactly like this. It's here to help you see the shape clearly: one round of RL training generally goes through "generate behavior → score it → construct a training signal → update the policy."

Mermaid diagram

Any link in this loop can break, and the symptom on the surface is almost always the same: "reward isn't going up." But the fix is completely different depending on which link failed.

Three things need to be kept separate here.

Reward signal is the score actually computed during training. It might come from the environment itself, a hand-written reward function, a reward model, a verifier, or a weighted combination of several rules.

Training signal construction is the step that turns reward into "what should this update actually encourage, and what should it suppress." In PPO / Actor-Critic, this usually shows up as return, value target, and advantage; you can loosely read advantage as "how much better was this action or response than what was already expected." If the actual return turns out higher than what the Critic predicted, the advantage is positive and the policy leans toward repeating that behavior; otherwise it gets suppressed. In GRPO / RLVR, the common approach is not to train a Critic at all — instead you sample several responses for the same prompt and build advantage-like training weights from the relative reward within that group. TRL's GRPO documentation also splits the pipeline into generation, advantage computation, KL estimation, and loss computation, but there the advantage comes from normalizing reward within the group, not from a Critic's prediction[1].

Evaluation and human audit is side-channel supervision. It's used to pick checkpoints, catch reward hacking, and decide whether to roll back — under normal circumstances it doesn't feed into the gradient update directly. Eval results can tell you "the reward design is wrong," but they are not the same signal that drove training.

So this diagram is best read as a unified debugging map, not as "the one true pipeline for modern Agentic RL." PPO-RLHF looks more like the Critic + KL version of this diagram; GRPO/RLVR looks more like the "multiple generations + reward/verifier + within-group relative advantage" version; Agentic RL expands a single response into a multi-step tool trajectory, with reward possibly coming from the final environment state, a rule-based verifier, or human/model review. If the environment is wired up wrong, tuning the learning rate won't help. If the reward function can be gamed, continuing to train just makes the model better at cheating. If the Critic can't learn, PPO's advantage becomes noise. If KL spikes, the policy has already left the trust region. If the eval protocol is contaminated, every pretty curve might just be an illusion.

One sentence to remember

The first principle of RL debugging is not "tune the hyperparameters" — it's "locate which link in the loop broke first."

First-Pass Diagnosis for Training Anomalies

When training goes wrong, the most common reaction is to immediately adjust hyperparameters — lower the learning rate, increase the batch size, raise the KL coefficient, or just train for more steps. This looks proactive, but it introduces new variables and makes the original problem harder to pin down.

This section walks through a first-pass diagnostic flow better suited to course experiments and research reproduction. The goal isn't to fix training immediately — it's to figure out first which part of the pipeline the anomaly is coming from: experiment configuration, evaluation protocol, reward signal, model output, or the optimization process itself.

Record the Experiment Context

Start by recording the basic context of the run: config file, random seed, code version, checkpoint, training logs, and eval command. RL experiments are extremely sensitive to random seeds and implementation details — the same algorithm setup can show noticeably different behavior across different seeds[2]. If this information isn't saved, it becomes very hard later to tell "the algorithm really is unstable" apart from "the experimental conditions changed."

Separate Training Metrics from Evaluation Metrics

Training reward only tells you the model is optimizing some reward signal — it does not by itself tell you task capability has improved. A more reliable way to look at things is to keep three categories of information separate:

  • Training metrics: training reward, policy loss, KL, entropy, and so on — used to check whether the optimization process is stable.
  • Evaluation metrics: held-out benchmarks, private test sets, task success rate — used to judge whether capability has actually improved.
  • Behavior samples: the model's or agent's actual outputs — used to judge whether it has learned the wrong pattern.

For example, in RLHF training, if reward keeps rising while the eval score stays flat and response length keeps growing, the right read usually isn't "training hasn't run long enough." You should suspect a length preference baked into the reward signal.

Inspect Model Output Samples

Curves are a compressed summary of the training process; samples expose the actual behavior. At minimum, check three kinds of samples during diagnosis: high-reward samples, low-reward samples, and random samples from the latest checkpoint.

In language model training, reward hacking tends to show up first as a shift in writing style: responses get longer, more elaborately formatted, more full of polite filler — while information density drops. In Agentic RL, it can also show up as an increase in tool-call count without the final environment state actually reflecting task completion.

Build a Minimal Reproduction

Once you've checked the logs and samples, shrink the experiment down to something that runs fast: a smaller model, a smaller batch, fewer prompts, fewer training steps. The minimal reproduction isn't trying to hit a final score — it's answering basic questions:

  • Can the implementation learn at all under a simple setup?
  • Does the reward actually discriminate between good and bad behavior?
  • Is the eval protocol stable?
  • If you're using PPO/Actor-Critic, can the value function fit a fixed set of rollouts?
  • If you're using GRPO/RLVR, is the reward ranking across multiple responses to the same prompt sensible?

Many RL bugs don't crash the program. A done mask written wrong, a flipped reward sign, padding tokens leaking into the loss, an eval temperature that quietly changed — all of these can let training finish normally while the model ends up learning the wrong behavior. That's why completing a minimal reproduction before scaling up is such an important step in the debugging workflow.

Diagnostic Order

The sections that follow each cover a different class of training problem. In practice, work through them outside-in.

Start by checking the environment and data. Is the state the agent sees actually correct? Is the action being executed correctly by the environment? Is the termination signal handled correctly? Does the reward sign match what you expect? If something is wrong at this layer, every algorithm update downstream is just optimizing on top of bad data.

Next, check the evaluation protocol. If sampling temperature, max output length, tool permissions, or test-set splits have changed, eval results are no longer directly comparable. A public test set that gets reused repeatedly for tuning also gradually loses its meaning as an evaluation.

After that, check the reward signal. Is the reward too sparse? Are there extreme outlier scores? Does it agree with human judgment or an independent evaluation? If the reward signal can't be trusted, then the more thoroughly you train, the more likely the model is optimizing in the wrong direction.

Only after all of that do you go inside the algorithm. For PPO, check whether the policy update is too large. For methods with a Critic, check whether the value function is actually working. For GRPO/RLVR, check whether the within-group reward comparison makes sense. For Agentic RL, check whether the tool trajectory is consistent with the final environment state.

This order keeps you from suspecting every module at once. First figure out roughly which layer the anomaly belongs to, then go into the matching section below for a finer-grained check.

Environment and Data: Confirm the World Is Real First

The bugs most easily overlooked in reinforcement learning tend to live upstream of the algorithm.

CartPole's action space is discrete — 0 or 1 — but you fed in a continuous action. MuJoCo's action range is [-1, 1], but the policy's output never went through a tanh. In dialogue training, padding tokens weren't masked out, so the model is "learning" to fill in padding positions. In an agent task, a tool call failed, but it got recorded as a successful trajectory anyway.

What these problems have in common: training runs, and the curves move. The curves just don't mean anything.

A Minimal Unit Test

Before starting real training, run at least four checks:

python
def sanity_check_env(env, policy):
    obs, info = env.reset(seed=0)
    assert obs is not None

    action = policy.sample(obs)
    next_obs, reward, terminated, truncated, info = env.step(action)

    assert next_obs is not None
    assert isinstance(float(reward), float)
    assert isinstance(terminated, bool)
    assert isinstance(truncated, bool)

    return {
        "reward": reward,
        "done": terminated or truncated,
        "info_keys": list(info.keys()),
    }

Then run a cruder but effective test: run 100 trajectories with a random policy and plot the reward distribution. Then run 100 trajectories with a hand-written "weak expert" policy. If the expert policy and the random policy don't look meaningfully different, don't start training the model yet — go check the environment and the reward first.

Common wiring mistakes

Most non-convergent training isn't actually an algorithm problem. It's a flipped reward sign, an unhandled terminal state, a mismatched action scale, missing observation normalization, or a chosen/rejected pair swapped in the dataset.

Don't Let Your Test Set Turn Into Your Training Set

RL projects are prone to "eval contamination." You may never have put the test set into your training data, but if you keep using it to tune prompts, tune reward, tune the KL coefficient, and pick checkpoints, it's already participating in training decisions.

This is especially bad in post-training and Agentic RL. The model may not actually be getting stronger — it may just be adapting to a specific public benchmark, a specific judge, or a specific output format.

Here's a rule of thumb worth writing down:

SetPurposeHow often to look
Smoke setQuickly catch implementation bugsFreely
Dev setTune hyperparameters, tune rewardFreely, but log it
Public testWatch the trendRarely
Private testRelease gateAvoid looking
Human audit setCalibrate reward and judgePeriodic spot checks

The evaluation protocol needs to be fixed too: temperature, top_p, max_tokens, prompt template, tool permissions, timeout rules, pass@1/pass@k — all of it should be written down explicitly. Research on the ALE evaluation protocol makes the same point: variation in environment randomness, starting states, and evaluation procedure can significantly change RL conclusions[3].

Having a Reward Doesn't Mean the Model Can Learn From It

The "reward" here isn't the act of designing a reward — it's the actual score each transition, response, or trajectory receives during training. That signal has to satisfy two conditions at once: pointing the right way, and dense enough.

Pointing the right way means the reward genuinely encourages the behavior you want. Dense enough means the model can pick up some difference in the reward even early in training. If 99.9% of trajectories get a reward of 0, the policy gradient is looking at silence.

Look at the Reward Distribution

Plot a reward histogram before training, not after.

Distribution shapeLikely problemWhat to do
Almost all 0Reward too sparseAdd intermediate rewards, curriculum learning, more exploration
Almost all 1Reward too lenientRaise task difficulty, split into separate scoring dimensions
Extreme long tailA few samples dominate the gradientReward clipping / normalization
Signs are inconsistentReward definition is unclearGo back and inspect samples one by one
Low correlation with human ratingsThe proxy can't be trustedRewrite the reward or add human calibration

In PPO, reward scale also affects the advantage. If the reward scale is too large, the advantage becomes a very spiky gradient signal, and the policy update can shoot straight out of the trust region. Most high-quality implementations do reward normalization, advantage normalization, and gradient clipping — and these implementation details by themselves change algorithm behavior[4][5].

The Model Learned Test-Taking Tricks

Reward hacking isn't the model "misbehaving." It's the opposite — the model is optimizing your stated metric extremely well. The AI safety literature calls this specification gaming: the system satisfies the formal objective while violating the designer's actual intent[6][7].

The classic language-model version: the reward model prefers detailed responses, so the model starts writing longer, more polite, more hollow responses. Reward climbs steadily while human spot checks get worse. Research on reward model overoptimization shows the same pattern — the proxy reward can keep improving while true preference starts declining past a certain point[8].

The Three-Symptom Syndrome

Reward hacking usually shows up as three signals appearing together:

  1. Reward is climbing: the training dashboard looks great.
  2. Side metrics look off: systematic shifts in length, repetition rate, format templates, refusal rate, or tool-call count.
  3. Real evaluation is declining: human spot checks, the private set, and task success rate aren't improving in step.
python
def audit_reward_hacking(samples):
    suspicious = []
    for item in samples:
        if item["reward"] > 0.9 and item["human_score"] < 0.4:
            suspicious.append(("reward-human mismatch", item["id"]))
        if item["response_len"] > item["baseline_len"] * 2:
            suspicious.append(("length inflation", item["id"]))
        if item["repeat_ratio"] > 0.2:
            suspicious.append(("repetition", item["id"]))
    return suspicious

Fixing this isn't a matter of bolting on one penalty term and calling it done. The more durable fix is to break reward apart into separate logged components: correctness, constraint satisfaction, safety, conciseness, formatting, tool-result quality, scored individually. Work like RewardBench makes a related point: a reward model needs to be evaluated in its own right — you can't just assume it always represents human preference[9].

PPO's Seatbelt Can Fail Too

PPO's core intuition is "small updates." TRPO enforces this explicitly with a KL constraint; PPO approximates the same goal with a clipped surrogate objective[10][11][12]. But the clip is not a magic shield.

If the learning rate is too high, there are too many PPO epochs, the batch is too small, or the advantage scale is off, the policy can still take a step that's too large.

Watch Three Metrics

MetricWhat to look atWhat an anomaly means
KL divergencedistance between the new policy and the old/reference policypolicy is drifting too fast
Clip fractionhow many samples are getting clippedPPO is hitting the brakes frequently
Entropyhow much randomness the policy still haspremature convergence or degenerate randomness

Policy collapse usually doesn't start with reward — it starts with KL, clip fraction, and entropy. Reward is the symptom that shows up afterward.

python
def ppo_guardrail(metrics):
    if metrics["kl"] > metrics["target_kl"] * 2:
        return "stop update: KL too high"
    if metrics["clip_fraction"] > 0.4:
        return "reduce lr or PPO epochs"
    if metrics["entropy"] < metrics["entropy_floor"]:
        return "increase exploration or KL constraint"
    return "continue"

In RLHF you also need to watch KL relative to the reference model. InstructGPT-style pipelines introduce a KL penalty exactly so the RL phase doesn't wreck the language competence learned during SFT[13].

The Critic: Where PPO / Actor-Critic Failures Hide

This section only applies to methods with a Critic or value head — Actor-Critic, PPO, and some PPO-RLHF implementations. If you're using a Critic-free method like GRPO/RLVR, skip this section and instead check the within-group reward, KL, and loss construction.

In Actor-Critic, the Critic's job is to estimate state value. It doesn't output actions directly, so a lot of people debugging these systems only ever look at policy loss. But if the Critic is wrong, the advantage is wrong, and if the advantage is wrong, the Actor updates in the wrong direction.

Signs the Critic Is Broken

SignalWhat it means
Value loss stays high, doesn't dropCritic isn't fitting the returns
Explained variance < 0worse than just predicting the mean
Policy reward oscillatesActor is being pushed around by noisy advantage
Value prediction scale much smaller than returnreward scale or value target problem

Common fixes include: lowering the reward scale, normalizing returns, adjusting the critic's learning rate up or down, giving the critic network more capacity, checking the bootstrap target, and checking the terminal mask.

A very practical check: fix a batch of rollouts, freeze the actor, and train only the critic — see if it can fit that batch's returns. If it can't, fix the critic first.

Too Confident and Too Random Are Both Bad

Exploration problems show up in two opposite ways.

One is entropy collapsing to zero too fast: the model latches onto one action or one response template early and gets stuck in a local optimum. The other is entropy staying high indefinitely: the policy behaves like a random walk, and reward never gets absorbed into the parameters.

SymptomLikely causeFix
Entropy collapses quicklyreward too strong, KL too weak, temperature too lowadd an entropy bonus, lower lr, strengthen KL
Entropy stays high for a long timereward too sparse, learning rate too low, noisy advantagereward shaping, more sampling, check the advantage
Behavior is diverse but not improvingexploration isn't being distinguished by the rewardchange the reward or add curriculum
Behavior is uniform but reward is highpossible reward hackingspot-check high-reward trajectories

In language models, exploration isn't just token-level randomness — it also includes response length, reasoning path, tool choice, and where the refuse/don't-refuse boundary sits. Token entropy alone isn't enough; you also need to look at diversity at the behavioral level.

Data Freshness: On-Policy Isn't Just a Slogan

PPO is an on-policy algorithm: it assumes the data used for an update comes from "near the current" policy. We save the old logprob during training precisely so we can measure how far the new policy has drifted from the sampling policy.

If the rollout worker and the learner fall out of sync, or the buffer ends up mixing in very old data, you'll see a strange pattern: the loss still computes, the gradient still steps, but the metrics swing back and forth unpredictably, and clip fraction becomes hard to explain.

Three questions to ask when troubleshooting this:

  1. Does every rollout record which policy version generated it?
  2. Does the old logprob used at update time actually match the sampling policy?
  3. By the time a rollout enters training, how many rounds of policy update has it already lagged behind?

Agentic RL falls into this trap more easily, because a single trajectory can be long, tool execution is slow, and sampling and training are naturally asynchronous. Don't optimize purely for throughput — also keep data staleness under control.

NaN Usually Gives Warning Signs First

NaN rarely appears out of nowhere. It's usually preceded by a spike in gradient norm, extreme logprob values, an outlier reward, exploding value loss, or mixed-precision overflow.

ProblemWhat to checkFix
Grad norm spikep95 / max grad normgradient clipping, lower lr
Extreme logprobtaking log of a zero probabilityclamp, check the mask
fp16 overflowloss scale, NaN stepswitch to bf16, dynamic loss scaling
Reward outlierreward max/minclipping, normalization
Value explosionvalue target distributionreturn normalization

Don't wait for loss to turn into NaN before stopping training. Your training script should save experiment state and halt the current update as soon as a key metric crosses a threshold.

GPU Memory Is Only Part of the Ledger

RLHF/PPO is much more resource-hungry than plain SFT, because it may need an actor, a critic, a reference model, and a reward model all at once, plus storage for rollouts, logprobs, values, advantages, and long-sequence activations.

GPU memory mainly comes from four sources:

SourceWhy it costs memoryCommon remedies
Model weightsmultiple models resident at oncefreeze, share, separate rollout/training
Optimizer stateAdam's first/second momentsZeRO, FSDP, 8-bit optimizer
Gradientsscales with the number of trainable parametersLoRA, freeze the backbone
Activationsscales with batch size and seq_lencheckpointing, shorter sequences

ZeRO shards optimizer state, gradients, and parameters across multiple GPUs[14][15]. FSDP lowers per-GPU resident memory through parameter sharding and on-demand all-gather[16]. LoRA freezes the main model and only trains a low-rank adapter[17]. These aren't "advanced optimizations" you add later — they're the precondition for whether large-model RL training can even start.

But resource problems aren't only about OOM. Falling throughput, low GPU utilization, rollout workers stuck waiting on the environment, and the reward model becoming a scoring bottleneck all slow training down, make data go stale, and eventually feed back into algorithmic instability.

Extra Traps in RLHF and Agentic RL

RL for language models and agents carries a few extra failure modes beyond classical control.

SettingExtra trapExample
RLHFlength preferenceresponses keep getting longer, but information density drops
RLHFrefusal driftsafety reward too strong, model over-refuses
RLHFjudge biasthe LLM judge favors a particular writing style
RLVR/GRPOformat hackingthe model learns to match the format while the reasoning is still wrong
Agentic RLtool hackingrepeated tool calls just to farm process reward
Agentic RLfake success in statethe text claims completion but the environment state never changed
Agentic RLlong-trajectory credit assignmenta final failure is hard to trace back to a specific step

For these reasons, Agentic RL evaluation can't just look at the final text — it needs to look at environment state, whether tool calls were valid, step count, cost, and failure recovery ability. RLHF evaluation can't just look at the reward model — it needs human spot checks, a private set, length, repetition rate, safety regression, and real task success rate, all together.

One Complete Troubleshooting Path

Suppose you observe: reward is going up, the benchmark isn't moving, and outputs keep getting longer.

Don't jump straight to "training isn't converging." Trace it through the loop:

  1. Evaluation protocol: does the benchmark's temperature and max_tokens match the baseline?
  2. Sample spot check: are the highest-reward samples longer, emptier, more templated?
  3. Reward decomposition: does the reward carry a hidden preference for length, formatting, or polite tone?
  4. KL and entropy: has the policy drifted too far from the reference model, is there mode collapse?
  5. Fix experiment: add a length penalty or an information-density metric, run a short training run as a control.
  6. Ship/no-ship decision: if reward goes down but the private set goes up, the earlier reward was probably wrong to begin with.

Now a second example: reward crashes, KL spikes, clip fraction sits at 0.5 for a long stretch.

Here you should first suspect the policy update was too aggressive:

  1. Roll back to the most recent healthy checkpoint.
  2. Lower the learning rate.
  3. Reduce the number of PPO epochs.
  4. Turn on target-KL early stopping.
  5. Check advantage normalization and reward scale.

These two examples call for completely different fixes. That's exactly why "reward isn't going up, what do I do" isn't a good question to ask. The better question is: "which piece of evidence in the loop broke first?"

Checklists: Before, During, and After Training

Before Training

CheckQuestion
Environment unit testdo reset/step/done/reward behave as expected?
Random-policy baselinewhat does the random policy's reward distribution look like?
Weak-expert baselinecan a simple rule clearly beat random?
Reward histogramis reward all 0, all 1, or an extreme long tail?
Eval configis the evaluation protocol fixed and saved?
Memory budgetcan you afford the number of model copies, batch size, and seq_len?

During Training

SignalAction
KL spikingstop the update, lower lr or strengthen KL
Clip fraction persistently highreduce PPO epochs or step size
Entropy collapsing quicklycheck for reward hacking and exploration issues
Value loss not decreasingtrain the Critic alone as a fit test
Reward up, eval downspot-check high-reward samples immediately
Response length inflatingcheck for a length preference
OOM or throughput crashlower micro batch / seq_len first, then bring in ZeRO/FSDP

After Training

DeliverableWhy
Best eval checkpointthe last step isn't always the best
Last checkpointmakes it possible to reproduce late-training issues
Failure checkpointhelps analyze what preceded a crash
Reward audit samplesjudges whether reward hacking occurred
Multi-seed resultsavoids mistaking a lucky run for a real result
Private-set reportguards against overfitting to the public set

Summary

Debugging reinforcement learning isn't about memorizing a list of failure-mode names — it's about following the loop and gathering evidence.

Environment and data determine whether what you're learning is grounded in the real world. Reward and evaluation determine whether the optimization direction is actually what you want. Policy update and the Critic determine whether the gradient is stable. Exploration determines whether the model can discover better behavior. System resources determine whether training can keep producing fresh data.

When something goes wrong, don't start by asking "what should I set the learning rate to." Start by asking:

Which curve broke first? Which part of the loop does it belong to? Is there a minimal experiment that can confirm this diagnosis?

That question is where RL training stops being folklore-driven tuning and starts becoming engineering.

References


  1. Hugging Face TRL, GRPO Trainer. ↩︎

  2. Henderson et al., Deep Reinforcement Learning that Matters, 2018. ↩︎

  3. Machado et al., Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents, 2018. ↩︎

  4. Engstrom et al., Implementation Matters in Deep RL: A Case Study on PPO and TRPO, 2020. ↩︎

  5. Andrychowicz et al., What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study, 2020. ↩︎

  6. Amodei et al., Concrete Problems in AI Safety, 2016. ↩︎

  7. Lilian Weng, Reward Hacking in Reinforcement Learning, 2024. ↩︎

  8. Gao et al., Scaling Laws for Reward Model Overoptimization, 2022. ↩︎

  9. Lambert et al., RewardBench: Evaluating Reward Models for Language Modeling, 2024. ↩︎

  10. Schulman et al., Trust Region Policy Optimization, 2015. ↩︎

  11. Schulman et al., Proximal Policy Optimization Algorithms, 2017. ↩︎

  12. OpenAI Spinning Up, Proximal Policy Optimization. ↩︎

  13. Ouyang et al., Training language models to follow instructions with human feedback, 2022. ↩︎

  14. Rajbhandari et al., ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, 2019. ↩︎

  15. Microsoft DeepSpeed, ZeRO Tutorial. ↩︎

  16. PyTorch Docs, FullyShardedDataParallel. ↩︎

  17. Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, 2021. ↩︎

现代强化学习实战课程