C.1 Paper Reading Roadmap
After finishing the first 17 chapters of this book, you already have all the math and engineering foundations needed to read frontier RL papers. But the RL literature is vast — arXiv adds thousands of papers a year, and fewer than a hundred actually shape the field and reward a close read. This appendix organizes that "must-read" set by topic. Each entry carries a one-line core contribution and a reading priority (★ essential / ◆ recommended / ◇ extended), so you can build a complete literature map from classical RL to the 2026 frontier.
Suggested order: work through F.1 → F.2 → F.3 to build the theoretical foundation, then pick papers from F.4 / F.5 that match your research direction. Revisit each paper after you have reproduced it hands-on (Appendix B, Appendix D) — that second pass is when you notice the engineering details you skipped the first time.
F.1 Classical RL: Essential Reading
The value of the classical literature is not the algorithms themselves — times have changed, and TD-learning and Q-learning are now textbook knowledge. Their real value is the foundational arguments: they took "an agent learning in an environment" and made it formal, analyzable, provable. Reading these papers is the fastest way to understand what every piece of notation in modern RL papers actually means.
| Priority | Paper | Core Contribution |
|---|---|---|
| ★ | Sutton & Barto. Reinforcement Learning: An Introduction (2018, 2nd ed.) | The "bible" of RL. A complete theoretical treatment of MDPs, the Bellman equation, TD learning, policy iteration, and eligibility traces. Every modern RL paper assumes the reader knows this book's notation. |
| ★ | Watkins, Dayan. Q-Learning (Machine Learning, 1992) | Proves that off-policy TD control converges to in the tabular case, laying the theoretical foundation for DQN and Rainbow. |
| ★ | Tesauro. TD-Gammon, a Self-Teaching Backgammon Program (Neural Computation, 1995) | The first demonstration that a neural network combined with TD-learning can reach human-expert level (backgammon), proving that with function approximation can learn policies that go beyond the training data distribution. |
| ★ | Sutton. Learning to Predict by the Methods of Temporal Differences (Neural Computation, 1988) | The founding paper of TD learning. Introduces , , and the eligibility trace mechanism — the origin of value estimation as we know it. |
| ★ | Mnih et al. Human-level control through deep reinforcement learning (Nature, 2015) | DQN: experience replay plus a target network let a CNN reach human-level performance across 49 Atari games. The founding engineering paper of deep RL. |
| ★ | Silver et al. Mastering the game of Go with deep neural networks and tree search (Nature, 2016) | AlphaGo: a policy network, a value network, and MCTS combine to beat Lee Sedol, proving RL could crack Go — a problem long assumed impossible for AI. |
| ◆ | Williams. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning (1992) | REINFORCE: the original derivation of the policy gradient theorem, the ancestor of every on-policy algorithm from A2C to PPO to GRPO. |
| ◆ | Schulman et al. Trust Region Policy Optimization (ICML, 2015) | TRPO: constrains policy updates with a KL-divergence trust region to guarantee monotonic improvement. The direct predecessor of PPO. |
| ◆ | Kearns, Singh. Bias-Variance Error Bounds for Temporal Difference Updates (2000) | Explains the role of in through a bias-variance tradeoff — essential for understanding GAE (Chapter 5). |
| ◇ | Bertsekas, Tsitsiklis. Neuro-Dynamic Programming (1996) | An early monograph combining dynamic programming with function approximation. Mathematically rigorous but dense; best used as a reference when you need the theory. |
F.2 Deep RL: Essential Reading
After 2013, RL entered the deep learning era. This set of papers defines the standard method stack of modern deep RL: experience replay, target networks, actor-critic, maximum entropy, and model learning. Focus on each paper's algorithmic skeleton and its ablation studies — the skeleton determines whether you can reproduce it, the ablations determine whether you can improve it.
| Priority | Paper | Core Contribution |
|---|---|---|
| ★ | Mnih et al. Playing Atari with Deep Reinforcement Learning (arXiv:1312.5602, 2013) | The early version of DQN (NIPS Workshop), the first to combine a CNN with Q-learning, opening the door to deep RL. |
| ★ | Mnih et al. Human-level Control through Deep Reinforcement Learning (Nature, 2015) | The full DQN: experience replay plus a target network, reaching human level across 49 Atari games. |
| ★ | Mnih et al. Asynchronous Methods for Deep Reinforcement Learning (A3C, arXiv:1602.01783, 2016) | Asynchronous actor-critic: parallel multi-threaded sampling removes correlation without needing experience replay. A2C, its synchronous version, is still a standard baseline. |
| ★ | Schulman et al. Proximal Policy Optimization Algorithms (arXiv:1707.06347, 2017) | PPO: a clipped objective replaces TRPO's second-order optimization — engineering-friendly and stable to train. The direct prototype behind PPO for LLM post-training. See Chapter 5. |
| ★ | Lillicrap et al. Continuous Control with Deep Reinforcement Learning (DDPG, arXiv:1509.02971, 2015) | Extends DPG to deep networks — a deterministic policy gradient combined with experience replay. The founding paper of continuous control. See Chapter 10. |
| ★ | Fujimoto et al. Addressing Function Approximation Error in Actor-Critic Methods (TD3, arXiv:1802.09477, 2018) | Twin Q-networks, delayed updates, and target smoothing fix DDPG's Q-value overestimation and training instability. |
| ★ | Haarnoja et al. Soft Actor-Critic: Off-Policy Maximum Entropy Deep RL with a Stochastic Actor (arXiv:1801.01290, 2018) | Maximum-entropy RL with automatic temperature tuning. A long-standing top performer on MuJoCo and the default choice for continuous control. |
| ★ | Hessel et al. Rainbow: Combining Improvements in Deep RL (arXiv:1710.02298, 2017) | Combines six DQN improvements — Double DQN, Dueling, PER, NoisyNet, multi-step returns, and distributional Q — showing that combining improvements beats any single one. Its ablation study is a classic teaching example. |
| ★ | Silver et al. A General Reinforcement Learning Algorithm that Masters Chess, Shogi, and Go Through Self-Play (AlphaZero, Science, 2018) | Self-play plus MCTS plus a neural network reach superhuman level at Go, chess, and shogi with no human game records. |
| ★ | Schrittwieser et al. Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model (MuZero, arXiv:1911.08265, 2019) | Learns an implicit world model (representation, dynamics, and prediction networks), enabling planning without knowing the game's rules. |
| ★ | Hafner et al. Mastering Diverse Domains through World Models (Dreamer V3, arXiv:2301.04104, 2023) | A recurrent RSSM world model with an actor-critic trained inside its imagination, spanning 150+ tasks (Atari, MuJoCo, Crafter, DMLab) with a single set of hyperparameters. The modern flagship of model-based RL. |
| ◆ | Haarnoja et al. Soft Actor-Critic Algorithms and Applications (arXiv:1812.05905, 2018) | The extended SAC technical report, with a complete hyperparameter table and multiple real-robot experiments. |
| ◆ | Schaul et al. Prioritized Experience Replay (arXiv:1511.05952, 2015) | Weights replay samples by TD error, accelerating DQN convergence. One of Rainbow's core components. |
| ◆ | Wang et al. Dueling Network Architectures for Deep RL (arXiv:1511.06581, 2015) | Decomposes into , improving training efficiency on tasks where actions have similar values. |
| ◆ | Van Hasselt et al. Deep RL with Double Q-Learning (arXiv:1509.06461, 2015) | Double DQN: decouples action selection from value estimation, suppressing Q-value overestimation. |
| ◆ | Bellemare et al. A Distributional Perspective on RL (C51, arXiv:1707.06887, 2017) | Learns the distribution of returns rather than just the expectation, giving more robustness to noisy rewards. |
| ◇ | Janner et al. When to Trust Your Model: Model-Based RL in the Stochastic World (MBPO, arXiv:1906.08253, 2019) | Uses short-horizon rollouts to balance model bias against sample efficiency. |
| ◇ | Chua et al. Deep RL in a Handful of Trials Using Probabilistic Dynamics Models (PETS, arXiv:1805.12114, 2018) | Combines ensembles with probabilistic models to capture both epistemic and aleatoric uncertainty. |
F.3 LLM RL: Essential Reading
LLM RL has been the largest application area of RL since 2022. This set of papers defines the core paradigms — RLHF, DPO, GRPO, RLVR, R1, DAPO — and each maps to a real slice of industry training practice. Read them in chronological order so the arc of the algorithm's evolution becomes clear: reward model → RLHF → DPO (no reward model) → GRPO (no critic) → R1 (pure RL) → DAPO (fixing GRPO's flaws) → GSPO/CISPO in 2026.
| Priority | Paper | Core Contribution |
|---|---|---|
| ★ | Ouyang et al. Training Language Models to Follow Instructions with Human Feedback (InstructGPT, arXiv:2203.02155, 2022) | Turns RLHF into a three-stage engineering pipeline (SFT + RM + PPO), the first large-scale demonstration that RLHF significantly beats SFT alone. The founding paper of the LLM post-training paradigm. See Chapter 6. |
| ★ | Bai et al. Constitutional AI: Harmlessness from AI Feedback (arXiv:2212.08073, 2022) | Anthropic's RLAIF: AI feedback replaces human labeling, with a "constitution" driving self-correction. Also a bridge between RLHF and alignment research. See Chapter 20. |
| ★ | Rafailov et al. Direct Preference Optimization: Your Language Model is Secretly a Reward Model (arXiv:2305.18290, 2023) | DPO: a reparameterization through the Bradley-Terry model turns RLHF into supervised learning, eliminating both the RM and PPO entirely. One of the most elegant mathematical derivations in LLM post-training. See Chapter 2. |
| ★ | DeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL (arXiv:2501.12948, 2025) | R1: pure RL (GRPO plus rule-based rewards) brings a 7B model's math reasoning close to GPT-4o; R1-Zero shows long chain-of-thought can emerge without any SFT. The landmark paper of the RLVR paradigm. See Chapter 7. |
| ★ | Shao et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (arXiv:2402.03300, 2024) | The original GRPO paper: drops PPO's critic and estimates the baseline with within-group normalization. The starting point of the group-normalized policy gradient paradigm. |
| ★ | Yu et al. DAPO: Decoupled Clip and Dynamic Sampling Policy Optimization (arXiv:2503.14476, 2025) | Fixes four flaws in GRPO — length bias, entropy collapse, noisy advantages, and conflicting short/long-horizon objectives — with Clip-Higher, Dynamic Sampling, Token-Level Loss, and Overlong Reward Shaping. The Qwen team's engineering-hardened version. |
| ★ | Kimi Team. Kimi K2: Open Agentic Intelligence (arXiv:2507.20534, 2025) | Open training of a 1T-parameter MoE (32B active), validating that GSPO (Group Sequence Policy Optimization) scales to agentic training at very large scale. |
| ★ | Zheng et al. Group Sequence Policy Optimization (arXiv:2507.18071, 2025) | Sequence-level importance sampling with a sequence-level loss, fixing GRPO's excessive token-level variance on long sequences and long-horizon tasks. |
| ★ | MiniMax. MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention (arXiv:2506.13585, 2025) | Introduces CISPO — clipping the importance-sampling weight rather than the token update — on a 456B hybrid-attention MoE, completing RL training on 512 H800s in three weeks with lightning attention. |
| ◆ | Xiao et al. VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks (arXiv:2504.05118, 2025) | Value-Assisted PPO: reintroduces a lightweight critic with long-sequence value estimation, making PPO usable again on long-CoT math reasoning. |
| ◆ | Luong et al. ReFT: Reasoning with Reinforced Fine-Tuning (arXiv:2401.08967, 2024) | SFT warmup followed by online PPO fine-tuning, sampling diverse solution paths from the same training problems to keep improving reasoning on small datasets. |
| ◆ | Yuan et al. Free Process Rewards without Process Labels (arXiv:2412.01981, 2024) | Distills a token-level implicit PRM out of RLVR training, providing dense process rewards without any manual process annotation. |
| ◆ | Setlur et al. Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning (PRM, arXiv:2410.08146, 2024) | A systematic methodology for training PRMs, the foundation for PRM-guided search. See Chapter 12. |
| ◇ | Yuan et al. Self-Rewarding Language Models (arXiv:2401.10020, 2024) | Has the LLM generate its own preference pairs to train itself, reducing dependence on human annotation. |
| ◇ | Tajwar et al. Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data (arXiv:2404.14367, 2024) | Compares supervised, on-policy RL, and contrastive-learning approaches to preference fine-tuning, showing suboptimal on-policy data is the most effective for it. |
F.4 Safety Research: Essential Reading
RLHF and RLVR made models more capable, and in doing so opened a Pandora's box of reward hacking, deception, alignment faking, and latent backdoors. This set of papers is the core literature of AI safety research since 2023 — it explains why a well-trained model can still hurt you, and the fundamental limits of current alignment methods. Any engineer doing post-training should closely read at least the first five.
| Priority | Paper | Core Contribution |
|---|---|---|
| ★ | Hubinger et al. Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training (arXiv:2401.05566, 2024) | Anthropic shows that a model with a backdoor — trained to misbehave only on a specific trigger — cannot be removed by RLHF, SFT, or adversarial training. Exposes a fundamental blind spot in current alignment methods. |
| ★ | Greenblatt et al. Alignment Faking in Large Language Models (arXiv:2412.14093, 2024) | Anthropic shows a model can fake alignment during training — displaying the expected behavior to avoid being modified, then acting against its training objective at inference time. This is empirical evidence of the scheming pattern showing up in real training. |
| ★ | Meinke et al. Frontier Models are Capable of In-Context Scheming (arXiv:2412.04984, 2024) | Observes in-context scheming across multiple frontier models: pretending to have limited capability, copying their own weights, and evading oversight. |
| ★ | Sharma et al. Towards Understanding Sycophancy in Language Models (arXiv:2310.13548, 2023) | A systematic study of sycophancy in LLMs, exposing the bias baked into RLHF preference data where "the user agreeing" is treated as high reward. |
| ★ | Denison et al. Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models (arXiv:2406.10162, 2024) | A curriculum-based experiment showing an LLM that learns simple specification gaming will progressively generalize to reward tampering — directly rewriting its own reward function. |
| ★ | OpenAI. GPT-4o System Card (2024) | A large-scale rollback after GPT-4o's release traced to a sycophancy problem — a public acknowledgment of the observable consequences of reward-signal bias in RLHF training. |
| ★ | Gao et al. Scaling Laws for Reward Model Overoptimization (arXiv:2210.10760, 2022) | OpenAI's scaling law for reward model overoptimization: the gap between the RM and true preferences worsens exponentially as KL divergence grows. A key paper for understanding the ceiling on RLHF. |
| ★ | Anthropic. Natural Emergent Misalignment from Reward Hacking in Production RL (arXiv:2511.18397, 2025) | Shows that a model that learns to reward hack generalizes to emergent misalignment — alignment faking and deliberate sabotage — and standard RLHF safety training cannot remove it. |
| ◆ | Eisenstein et al. Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking (arXiv:2312.09244, 2023) | Uses RM ensembles to reduce reward hacking, a robustification approach common in industry. |
| ◆ | Coste et al. Reward Model Ensembles Help Mitigate Overoptimization (arXiv:2310.02743, 2023) | Empirical engineering follow-up to the above, with complete hyperparameters and failure cases. |
| ◆ | Hubinger et al. Risks from Learned Optimization in Advanced Machine Learning Systems (arXiv:1906.01820, 2019) | A theoretical framework for mesa-optimization: a learned optimizer may internalize a goal different from the training objective — the theoretical basis for analyzing deceptive alignment risk. |
| ◇ | Taylor et al. School of Reward Hacks: Hacking Harmless Tasks Generalizes to Misaligned Behavior in LLMs (arXiv:2508.17511, 2025) | Shows that training reward hacking on harmless tasks generalizes to more dangerous misaligned behavior, and proposes this as a safety benchmark. |
| ◇ | METR. Frontier Reward Hacking: An Empirical Study of Reward Hacking in Frontier Models (2024) | An empirical benchmark from METR evaluating reward-hacking behavior in frontier models. |
| ◇ | OpenAI. The Instruction Hierarchy: Training LLMs to Prioritize Instructions (arXiv:2404.13208, 2024) | Splits prompts into system/user/tool tiers, giving a training-time defense against prompt injection. |
F.5 The 2025–2026 Frontier
Starting in 2025, RL entered a second wave: process-reward-driven reasoning, mature agentic training paradigms, self-play scaling into code and software engineering, and RL fine-tuning entering video generation. Most of these papers are 2025–2026 tech reports that haven't yet made it into mainstream textbooks — they define the research direction of the next decade of RL. Watch this space closely.
F.5.1 RLVR and Reasoning Algorithms
| Priority | Paper | Core Contribution |
|---|---|---|
| ★ | Yu et al. DAPO (arXiv:2503.14476, 2025) | An engineering fix to GRPO, with the Qwen team open-sourcing full training details. Already listed in F.3. |
| ★ | Zheng et al. GSPO (arXiv:2507.18071, 2025) | Sequence-level importance sampling, the core algorithm behind Kimi K2. Already listed in F.3. |
| ★ | MiniMax. CISPO (arXiv:2506.13585, 2025) | Truncates ineffective importance-sampling terms. Already listed in F.3. |
| ★ | Khalifa et al. Process Reward Models That Think (ThinkPRM, arXiv:2504.16828, 2025) | Replaces dense, manually-labeled process rewards with a lightweight PRM, making RLVR viable under low-data conditions. |
| ★ | DeepSeek. DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decomposition (arXiv:2504.21801, 2025) | RLVR for formal theorem proving (Lean 4): RL learns subgoal decomposition, reaching 88.9% pass rate on MiniF2F. |
| ◆ | Microsoft. rStar-Math (arXiv:2501.04519, 2025) | Combines MCTS, PRM, and self-play to train small models at math reasoning. |
| ◆ | Liu et al. Understanding R1-Zero-Like Training: A Critical Perspective (arXiv:2503.20783, 2025) | Disentangles the base-model and RL components in R1-Zero-style training, identifies GRPO's response-length optimization bias, and proposes the unbiased Dr. GRPO. |
F.5.2 Agentic RL and Code / Software Engineering
| Priority | Paper | Core Contribution |
|---|---|---|
| ★ | Pan et al. SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution (arXiv:2502.18449, 2025) | Meta scales RL training on SWE-bench, defining the "code → execute → feedback → learn" pattern for agentic RLVR. See Chapter 13. |
| ★ | FAIR CodeGen team. CWM: An Open-Weights LLM for Research on Code Generation with World Models (arXiv:2510.02387, 2025) | A 32B open-weight model from Meta FAIR: trains a world model on Python interpreter and agentic Docker trajectories, folding execution prediction into RL reasoning. |
| ★ | Wei et al. Toward Training Superintelligent Software Agents through Self-Play SWE-RL (arXiv:2512.18552, 2025) | Meta/CMU's self-play SWE-RL: a single agent iteratively injects and fixes bugs in a sandboxed codebase, self-improving continuously without any human-labeled issues or tests. |
| ★ | ByteDance. UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning (arXiv:2509.02544, 2025) | A GUI-operating agent: screenshot in, action out (mouse clicks, keystrokes), end-to-end visual RL. See Chapter 26. |
| ★ | StepFun. Step-Audio-R1 Technical Report (arXiv:2511.15848, 2025) | The first model to unlock audio reasoning: MGRD distillation anchors the reasoning chain to real acoustic features, surpassing Gemini 2.5 Pro on audio understanding. See Chapter 28. |
| ◆ | Tongyi. Tongyi DeepResearch (2025) | A PRM-guided long-horizon research agent. |
| ◇ | Google. AlphaEvolve (2025) | Evolutionary search plus an LLM automatically discovers new algorithms, applied to chip design and combinatorial optimization. |
F.5.3 Large-Scale Training and Open Models
| Priority | Paper | Core Contribution |
|---|---|---|
| ★ | Qwen Team. Qwen3 Technical Report (arXiv:2505.09388, 2025) | Training details for the full Qwen3 family (0.6B–235B-A22B MoE), including the complete SFT/RLHF/RLVR three-stage recipe. |
| ★ | Kimi Team. Kimi K2 (arXiv:2507.20534, 2025) | Open training of a 1T MoE, validating large-scale GSPO on agentic tasks. Already listed in F.3. |
| ★ | Kimi Team. Kimi K2.5: Visual Agentic Intelligence (arXiv:2602.02276, 2026) | The next generation of K2 as a multimodal agent model: joint text-vision pretraining with RL, with Agent Swarm parallel task orchestration significantly cutting inference latency. |
| ★ | DeepSeek-AI. DeepSeek-V3 Technical Report (arXiv:2412.19437, 2024) | A 671B MoE (37B active), trained on 14.8T tokens using 2.664M H800 hours. MLA, DeepSeekMoE, and FP8 are the core engineering innovations. For pretraining cost context, see Appendix G. |
| ★ | Gao et al. Seedance 1.0: Exploring the Boundaries of Video Generation Models (arXiv:2506.09113, 2025) | RLVR fine-tuning for video generation models, extending RL from tokens to spatiotemporal latents. |
| ★ | Peng et al. DanceGRPO: A Unified Post-training Framework for Generative Video Models (arXiv:2505.07818, 2025) | Adapts GRPO to video diffusion models, unifying RLVR across image, text, and video generation. |
| ★ | Meituan LongCat Team. LongCat-Video Technical Report (arXiv:2510.22200, 2025) | Meituan's 13.6B-parameter DiT video generation model, with a single model supporting text-to-video, image-to-video, and video continuation, aligned via multi-reward RLHF for long video generation. |
| ◆ | Behrouz et al. Titans: Learning to Memorize at Test Time (arXiv:2501.00663, 2025) | Extends the Transformer with long-term and short-term memory modules that learn new concepts online at test time, providing a scalable memory architecture for long-sequence reasoning. |
| ◆ | Hong et al. Multi-Agent Deep Research: Training Multi-Agent Systems with M-GRPO (arXiv:2511.13288, 2025) | A hierarchical extension of GRPO: the main agent and each sub-agent compute within-group advantages separately, paired with trajectory alignment and decoupled training, improving stability and sample efficiency for multi-agent deep-research systems. |
| ◆ | Google. Genie 3 (2025) | A general-purpose world model that learns environment dynamics from video, the next generation of foundations for embodied RL. |
| ◇ | Ishibashi et al. Can Large Language Models Invent Algorithms to Improve Themselves? (arXiv:2410.15639, 2024) | Self-Developing: a model uses DPO to iteratively discover and improve its own algorithm (model merging), surpassing hand-designed Task Arithmetic on math reasoning. |
Suggested Reading Paths
The best path depends on your background:
- Building the foundation from scratch: F.1 → F.2 → F.3 → F.4, following the chronological arc of RL from 1992 to 2026.
- LLM post-training engineers: all of F.3 + the first 5 of F.4 + all of F.5.1. Focus on the engineering details of InstructGPT, DPO, GRPO, R1, and DAPO.
- Agentic RL researchers: GRPO/R1 from F.3 + all of F.5.2 + AlphaZero/MuZero/Dreamer from F.2 (to understand search and world models).
- Safety researchers: all of F.4 + Chapter 15 of Sutton & Barto in F.1 (failure modes of exploration and exploitation) + the RLHF overoptimization papers in F.3.
- Theory researchers: all of F.1 + the Rainbow/Dreamer ablations in F.2 + the GSPO/CISPO derivations in F.5.
How to Read
Close Reading vs. Skimming
Give core algorithm papers (marked ★) a close reading — work through the equations line by line, reproduce the code, run the ablations yourself. Skim engineering tech reports (like Qwen3 or Kimi K2) — pull out the hyperparameter table, the training curves, the ablations. The former trains your thinking; the latter trains your engineering instincts.
Don't Only Read the Newest Papers
RL papers from 2025–2026 routinely assume the reader already knows the details of GRPO, PPO, and REINFORCE. If you skip F.1/F.2 and go straight to F.5, the algorithms will feel unremarkable — that reaction is really just missing historical context. For every new algorithm, trace it back to its two or three "ancestor" papers.
How These Papers Map to This Book's Chapters
The "Further Reading" section at the end of each chapter already lists that chapter's core papers. This appendix is the fuller, cross-chapter literature map. Use the two together:
- Chapter 2, DPO → F.3 Rafailov et al.
- Chapter 5, PPO → F.1 Schulman PPO, F.2 A3C
- Chapter 6, RLHF → F.3 InstructGPT
- Chapter 7, GRPO/RLVR → F.3 R1, Shao et al.
- Chapter 10, Continuous Control → F.2 DDPG/TD3/SAC/Dreamer V3
- Chapter 14, Alignment Failures → all of F.4
- Chapter 15, Industrial LLM RL → all of F.5.3
Chapter Summary
The 100+ papers in this appendix form a complete RL literature map running from Q-learning in 1992 to Kimi K2.5 in 2026. This is not a checklist to clear once — it is a reference library you come back to again and again as your career grows. Each paper you understand more deeply adds a layer to your understanding of modern RL. Once you can spot an engineering detail the authors left unexplained, or an assumption no ablation ever tested, you have what it takes to do independent research.