Hands on Modern RL

English

简体中文

English

简体中文

Appearance

中

25.2 Future Directions for Multimodal Audio Agents ​

27.1 covered audio reward design. This section looks at the frontier of audio RL — multimodal audio agents (Step-Audio-Chat, Qwen2-Audio), real-time voice dialogue (GPT-4o Voice), and where the field is headed.

A Minimal Voice-Dialogue RL Pipeline ​

This section walks through a minimal, runnable pipeline that demonstrates the core mechanics of audio RL. A full industrial training run needs 8 H100s and several weeks; here we only demonstrate how reward design couples to the PPO update.

Experimental Setup ​

python
# requirements: torch, transformers, librosa, soundfile
import torch
import torch.nn as nn
import torch.nn.functional as F

class AudioDialogueConfig:
    # audio encoder (pseudocode: production uses the Qwen2-Audio encoder)
    audio_encoder_dim = 1280
    audio_frame_rate = 12.5  # Hz, after downsampling
    # LLM decoder (production uses Qwen2.5-32B; simplified here)
    llm_hidden = 4096
    vocab_size = 152000
    # RL config
    group_size = 16         # number of samples per GRPO group
    max_response_len = 1024
    clip_eps = 0.2          # PPO clip
    beta_kl = 0.0           # Step-Audio sets this to 0, allowing free exploration

Model Architecture ​

python
class AudioDialoguePolicy(nn.Module):
    """Audio dialogue policy: audio encoding → LLM reasoning → text + codec generation"""
    def __init__(self, config):
        super().__init__()
        # audio encoder (frozen)
        self.audio_encoder = AudioEncoder(config.audio_encoder_dim)
        for p in self.audio_encoder.parameters():
            p.requires_grad = False
        # adaptor: 25 Hz → 12.5 Hz
        self.adaptor = nn.Conv1d(config.audio_encoder_dim, config.llm_hidden,
                                  kernel_size=2, stride=2)
        # LLM decoder
        self.llm = TransformerDecoder(config.llm_hidden, config.vocab_size)

    def forward(self, audio, question, response_tokens):
        # 1. Encode the audio
        audio_feat = self.audio_encoder(audio)         # (B, T, D)
        audio_feat = self.adaptor(audio_feat.transpose(1,2)).transpose(1,2)

        # 2. Concatenate the [audio, question, response] sequence
        inputs = concat_modalities(audio_feat, question, response_tokens)

        # 3. Autoregressively predict logits for the response
        logits = self.llm(inputs)
        return logits

Reward Function ​

Implements the three reward categories described in Section 30.4:

python
class AudioReward:
    def __init__(self, grm_model, prosody_ref_dist):
        self.grm = grm_model                # generative reward model
        self.prosody_ref = prosody_ref_dist # reference distribution of human prosody

    def content_reward(self, response_text, ground_truth):
        """Content correctness"""
        # Use LLM-as-judge to decide semantic equivalence
        prompt = f"Judge whether the answer is equivalent to the reference:\nReference: {ground_truth}\nAnswer: {response_text}\nReturn 1 if equivalent, otherwise 0"
        return float(self.grm(prompt))

    def prosody_reward(self, response_audio):
        """Prosodic naturalness"""
        f0 = librosa.pyin(response_audio)         # fundamental frequency
        f0_var = np.std(f0)
        # Wasserstein distance to the human distribution
        f0_w = wasserstein_distance(
            np.histogram(f0, bins=50)[0] / len(f0),
            self.prosody_ref['f0_hist']
        )
        # penalize flattening (a common RLVR failure mode)
        flat_penalty = -max(0, 0.3 - f0_var)
        return -f0_w + 0.5 * flat_penalty

    def format_reward(self, response_text):
        """Check for <think>...</think> formatting (MGRD's key trick)"""
        has_think = '<think>' in response_text and '</think>' in response_text
        return 1.0 if has_think else 0.0

    def total(self, response_text, response_audio, ground_truth, weights=(0.7, 0.2, 0.1)):
        w_c, w_p, w_f = weights
        return (w_c * self.content_reward(response_text, ground_truth)
              + w_p * self.prosody_reward(response_audio)
              + w_f * self.format_reward(response_text))

What the format reward is for

The Step-Audio-R1 paper found that dropping the format reward (setting ) caused reasoning token count to collapse from 2,800 to 1,500, and MMAU dropped by 1.2 points. The reason is that the RL optimizer naturally gravitates toward the most token-efficient policy — answer directly, skip <think>.

Setting the format reward to 0.2 (20% of the total reward) is enough to stabilize the reasoning behavior. This is a key difference between audio RL and text RL: text RL's reward signal is dense enough that CoT emerges naturally, while audio RL has to explicitly reward the reasoning process.

The GRPO Training Loop ​

Training uses GRPO (Group Relative Policy Optimization) — it needs no critic, which suits large models better:

python
def grpo_train_step(policy, ref_policy, reward_fn, batch, config):
    """A single GRPO training step"""
    advantages = []
    log_probs_all = []

    for prompt, audio in batch:
        # 1. Sample G responses for each prompt
        responses = []
        for _ in range(config.group_size):
            with torch.no_grad():
                resp = policy.sample(audio, prompt, config.max_response_len)
            responses.append(resp)

        # 2. Compute the reward for each response
        rewards = torch.tensor([
            reward_fn.total(r.text, r.audio, r.gt) for r in responses
        ])

        # 3. Normalize within the group to get the advantage (the core of GRPO)
        adv = (rewards - rewards.mean()) / (rewards.std() + 1e-8)
        advantages.extend(adv.tolist())

        # 4. Compute the new policy's log π(a|s)
        for resp, a in zip(responses, adv):
            log_probs = policy.log_prob(audio, prompt, resp.tokens)
            log_probs_all.append(log_probs)

    # 5. PPO clip objective (Step-Audio sets β_kl = 0)
    advantages = torch.tensor(advantages).unsqueeze(1)
    policy_loss = 0
    for logp_new, resp in zip(log_probs_all, [r for b in batch for r in [None]]):
        # simplified: a real implementation computes the ratio per token
        pass

    # full PPO clip (see Chapter 5)
    # ratio = exp(logp_new - logp_old)
    # clipped = clip(ratio, 1-eps, 1+eps)
    # loss = -min(ratio * adv, clipped * adv).mean()

    return policy_loss

# main loop
for epoch in range(num_epochs):
    for batch in dataloader:
        loss = grpo_train_step(policy, ref_policy, reward_fn, batch, config)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
Why GRPO instead of PPO

Almost every industrial audio LLM uses GRPO (DeepSeek-R1) or a variant of it, not classic PPO. The reasons:

  1. No critic to train: training a critic for a 32B model costs another full copy of memory; GRPO replaces the critic with in-group normalization
  2. Better suited to discrete rewards: audio rewards are mostly 0/1 binary, which a critic struggles to learn
  3. More stable training: the in-group baseline adapts naturally to varying difficulty

Step-Audio-R1's RL implementation is literally on-policy PPO that samples 16 responses per prompt and normalizes within the group — which is just GRPO expressed as an engineering recipe.

Correcting Self-Cognition ​

There's an issue in industrial audio RL that gets little attention but matters a lot: the model forgets that it's an audio model. Because pretraining data is overwhelmingly text, the model often replies "I can't hear any sound" or "I'm a text model." Step-Audio-R1's correction pipeline:

python
def self_cognition_correction(policy):
    """Three-stage correction of self-cognition errors"""
    # Stage 1: iterative self-distillation + LLM-judge filtering
    for t in range(T):
        responses = policy.sample(audio_perception_queries)
        # the judge keeps only responses with correct self-cognition
        correct = [r for r in responses if judge_acknowledges_audio(r)]
        policy.sft(correct)

    # Stage 2: DPO refinement
    # 8,000 preference pairs: correct cognition (w) vs. text-only cognition (l)
    pref_pairs = build_preference_pairs(correct_cog=positive, text_only=negative)
    policy.dpo(pref_pairs, beta=0.1)

Results:

Training stageSelf-cognition error rate
Base model6.76%
Iterative self-distillation2.63%
Iterative self-distillation + DPO0.02%

DPO's precise alignment pushes the error rate close to zero. This step looks minor, but it matters a great deal at deployment time — users expect the model to handle audio input with confidence, not apologize that it "can't hear."

Chapter Summary ​

Audio RL is the last piece of the puzzle for RL in the LLM era, 2025-2026. This chapter covered three core advances:

  1. Step-Audio-R1's MGRD: solved the inverted-scaling problem in the audio domain — the root cause is text substituting for reasoning, and the fix is iterative distillation that migrates the reasoning substrate from text to acoustics. R1 is the first audio model to actually benefit from test-time compute scaling
  2. Step-Audio-R1.5's shift toward RLHF: identified and broke the "verifiable reward trap" — RLVR optimizes what is said, while users care about how it's said, so RLHF's multi-dimensional preference modeling is needed to fill in prosody, emotion, and coherence
  3. Audio reward design: a weighted combination of three layers — content, prosody, and real-time responsiveness — with a rubric-based generative RM replacing a scalar RM, which is the core engineering difference between audio RL and text RL

At the methodological level, this chapter surfaces three lessons that generalize:

  • Modality grounding determines reasoning quality: reasoning ability can transfer across modalities, but it must be explicitly anchored to features of the correct modality
  • Data quality matters far more than data quantity: a curated 5K-sample set with pass@8 ∈ [3, 6] beats 200K unfiltered samples
  • Reward design is the soul of RL: a single verifiable reward collapses model behavior; a multi-dimensional rubric is what keeps it aligned with real user experience

For the training methodology behind multi-agent collaborative RL, see Chapter 20's Multi-Agent Collaboration and Agent Swarms: when multiple LLM agents cooperate on a task, credit assignment and reward assignment must be handled together.

Further Reading ​

现代强化学习实战课程

Copyright © WalkingLabs