Policy Distribution and On-policy / Off-policy Theory

If post-training of large language models is understood merely as a process of changing datasets, one cannot control architectural limitations and bottlenecks. The essence of post-training is a continuous alignment process that shifts the Policy Distribution, which exists in a high-dimensional token sequence space, towards our target distribution.

In this document, I will re-interpret SFT, Off-policy Distillation, On-policy SFT, DPO, and GRPO from the perspective of geometric distribution shift, and analyze the mathematical differences in how the data generation entity affects the backpropagation signal of the model's internal loss function.

Why are answers generated by a Teacher and answers generated by the current model different learning signals, even if they are both correct? #

To put it simply, the two types of data have a physical difference: whether they exist within or outside the Support of the Current Policy of the model's current probability distribution.

This leads to completely different mathematical directions and stability for model weight updates.

• Teacher-generated correct answer (Off-policy Signal): Data sampled from the Teacher model's distribution $\pi_{teacher}(y|x)$ is likely to be in a high-dimensional sparse region (Out-of-support) with near-zero probability from the perspective of the current Student model's distribution $\pi_\theta(y|x)$. If the model is forced to imitate this correct answer (Teacher Forcing), it must ignore its internal transition probabilities and forcibly pull up that trajectory, leading to weight distortion. This results in Exposure Bias, where the model cannot recover if it deviates even slightly from the path during inference.
• Current model-generated correct answer (On-policy Signal): A correct answer found by the current model through self-exploration within its distribution $\pi_\theta(y|x)$ is closely mapped onto the gradient space of the current weight state $\theta$. In other words, it's a signal that anchors the optimal path among the transition paths the model already had as a high-probability support set. This is not about forcibly memorizing a new space, but rather an operation that compresses the existing internal density distribution, safely and robustly solidifying inference success rates without drastic changes to the distribution.

7 Key Concepts of Post-Training #

Let's explore the 7 core theoretical indicators and definitions that run through the LLM post-training chain.

Policy Distribution ($\pi_\theta(y|x)$) #

Given a prompt sequence x, this is the conditional probability distribution for the token sequence y to be autonomously generated. Due to the nature of autoregressive models, it is formalized as follows:

Behavior Policy ($\pi_b$) #

This refers to the source distribution that generated (rolled out) the actual token trajectory to calculate the loss function during the training phase.

If it's a human-generated dataset, it's $\pi_{human}$; if it's a GPT-4 generated dataset, it's $\pi_{gpt4}$; if the model itself generates it in real-time, it becomes $\pi_\theta$.

Target Policy ($\pi_\theta$) #

This is the optimization target policy that we aim to reach by finally updating the weights $\theta$ through the loss function.

Reference Policy ($\pi_{ref}$) #

During Preference Optimization (e.g., DPO) or PPO training, this is a baseline distribution used to prevent Reward Hacking, where the model excessively pursues rewards, leading to a collapse of its language capabilities.

It typically uses fixed weights from the point where SFT is completed, serving as a regularization term for the KL Divergence penalty ($D_{KL}(\pi_\theta || \pi_{ref})$).

Distribution Shift & Covariate Shift #

This is a phenomenon where the input context distribution $\pi_b(y_{<t}|x)$ experienced by the model during training and the input context distribution $\pi_\theta(y_{<t}|x)$ encountered by the model itself in a real service inference environment mismatch, leading to an exponential accumulation of errors with each successive layer.

Exposure Bias #

In offline training environments like SFT/Distillation, transition probabilities are always learned only in settings where the previous target correct token is received as input.

However, in the actual inference phase, the model must generate the next token by receiving an incorrectly generated token $\hat{y}_{<t}$ from its previous step. This is a mechanistic debt where a single mistake can lead to the collapse of the entire sequence.

Interpretation of Policy Distribution Shift Mechanisms #

There are five major algorithms. Let's re-interpret these five alignment methodologies, going beyond "what data was used" to "by what distribution law does the weight space shift".

SFT (Supervised Fine-Tuning) #

Geometrically, it forcibly aligns the density of the target policy $\pi_\theta$ onto the fixed sample paths of the $\pi_{human}$ or $\pi_{expert}$ distribution.

Distribution characteristics: It is a completely offline learning process. Since it maximizes only the log-likelihood of the correct trajectory without considering the actual probability space landscape within the model, risks of Covariate Shift and Exposure Bias persist during training.

Off-policy Distillation #

Geometric mechanism: Implants the output distribution of a large, superior model's fixed generation policy $\pi_{teacher}$ into a subordinate model $\pi_\theta$.

Distribution characteristics: A perfect Off-policy structure where the Behavior Policy is fixed as $\pi_{teacher}$.

Due to the parameter capacity limitations of the Student model, it cannot fully imitate the Teacher distribution's boundary (Support boundary), leading to an Imitation Gap where it only superficially resembles the Teacher.

On-Policy SFT (Rejection Sampling / RFT) #

Geometrically, it multi-explores the surrounding space from the current weight state $\pi_{\theta_{old}}$ through a temperature kernel, then contracts $\pi_\theta$ onto the set of successful trajectories $\mathcal{D}_{pass} \sim \pi_{\theta_{old}}$ using a validation filter.

Distribution characteristics: From a data acquisition perspective, it is on-policy because the data's origin is the current model's self-distribution. However, since the weight optimization mathematical formula itself uses a cross-entropy offline loss function, there is a risk of mode collapse where the self-distribution becomes fixated on a specific local basis if the loop is excessively repeated.

DPO (Direct Preference Optimization) #

Based on the probability landscape of the Reference Policy $\pi_{ref}$, it applies an attractive force (Pull) to increase the relative density of Chosen paths and a repulsive force (Push) to decrease the density of Rejected paths.

Generally, it uses a Preference Dataset (offline fixed distribution), making it mathematically closer to Off-policy optimization. To compensate for this, it is evolving into an Online DPO architecture that performs DPO by sampling Chosen/Rejected tokens in real-time based on tokens directly generated by the learning $\pi_\theta$.

The difference from RFT is that RFT only remembers good answers and discards bad ones, whereas DPO increases the probability of chosen answers and decreases the probability of rejected answers, thus considering the contrast between good and bad answers.

GRPO (Group-Relative Policy Optimization) #

It dynamically transforms the entire distribution by converting the relative reward landscape (Advantage Landscape) among $G$ samples extracted from the currently active online distribution $\pi_\theta$ into a mathematical gradient surface.

As a perfectly pure online on-policy structure, it directly interacts with the actual probabilities activated by the current weights, gradually carving out subtle weight gradients. This fundamentally destroys Exposure Bias and becomes the driving force for the model to acquire its own error recovery paths.

Algorithmic Classification Concepts from a Distribution Perspective #

This is a specification that summarizes, at a glance, which distribution matrices each post-training architecture binds on VRAM memory and how it induces weight transitions.

| Generation | Data Source | Coding Benchmark (HumanEval Pass@1) | Data Fragmentation/Redundancy (Token Diversity) | Incorrect Answer Deviation Rate (Exposure Bias) | W&B Reward Convergence (Implicit Reward Margin) |
|------------------|--------------------------------|---------------------------------:|----------------------------------------:|----------------------------|------------------------------------------:|
| Iteration 0 | Human SFT Baseline | 45.2% | 88.5% | High | 0.00 (Baseline) |
| Iteration 1 | πθ₀ Rollout + Verify | 52.4% | 86.1% | Medium | +0.22 |
| Iteration 2 | πθ₁ Rollout + Verify | 58.9% | 84.0% | Low | +0.45 |
| Iteration 3 | πθ₂ Rollout + Verify | 64.1% | 82.5% | Minimal (Stable) | +0.68 (Optimal Convergence) |
| Iteration 4 | πθ₃ Rollout + Verify | 63.8% | 71.2% | Minimal | +0.72 (Stagnation Occurs) |
| Iteration 5 | πθ₄ Rollout + Verify | 61.2% (Signs of Collapse) | 52.4% (Risk) | Minimal | +0.95 (Overfitting) |

렌더링 결과:

When designing large-scale post-training clusters, one must move beyond merely collecting good data. It is necessary to proactively define what kind of policy distribution interface communication network to build, tailored to hardware budget and target domain characteristics. Only then can performance degradation and budget inefficiencies be controlled during the deployment phase.