Self-Distillation and Iterative Self-Improvement

The post-training paradigm is moving beyond the stage of refining fixed static datasets, converging towards an Iterative Self-Improvement structure where models autonomously generate and validate data, progressively evolving their architecture.

This post diagnoses the causes of Model Collapse, a technical risk that arises during self-data training, and analyzes the low-level mechanisms of On-Policy Distillation and iterative optimization loops that defend against it.

Does Self-Data Training Improve Performance or Amplify Errors? #

Fundamental Risks: Error Amplification and Model Collapse #

If a model repeatedly undergoes self-training (SFT/DPO) using only datasets it has generated ($\mathcal{D}_{syn} \sim \pi_\theta$) without strict external control, subtle hallucinations and biases within the model accumulate with each generation.

As data entropy decreases, Model Collapse occurs, leading to the repetition of specific words or the simplification of logic.

In other words, the model ends up learning its own simple errors more strongly.

Solution: Strict Filtering (Sifting) and On-Policy Distillation #

To achieve self-improvement, two engineering conditions must be met.

1. Deterministic Verifier (Programmatic Verifier) or Powerful Superior Model (Teacher Judge): A ruthless filtering system must be integrated to retain only data where truth/falsity is definitively determined among the generated data.
2. Resolving Trajectory Mismatch: Off-Policy Distillation learns only optimal paths sampled from the Teacher model's distribution ($\pi_{teacher}$), leading to a problem where the Student model cannot recover if it deviates from the path during the actual inference stage. In contrast, the latest On-Policy Distillation techniques allow the Student model to directly generate trajectories of failure and success (Trajectory, $\tau \sim \pi_\theta$), and then align the learning environment by reading the Teacher model's feedback (Supervision) on these 'incorrect notes,' thereby fundamentally preventing inference distribution errors.

The OPD (On-Policy Distillation) mentioned here is different from the On-Policy SFT discussed previously. OPD (SD) is an on-policy distillation where the model generates answers itself, then takes the token probability distribution Logits to calculate Kullback-Leibler Divergence and updates. That is, it's knowledge-based on word-level probability information, whereas OnSFT looks at the results themselves.

Iterative Self-Improvement Loop Architecture #

The iterative improvement system doesn't end with a single training session; instead, it forms a virtuous cycle where model $\pi_{\theta_t}$ becomes the data source for the next generation, $\pi_{\theta_{t+1}}$.

1. Self-Generate (Online Exploration): Perform multi-sampling (Rollout) on a large set of prompts using the current policy $\pi_{\theta_t}$.
2. Judge / Verifier (Multi-faceted Evaluation): Precisely score the generated text results using Execution, Regex, or LLM-as-a-Judge.
3. *Filter (Extracting Incorrect Notes and Forming Preference Pairs) SFT Target: Discard incorrect paths and extract only high-quality paths that were coincidentally successful. (STaR algorithm family)
   1. DPO Target: Build a Preference dataset by matching successful and failed responses derived from the same prompt.
4. Train & Evolve (Weight Update): After updating the model via the optimization engine TRL, derive $\pi_{\theta_{t+1}}$, monitor W&B convergence, and then re-inject it as the acceleration engine for the next iteration.

Self-Distillation means reusing the complete sequences directly generated by the model as a dataset, rather than passing word-level distribution Logits as knowledge, as in traditional deep learning.

At this point, the sentences generated by the model are scored to create chosen and rejected pairs, and DPO is the learning tool that most efficiently updates the weight distribution using these pairs, which is why it's included in the pipeline.

While the SFT method, which only learns correct answers, leads to the problem of reinforcing self-errors because it cannot control the incorrect paths it generates, combining it with DPO allows the model to directly reduce the probability of its own mistakes, thereby defending against error amplification.

In summary, Self-Distillation is a supply method where the model autonomously produces text data, and DPO is a mathematical algorithm that prunes weights by comparing preferences within that generated data.

Iterative Self-Improvement Pseudocode #

Let's look at the conceptual code for dynamically obtaining and progressively updating a DPO dataset by collecting and filtering self-generated data in each generation.

import json
from vllm import LLM, SamplingParams
from transformers import AutoTokenizer
from trl import DPOConfig, DPOTrainer
from datasets import Dataset

def run_iterative_loop(model_path, prompts, iteration_id):
    print(f"=== START ITERATION {iteration_id} ===")
    
    # 1. Self-Generate: 현재 세대 모델로 Rollout 샘플 추출
    # 다양성 확보를 위해 temperature를 높여 프롬프트당 4개씩 생성
    llm = LLM(model=model_path, tensor_parallel_size=2)
    sampling_params = SamplingParams(n=4, temperature=0.7, max_tokens=256)
    raw_outputs = llm.generate(prompts, sampling_params)
    
    # vLLM 인스턴스 해제 (VRAM 확보 후 SFT/DPO 트레이닝에 자원 반환)
    import gc; import torch
    del llm; gc.collect(); torch.cuda.empty_cache()
    
    # 2 & 3. Judge & Filter: Preference Pair 구축
    preference_data = []
    for output in raw_outputs:
        prompt_text = output.prompt
        completions = [out.text.strip() for out in output.outputs]
        
        # 외부 Verifier 로직 작동 (여기서는 코드 구문 컴파일 및 길이 기반 가상 검증)
        scored_completions = []
        for c in completions:
            score = 1.0 if ("def " in c and len(c) > 50) else 0.0 # 간단한 룰셋 예시
            scored_completions.append((c, score))
            
        # 정답(Chosen)과 오답(Rejected) 분리 매칭
        chosen = [c for c, s in scored_completions if s == 1.0]
        rejected = [c for c, s in scored_completions if s == 0.0]
        
        if chosen and rejected:
            preference_data.append({
                "prompt": prompt_text,
                "chosen": chosen[0],
                "rejected": rejected[0]
            })
            
    if len(preference_data) < 10:
        print("유효 데이터 부족으로 루프 조기 종료.")
        return model_path

    # 4. Train: TRL 엔진을 활용한 DPO 갱신
    train_dataset = Dataset.from_list(preference_data)
    tokenizer = AutoTokenizer.from_pretrained(model_path)
    model = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype=torch.bfloat16, device_map="auto")
    
    output_dir = f"./checkpoint_iter_{iteration_id}"
    dpo_config = DPOConfig(
        output_dir=output_dir,
        learning_rate=2e-6,
        per_device_train_batch_size=2,
        num_train_epochs=1,
        bf16=True,
        report_to="wandb" # Weights & Biases 실험 추적 연동
    )
    
    trainer = DPOTrainer(model, args=dpo_config, train_dataset=train_dataset, tokenizer=tokenizer)
    trainer.train()
    
    # 모델 저장 후 자원 해제 및 다음 세대 경로 반환
    trainer.save_model(output_dir)
    del model, trainer; gc.collect(); torch.cuda.empty_cache()
    
    return output_dir

$\pi$ denotes the model's token generation probability distribution (policy), and the subscript $\theta_0$ refers to the weight (parameter) state of the 0th generation (initial Baseline) model. In other words, $\pi_{\theta_0}$ is an academic notation for 'the 0th generation model itself.' Here, 'Rollout' refers to the process where this 0th generation model is given prompts and outputs (infers) multiple answer sentences. 'Verify' is the stage where these answers, directly generated by the model, are scored as true/false using a verifier like a compiler or a regular expression parser. In conclusion, '$\pi_{\theta_0}$ Rollout + Verify' means that the 0th generation model's self-generated answers are filtered by a verifier, and those that pass are then used as a self-dataset for the next generation's training source.

Intelligence Bootstrapping Phase: Iteration 1 -> 3 #

Up to the initial 3 generations, the coding accuracy Pass@1 grows linearly from 45.2% to 64.1%.

The reason for this is that various failure paths derived within the Student model's current computational capabilities are filtered by the superior Verifier, and log probabilities are mapped primarily to successful paths. This compresses the probability space for correcting and recovering from self-generated errors. The Exposure Bias (error deviation rate) is drastically mitigated.

Self-Saturation and Model Collapse Limit (Iteration 4->5) #

Beyond the 4th generation, the accuracy increase halts, and by the 5th generation, performance regresses to 61.2%. At this point, the Token Diversity metric plummets to 52.4%.

The reason is that without the injection of new external knowledge, learning cycles only within the self-data pool, leading the model to excessively reuse (Mode Collapse) specific coding styles and word sets that allow it to safely produce correct answers. Although the W&B Reward Margin continues to widen (mathematical optimization proceeds), the diversity of language is compromised, and benchmark performance is destroyed, a typical sign of overfitting.

Optimization Strategy #

When designing Iterative Self-Improvement at a commercial level, an infinite loop should not be run.

The point at which the token diversity metric falls below 80% (between Iteration 3 and 4 in this metric) should be set as the Early Stopping threshold for training.

Alternatively, an architectural defense line that periodically processes and injects new human instructions or external data sources must be combined to secure a robust agent brain without infrastructure waste.