SourceYour Weak LLM is Secretly a Strong Teacher for Alignment
The paper compares four supervisor models for providing preference labels on unlabeled data: OPT-125M (weak), OPT-1.3B (moderate), LLaMA-3-8B (strong), and GPT-4 (very strong, via prompt engineering following Lee et al. 2023). The resulting labels are used to align an OPT-1.3B student via DPO. Gold rewards are 4.63, 4.84, 4.78, and 4.91 respectively. The performance under weak, moderate, and strong supervisors is nearly comparable, and the paper notes that in some instances the 125M model outperforms GPT-4, indicating that a task-specific weak LLM can serve as a more effective supervisor than a larger model relying solely on prompt engineering.