The paper compares OpenAI-o1-mini and o1-preview against GPT-4o on CS-Bench. On reasoning-type questions, o1-mini scores 76.12% (+11.97 over GPT-4o's 64.15%) and o1-preview scores 80.98% (+16.83). On knowledge-type questions the gains are smaller (+0.65 and +6.66 respectively). The performance gain comes at a high token cost: o1-mini consumes 341.8 reasoning tokens (×14.2 over GPT-4o) and o1-preview consumes 732.79 (×30.04). Performance increases roughly logarithmically with token expenditure on reasoning questions.
The comparison is limited to English CS-Bench. The token cost makes o1 impractical for many deployment scenarios. The paper notes o1's unique reasoning form required separate analysis from other models.