The paper evaluates how Llama 7B's average common sense reasoning score changes when its weights are compressed via the JLCM codebook method. At a compression rate of 7.0 (approximately 2 bits per weight), the model's score drops from 56.140 to 53.081, retaining roughly 95% of original performance. At a milder compression rate of 5.33, the score is 55.940. The authors note this enables loading the model on a 2 GB device such as an iPhone 8.
Evidence
correlational
Key metric
original score: 56.140; JLCM at compression 7.0: 53.081; JLCM at compression 5.33: 55.940; JLCM (+REX) at compression 6.95: 56.002
Caveat
The compression is applied per-layer with a small calibration set; results depend on the specific JLCM hyperparameters and the calibration data used.