IC-443Model inconsistency on probing questions negatively correlates with skill-slice accuracy (r = -0.675), with models contradicting themselves more often on skills where they perform poorly
The paper generates 20 probing questions per skill targeting a single skill in isolation, obtains 5 responses per probe per model, and measures the rate of self-contradiction. Across three sets of 20 skills (lowest, median, highest slice accuracy), models contradict themselves at significantly higher rates for low-accuracy skills. The overall Pearson correlation between probe inconsistency rate and skill-slice accuracy is r = -0.675. Skills where all models contradict themselves more often than not include musical note identification, eye direction analysis, and enumerating objects.
Evidence
correlational
Key metric
r = -0.675 correlation between probe inconsistency rate and skill-slice accuracy
Caveat
Probe inconsistency rate can only be a factor of 0.05 since the implementation checks 20 claims per probed skill, limiting the resolution of the correlation.