IC-955All baseline knowledge tracing models either lack significant correlation or negatively predict causal support from their inferred prerequisite graphs

Hanqi Zhou, Robert Bamler, Charley M Wu, Álvaro Tejero-Cantero

SourcePredictive, scalable and interpretable knowledge tracing on structured domains

The paper evaluates whether the prerequisite graphs inferred by each baseline model correspond to genuine causal support between knowledge components, measured via Bayesian causal induction on consecutive interaction data. For every baseline (HLR, PPE, DKT, DKTF, HKT, AKT, GKT, QIKT), the regression coefficient relating inferred edge probability to causal support is either non-significant or negative. The best baseline achieves coefficients of 1.05 (p=.253) on assist12, 0.22 (p=.792) on assist17, and 0.42 (p=.593) on junyi15, none reaching significance. In contrast, the 4-dimensional model's edge probabilities yield significant positive coefficients across all three datasets.

Evidence

correlational

Key metric

Best baseline causal support regression: coefficient 1.05 p=.253 (assist12), 0.22 p=.792 (assist17), 0.42 p=.593 (junyi15); graph alignment MRR 0.0082, JS expert 0.0015, JS crowd 0.0047, NLL 3.03 (junyi15)

Caveat

The paper reports only the 'best baseline' aggregate for the causal support regression; individual per-model coefficients are deferred to appendix Figure 12. The NLL for the best baseline (3.03) is lower than PSI-KT's (4.11), indicating the baseline is better on that particular graph-alignment metric.

Model

HLR, DKTF, AKT, GKT, QIKT

Datasets

junyi15 [eval]

Methods

Jaccard similarity [eval]

Extraction

automatic-extraction