Big Day in Education: Deep Knowledge Tracing Algorithms

Introduction

• Presenter: Ryan Baker
• Topic: Deep knowledge tracing (DKT) family of algorithms
• Collaboration with Richard Scruggs
• Originated from DKT by Chris Peach and colleagues
• Based on long short-term memory networks (LSTM)
• Predicts student performance on future items

DKT Overview

• Initial claims of better performance than BKT and PFA
• Xiaoluxiang et al. criticized the initial study for data misuse
• Comparison studies showed similar performance among DKT, BKT, and temporal IRT
• Led to the emergence of DKT family algorithms

Key Issues and Improvements

• Degenerate Behavior:

  • Reported by Jung and Jung
  • Wild swings in probability estimates
  • Solutions: Regularization techniques by Jung and Jung, leading to DKT+

• Interpretability:

  • DKT predicts item correctness, not skills
  • Zhang et al. proposed DKVMN for skill mapping
  • Li and Jung proposed KQN for skill estimates
  • Jung 2019 proposed DeepIRT for more interpretable estimates

• Skill Estimation Validity:

  • Difficulty in validating skill estimates
  • Proposed solution: AOA by Scruggs et al., using human-derived skill item mapping

Algorithm Performance and Evaluation

• What is DKT Learning?:

  • Findings by Ding and Larson: DKT learns student overall ability
  • Zhang et al.: DKVMN primarily benefits the first attempt

• Further Variants and Enhancements:

  • SACT by Pandey and Kouripas: Attentional weights on exercises
  • AKT by Gauch et al.: Uses past practice history
  • ProcessBERT: Incorporates timing and resource use

• Methodological Concerns:

  • Issues with validation and overfitting
  • Poor cross-validation in DKT family papers
  • Gervais et al. showed DKT benefits with solid evaluation
  • Schmucker et al. found feature-based logistic regression often outperforms DKT

Frontiers and Future Work

• Beyond Correctness:
  • Option tracing and open-ended knowledge tracing
  • Future work involves refining deep learning frameworks

When to Use DKT Algorithms

• Benefits: Predicting next problem correctness, datasets with balanced attempts
• Limitations: Interpretability, small datasets, adding new items

Conclusion

• DKT family is evolving with ongoing research
• Next topic: Memory algorithms