Explaining Models for Predicting At-Risk Students
Models for predicting at-risk students involve creating a predictive model to identify students at risk of failing a module early on. These models utilize machine learning algorithms, training data, and various factors to predict outcomes. Interpretable models focus on easily understandable criteria, while black box models, though potentially more accurate, are harder to interpret. The importance of interpretability lies in building trust and ensuring the validity of model decisions in educational settings like OU Analyse.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Explaining models for predicting at-risk students Look under the bonnet Jakub Kocvara, Dr. Martin Hlosta, Prof. Zdenek Zdrahal
What is OU Analyse? Creating a predictive model that identifies students at risk of failing the module as early as possible so that OU intervention is efficient and meaningful. Tutor facing dashboard (Early Alert Indicators) Supporting around 250 modules at the moment
How does a machine learning model work? Current presentation demographic factors previous study results online activity Prediction data Machine Learning algorithm Training data Model Previous presentation demographic factors previous study results online activity assignment result Predicted assignment result Predicted outcome
What are some properties of a model? Predictive power e.g. accuracy Training speed training can be a very expensive operation Interpretability Interpretability how much do we understand the inner workings of a model often connected to complexity Complexity
Interpretable models Last assignment score > 60 Linear model, decision trees, Last week online activity Online activity last week = Yes Forum Posts > 5 Not Submit Not Submit Submit Submit Last assignment score
Black box models More complex models can be more accurate, but are inherently harder to interpret Large decision trees, ensemble methods, neural networks,
Why is intepretability important? Trust providing tutors with reasoning behind model s decisions If the users do not trust a model or a prediction, they will not use it Ribeiro, et al. (2016) Error checking assessing validity of the model by analysing suspicious predictions Can point to incomplete or wrongly chosen training dataset Ethical and legal reasons GDPR s right to explain
Explaining predictions in OU Analyse Explanations generated by an independent Na ve Bayes model Some intrinsic disadvantages Expecting independent variables Explanations are usable only when Na ve Bayes agrees with the main model
The importance of variables in the model
LIME Local Interpretable Model-agnostic Explanations Takes an individual prediction and tweaks individual features to create new instances Weighs them by proximity to the original Tries to train a simple model (i.e. linear regression), that approximates the black-box model well in the vicinity of the original prediction Evaluates which feature values confirm or contradict black-box model s hypothesis
Future of OU Analyse Work in progress: Incorporating LIME module into the predictions pipeline Goal is to have every student s prediction explained and showing on tutor s dashboard This would aid tutors in while deciding whether to act on the basis of a prediction or not Increased trust in our models would lead to increased adoption and ultimately retention
Sources "Why Should I Trust You?": Explaining the Predictions of Any Classifier (2016) Ribeiro, Singh, Guestrin Interpretable machine learning (2019) Cristoph Molnar A Unified Approach to Interpreting Model Predictions (2017) Lundberg, Lee
