Model Evaluation in Python: Metrics and Techniques
From converting binary to multiclass problems to evaluating clustering performance using metrics like Rand index and adjusted Rand index, this resource delves into techniques for assessing model performance in Python. Explore ways to extend binary metrics, understand different averaging methods, and learn about the importance of adjusted Rand index in countering random label assignments. Dive into detailed explanations and practical examples for comprehensive model evaluation.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
BMEG3105 BMEG3105- -Model Evaluation in Python Model Evaluation in Python YixuanWang( ) Sunday, April 13, 2025 yxwang@cse.cuhk.edu.hk Department of Computer Science and Engineering (CSE) The Chinese University of Hong Kong (CUHK)
Outline From binary to multiclass and multilabel Clustering performance evaluation Rand index Adjusted rand index Cross-validation: evaluating estimator performance 1
From binary to multiclass and multilabel Some metrics are essentially defined for binary classification tasks (e.g. f1_score, roc_auc_score). In extending a binary metric to multiclass or multilabel problems, the data is treated as a collection of binary problems, one for each class. There are then a number of ways to average binary metric calculations across the set of classes, each of which may be useful in some scenario. macro : simply calculates the mean of the binary metrics, giving equal weight to each class micro : gives each sample-class pair an equal contribution to the overall metric (except as a result of sample-weight) weighted : accounts for class imbalance by computing the average of binary metrics in which each class s score is weighted by its presence in the true data sample. https://colab.research.google.com/drive/1cXnsuqGvmhsYB-Irl-tnsw1OEiqEpjom?usp=sharing 2 https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html
From binary to multiclass and multilabel https://colab.research.google.com/drive/1cXnsuqGvmhsYB-Irl-tnsw1OEiqEpjom?usp=sharing 3 https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html
Clustering performance evaluation Rand index: a measure of the percentage of correct decisions made by the algorithm Where: TN is the number of true negatives, FP is the number of false positives, FN is the number of false negatives. TP is the number of true positives, ?? [0,1] 4 https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html
Clustering performance evaluation Problem: However, the Rand index does not guarantee that random label assignments will get a value close to 0. To counter this effect we can discount the expected RI, ?[??] of random labelings by defining the adjusted Rand index as follows: 5 https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html
Clustering performance evaluation Adjusted rand index: groupings or partitions(e.g. clusterings)of these elements, namely class(X) and cluster(Y). The overlap between X and Y can be summarized in a contingency table, where each entry n_ij denotes the number of objects in common between X_i and Y_j Given a set S of n elements, and two 6 https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html
Clustering performance evaluation https://colab.research.google.com/drive/1cXnsuqGvmhsYB-Irl-tnsw1OEiqEpjom?usp=sharing 7 https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html
Cross-validation: evaluating estimator performance https://colab.research.google.com/drive/1cXnsuqGvmhsYB-Irl-tnsw1OEiqEpjom?usp=sharing 8 https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html
