17
ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing , selecting classifiers based on their performance

ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based

Embed Size (px)

Citation preview

Page 1: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based

ROC

1. Medical decision making2. Machine learning3. Data mining research communities

A technique for visualizing, organizing , selecting classifiers based on their performance

Page 2: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based

ROC Confusion matrix

Page 3: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based

benefits

costs

ROC spaceAny classifier on the diagonal may be said to have on information about the class

Page 4: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based

ROC curve

A discrete classifierdecision trees rule sets

Y or N Produces a single point

a Naive Bayes classifier a neural network

probability score

Each threshold value

produces a different point

Vary a threshold from −∞ to +∞ and

tracing a ROC curve

Page 5: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based

ROC curve

Page 6: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based

ROC curve

Threshold= + ∞

Page 7: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based

ROC curve

ROC curves have an attractive property: they are insensitive to changes in class distribution.

Page 8: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based

ROC curve

Page 9: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based

ROC curve

Page 10: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based

AUCDefinition: Area under an ROC Curve

The AUC has an important statistical property

1. It is equivalent to the Wilcoxon test of ranks2. It is also closely related to the Gini coefficient Gini + 1 = 2 × AUC

Page 11: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based

Averaging ROC curvesThe error bars

Page 12: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based

Decision problems with more than two classes

Multi-class ROC graphs

Multi-class AUC

Page 13: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based

Iso-performance line

ability: 1. class skew 2. error costs

This equation defines the slope of an iso-performance line.

Conclusion: Lines “more northwest” (having a larger TP-intercept) are better because they correspond to classifiers with lower expected cost.

Page 14: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based

Combining classifiers

Page 15: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based

Conditional combinations of classifiers to remove concavities

1.idiosyncracies in learning 2.small test set effects

Page 16: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based

Conditional combinations of classifiers to remove concavities

Page 17: ROC 1.Medical decision making 2.Machine learning 3.Data mining research communities A technique for visualizing, organizing, selecting classifiers based

Logically combining classifiers

2. c4= c1 c2∨

1. c3 = c1 c2∧