wait till you see the roc and the a you see they're cool yeah stack quest hello i'm josh starmer and welcome to statquest today we're going to talk about roc and auc and they're going to be clearly explained note this stat quest builds on the confusion matrix and sensitivity and specificity stat quests so if you're not already down with those check out the quests also the example i give in this stat quest is based on logistic regression so even though roc and auc apply to more than just logistic regression make sure you understand those basics let's start with some data the y-axis has two categories obese and not obese the blue dots represent obese mice and the red dots represent mice that are not obese along the x-axis we have weight this mouse is not obese even though it weighs a lot it must be mighty mouse and just full of muscles this mouse doesn't weigh that much but it is still considered obese for its size now let's fit a logistic regression curve to the data when we're doing logistic regression the y-axis is converted to the probability that a mouse is obese now let's just look at the curve if someone told us that they had a heavy mouse that weighs this much then the curve would tell us that there is a high probability that the mouse is obese if someone told us that they had a light mouse that weighs this much then the curve would tell us that there is a low probability that the mouse is obese so this logistic regression tells us the probability that a mouse is obese based on its weight however if we want to classify the mice as obese or not obese then we need a way to turn probabilities into classifications one way to classify mice is to set a threshold at 0.5 and classify all mice with the probability of being obese greater than 0.5 as obese and classify all mice with the probability of being obese less than or equal to 0.5 as not obese using 0.5 as the cut off we would call this mouse obese and this mouse not obese if another mouse weighed this much then we would classify it as obese and if another mouse weighed this much then we would classify it as not obese to evaluate the effectiveness of this logistic regression with the classification threshold set to 0.5 we can test it with mice that we know are obese and not obese here are the weights of four new mice that we know are not obese and here are the weights of four new mice that we know are obese we know that this mouse is not obese and the logistic regression with the classification threshold set to 0.5 correctly classifies it as not obese this mouse is also correctly classified but this mouse is incorrectly classified we know that it is obese but it is classified as not obese the next mouse is correctly classified but this mouse is incorrectly classified the last three mice are correctly classified now we create a confusion matrix to summarize the classifications these three samples were correctly classified as obese and this sample was predicted to be obese but was not obese these three samples were correctly classified as not obese and this sample was predicted to be not obese even though it was obese once the confusion matrix is filled in we can calculate sensitivity and specificity to evaluate this logistic regression when 0.5 is the threshold for obesity little bam because so far this is all review now let's talk about what happens when we use a different threshold for deciding if a sample is obese or not for example if it was super important to correctly classify every ob sample we could set the threshold to 0.1 this would result in correct classifications for all four obese mice but it would also increase the number of false positives the lower threshold would also reduce the number of false negatives because all of the obese mice were correctly classified note if the idea of using a threshold other than 0.5 is blowing your mind imagine that instead of classifying samples as obese or not obese we were classifying samples as infected with ebola and not infected with ebola in this case it's absolutely essential to correctly classify every sample infected with ebola in order to minimize the risk of an outbreak and that means lowering the threshold even if that results in more false positives on the other hand we could set the threshold to 0.9 in this case we would correctly classify the same number of ob samples as when the threshold was set to 0.5 but we wouldn't have any false positives and we would correctly classify one more sample that was not obese and have the same number of false negatives as before with this data the higher threshold does a better job classifying samples as obese or not obese but the threshold could be set to anything between 0 and 1. how do we determine which threshold is the best for starters we don't need to test every single option for example these thresholds result in the exact same confusion matrix but even if we made one confusion matrix for each threshold that mattered it would result in a confusingly large number of confusion matrices so instead of being overwhelmed with confusion matrices receiver operator characteristic roc graphs provide a simple way to summarize all of the information the y-axis shows the true positive rate which is the same thing as sensitivity the true positive rate is the true positives divided by the sum of the true positives and the false negatives in this example the true positives are the samples that were correctly classified as obese and the false negatives are the ob samples that were incorrectly classified as not obese the true positive rate tells you what proportion of ob samples were correctly classified the x-axis shows the false positive rate which is the same thing as one minus specificity the false positive rate is the false positives divided by the sum of the false positives and true negatives the false positives are the non-ob samples that were incorrectly classified as obese and the true negatives are the samples correctly classified as not obese the false positive rate tells you the proportion of not ob samples that were incorrectly classified and are false positives to get a better sense of how the roc works let's draw one from start to finish using our example data we'll start by using a threshold that classifies all of the samples as obese and that gives us this confusion matrix first let's calculate the true positive rate there are four true positives and there were zero false negatives doing the math gives us one the true positive rate when the threshold is so low that every single sample is classified as obese is one this means that every single ob sample was correctly classified now let's calculate the false positive rate there were four false positives in the confusion matrix and there were zero true negatives doing the math gives us one the false positive rate when the threshold is so low that every single sample is classified as obese is also one this means that every single sample that was not obese was incorrectly classified as obese now plot a point at one comma one a point at one comma one means that even though we correctly classified all of the ob samples we incorrectly classified all of the samples that were not obese this green diagonal line shows where the true positive rate equals the false positive rate any point on this line means that the proportion of correctly classified ob samples is the same as the proportion of incorrectly classified samples that are not obese going back to the logistic regression let's increase the threshold so that all but the lightest sample are called obese the new threshold gives us this confusion matrix we then calculate the true positive rate and the false positive rate and plot a point at 0.75 comma 1. since the new point is to the left of the dotted green line we know that the proportion of correctly classified samples that were obese is greater than the proportion of samples that were incorrectly classified as obese in other words the new threshold for deciding if a sample is obese or not is better than the first one now let's increase the threshold so that all but the two lightest samples are called obese the new threshold gives us this confusion matrix we then calculate the true positive rate and the false positive rate and plot a point at 0.5 comma 1. the new point is even further to the left of the dotted green line showing that the new threshold further decreases the proportion of samples that were incorrectly classified as obese in other words the new threshold is the best one so far now we increase the threshold again create a confusion matrix calculate the true positive rate and the false positive rate and plot the point now we increase the threshold again create a confusion matrix calculate the true positive rate and the false positive rate and plot the point the threshold represented by the new point correctly classifies 75 percent of the ob samples and 100 percent of the samples that were not obese in other words this threshold resulted in no false positives now we increase the threshold again and plot the point now we increase the threshold again and plot the point lastly we choose a threshold that classifies all of the samples as not obese and plot the point the point at zero comma zero represents a threshold that results in zero true positives and zero false positives if we want we can connect the dots and that gives us an roc graph the roc graph summarizes all of the confusion matrices that each threshold produced without having to sort through the confusion matrices i can tell that this threshold is better than this threshold and depending on how many false positives i'm willing to accept the optimal threshold is either this one or this one bam now that we know what an roc graph is let's talk about the area under the curve or auc the auc is 0.9 bam the auc makes it easy to compare one roc curve to another the auc for the red roc curve is greater than the auc for the blue roc curve suggesting that the red curve is better so if the red roc curve represented logistic regression and the blue roc curve represented a random forest you would use the logistic regression double bam now one last thing before we're all done although roc graphs are drawn using true positive rays and false positive rates to summarize confusion matrices there are other metrics that attempt to do the same thing for example people often replace the false positive rate with precision precision is the true positives divided by the sum of the true positives and false positives precision is the proportion of positive results that were correctly classified if there were lots of samples that were not obese relative to the number of ob samples then precision might be more useful than the false positive rate this is because precision does not include the number of true negatives in its calculation and is not affected by the imbalance in practice this sort of imbalance occurs when studying a rare disease in this case the study will contain many more people without the disease than with the disease bam in summary roc curves make it easy to identify the best threshold for making a decision this threshold is better than this one and the auc can help you decide which categorization method is better the red method is better than the blue method hooray we've made it to the end of another exciting stat quest if you like this stat quest and want to see more please subscribe and if you want to support statquest well consider getting a t-shirt or a hoodie or buying one or two of my original songs the links for doing that are below alright until next time quest on