let's say that I run a website that currently has this off-white color for its background and I know the mean amount of time that people spend on my website let's say it is 20 minutes and I'm interested in making a change that will make people spend more time on my website my idea is to make the background color of my website yellow but after making that change how do I feel good about this actually having the intended consequence well that's where significance tests come into play what I would do is first set up some hypotheses a null hypothesis and an alternative hypothesis the null hypothesis tends to be a statement that hey your change actually had no effect there's no news here and so this would be that your mean is still equal to 20 minutes is still equal to 20 minutes after after the change to yellow in this case for our background and we would also have an alternative hypothesis our alternative hypothesis is actually that our mean is now greater because of the change that people are spending more time on my site so our mean is greater than 20 minutes after after the change now the next thing we do is we set up a threshold known as the significance level and you will see how this comes into play in a second so your significance level significance level is usually denoted by the Greek letter Alpha and you tend to see significant levels like 1 100th or five one hundredths or one-tenth or one percent five percent or ten percent you might see other ones but we're going to set a significance level for this particular case let's just say it's going to be 0.05 and what we're going to now do is we're going to take a sample of people visiting this new yellow background website and we're going to calculate statistics the sample mean the sample standard deviation and we're going to say hey if we assume that the null hypothesis is true what is the probability of getting a sample with the statistics that we get and if that probability is lower than our significance level if that probability is less than 500 if it's less than 5 percent then we reject the null hypothesis and say that we have evidence for the alternative however if the probability of getting the statistics for that sample are at the significance level or higher then we say hey we can't reject the null hypothesis and we aren't able to have evidence for the alternative so what we would then do I will call this step three in step three we would take a sample take sample so let's say we take a sample size let's say we take 100 folks who visit the new website the yellow background website and we measure sample statistics we measure the sample mean here let's say that the for that sample the mean is 25 25 minutes we are also likely to if we don't know what the actual population standard deviation is which we typically don't know we would also calculate the sample standard deviation then the next step is we calculate a P value and the p-value which stands for probability value is the probability of getting a statistic at least this far away from the mean if we were to assume that the null hypothesis is true so one way to think about it it is a conditional probability it is the probability that our sample mean our sample mean when we take a sample of size n equals 100 is greater than or equal to 25 25 minutes given given our null hypothesis is true and in other videos we have talked about how to do this if we assume that the sampling distribution of the sample means is roughly normal we can use the sample mean we can use our sample size we can use our sample standard deviation perhaps we use a t statistic to figure out roughly what this probability is going to be and then we decide whether we can reject the null hypothesis so let me call that step five so step five there are two situations if my P value if my P value if it is less than Alpha then I reject my null hypothesis reject reject my null hypothesis and say that I have evidence for my alternative hypothesis now if we have the other situation if my P value is greater than or equal to in this case 0.05 so if it's greater than or equal to my significance level then I cannot reject the null hypothesis I wouldn't say that I accept the null hypothesis I would just say that we do not do not reject reject the null hypothesis and so let's say when I do all of these calculations I get a p-value which would put me in this scenario right over here let's say that I get a p-value of 0.03 0.03 is indeed less than 0.05 so I would reject the null hypothesis and say that I have evidence for the alternative and this should hopefully make logical sense because what we're saying is hey look we took a sample and if we assume the null hypothesis the probability of getting that sample is three percent it's three one hundredths and so since that probability is less than our probability threshold here we'll reject it and say we have evidence for the alternative on the other hand there might have been a scenario where we do all of the calculations here and we figure out a p-value AP value that we get is equal to 0.5 which you can interpret as saying that hey if we assume the null hypothesis is true that there's no change due to changing making the background yellow I would have a 50 chance of getting this result and in that situation since it's higher than my significance level I wouldn't reject the null hypothesis a world where the null hypothesis is true and I get this result well you know it's it seems reasonably likely and so this is the basis for significant tests generally and as you will see it is applicable in almost every field you'll find yourself in now there's one last point of clarification that I want to make very very very clear our p-value the thing that we're using to decide whether or not we reject the null hypothesis this is the probability of getting your sample statistics given that the null hypothesis is true sometimes people confuse this and they say hey is this the probability that the null hypothesis is true given the sample given the sample statistics that we got and I would say clearly no that is not the case we are not trying to gauge the probability that the null hypothesis is true or not what we are trying to do is say hey if we assume the null hypothesis were true what is the probability that we got the result that we did for our sample and if that probability is low if it's below some threshold that we set ahead of time then we decide to reject the null hypothesis and say that we have evidence for the alternative