-
Z-Test vs T-Test
Based on the Wikipedia definitions for a Z-Test and T-Test:
*Difference between Z-test and t-test: Z-test is used when sample size is large (n>50), or the population variance is known. t-test is used when sample size is small (n<50) and population variance is unknown. … For large sample sizes, the t-test procedure gives almost identical p-values as the Z-test procedure.*
I see that most website A/B calculators and blogs advocate the use of T-Tests. Why wouldn’t a Z-Test be used at all times for digital advertising given the large sample sizes (thousands of impressions/clicks/conversions)? Also isn’t the population all users who were shown ads whether or not they clicked or converted, so you can calculate the variance?
-
3 Replies
-
Are you talking about split testing ad copy and knowing when the difference between the values are high enough to conclude a result along with the confidence level?
-
The real difference between Z-Test (Gauss Test) and T-Test is not the population size but the assumption about the standard deviation.
In the Z-Test, you assume that you know exactly what the real standard deviation is (which is very unrealistically of course). In the T-Test you assume that you don’t know it so you have to estimate it first which is a more realistic scenario. This is the reason why the T-Test is used in the real world and not the Z-Test.
-
It has been a while since I’ve taken my probability and statistics class so take this with a grain of salt.
You are correct, T tests are used for a small n while z tests are used on a large n. BUT, t tests require you to use the sample variance while z tests require you to use the population variance. So they require different numbers. But I also remember our professor telling us that if we dont have a a table and just a t table to use the t table with infinite degrees of freedom to get our z value. So they are kind of interchangeable. But there is a difference and follow that rule to get more accurate results. But also 50 isnt the definined line between z and t tests. Some people consider 40 a big n and will use that as their boundary when to switch between z and t.
Just remember it’s been a while since I’ve learned this so you might want to fact check this. I only posted because I didnt see anyone else and wanted to help a little.
