Let us recall the toy example from our
previous post:
We want to find out if introducing a new
feature in our product or web page will improve a given metric, like the
number of clicks on a certain button.
Let's call
N1 and
N2 the number of button clicks per unit of time before and after deploying the new feature. The naive approach to measuring progress in this example is to directly compare
m1 and
m2, the number of clicks measured during a period of time of length
T before and after deploying the new feature. We argued in our
previous post that this approach is essentially
wrong, since it doesn't acknowledge the fact that we are measuring things subject to
randomness.
The fallacy of the naive approach is to assume that
m1 and
m2 are equal to
N1 and
N2, respectively. Fortunately for us,
Probability Theory teaches us exactly what to do in this situation. Variables
N1 and
N2 should be considered to be
random variables (think of two dice) and
m1 and
m2 their
samples (think of the outcome of throwing each die once). We have to be careful when concluding that
N1 >
N2 just because
m1 >
m2 and take a look at the next section.