The Scientific Approach to Measuring Startup Progress (Part 2/2)

Summary of the Problem

Let us recall the toy example from our previous post:
We want to find out if introducing a new feature in our product or web page will improve a given metric, like the number of clicks on a certain button.
 Notation:
  • N1 and N2 are the number of button clicks per unit of time before and after deploying the new feature, respectively
  • n1 and n2 are the number of metric measurement samples before and after deploying the new feature, respectively
  • k1 and k2 are the sum of the values of all n1 and n2 samples, respectively
  • The null hypothesis of our experiment is E[N2] ≤  E[N1], where E[N1] and E[N2] are the average values of N1 and N2, respectively
  • a is the significance level, which is the probability of making a mistake when rejecting the null hypothesis. We assume that a=0.05

The Scientific Approach to Measuring Startup Progress (Part 1/2)

Let us recall the toy example from our previous post:
We want to find out if introducing a new feature in our product or web page will improve a given metric, like the number of clicks on a certain button.
Let's call N1 and N2 the number of button clicks per unit of time before and after deploying the new feature. The naive approach to measuring progress in this example is to directly compare m1 and m2, the number of clicks measured during a period of time of length T before and after deploying the new feature. We argued in our previous post that this approach is essentially wrong, since it doesn't acknowledge the fact that we are measuring things subject to randomness.

The fallacy of the naive approach is to assume that m1 and m2 are equal to N1 and N2, respectively. Fortunately for us, Probability Theory teaches us exactly what to do in this situation. Variables N1 and N2 should be considered to be random variables (think of two dice) and m1 and m2 their samples (think of the outcome of throwing each die once). We have to be careful when concluding that N1 > N2 just because m1 > m2 and take a look at the next section.

Measuring Success in your Lean Startup

"The Lean Startup provides a scientific approach to creating and managing startups and get a desired product to customers' hands faster"
Source: Principles of the Lean Startup
 I'm a big fan of the Lean Startup methodology. It proposes a simple Build-Measure-Learn process that allows you growing your business with maximum acceleration. This process should be executed in cycles and the main goal of a Lean Startup is to to reduce the time it takes to go through a whole cycle.

 The Lean Startup focuses on learning how to build a sustainable business. This learning has to be validated scientifically, by running experiments that test each element of the startup's vision. The results from these experiments are used to validate the experiment's hypotheses during the Measure step of the Build-Measure-Learn process.

The Measure step is the key to the Lean Startup process. Without a sound experimental design the lessons we learn from our measurements could be completely wrong. This observation applies not only to Lean Startups, but to any startup that uses some sort of metrics to measure and track its progress.