The Scientific Approach to Measuring Startup Progress (Part 2/2)

Summary of the Problem

Let us recall the toy example from our previous post:
We want to find out if introducing a new feature in our product or web page will improve a given metric, like the number of clicks on a certain button.
 Notation:
  • N1 and N2 are the number of button clicks per unit of time before and after deploying the new feature, respectively
  • n1 and n2 are the number of metric measurement samples before and after deploying the new feature, respectively
  • k1 and k2 are the sum of the values of all n1 and n2 samples, respectively
  • The null hypothesis of our experiment is E[N2] ≤  E[N1], where E[N1] and E[N2] are the average values of N1 and N2, respectively
  • a is the significance level, which is the probability of making a mistake when rejecting the null hypothesis. We assume that a=0.05

The Scientific Approach to Measuring Startup Progress (Part 1/2)

Let us recall the toy example from our previous post:
We want to find out if introducing a new feature in our product or web page will improve a given metric, like the number of clicks on a certain button.
Let's call N1 and N2 the number of button clicks per unit of time before and after deploying the new feature. The naive approach to measuring progress in this example is to directly compare m1 and m2, the number of clicks measured during a period of time of length T before and after deploying the new feature. We argued in our previous post that this approach is essentially wrong, since it doesn't acknowledge the fact that we are measuring things subject to randomness.

The fallacy of the naive approach is to assume that m1 and m2 are equal to N1 and N2, respectively. Fortunately for us, Probability Theory teaches us exactly what to do in this situation. Variables N1 and N2 should be considered to be random variables (think of two dice) and m1 and m2 their samples (think of the outcome of throwing each die once). We have to be careful when concluding that N1 > N2 just because m1 > m2 and take a look at the next section.

Measuring Success in your Lean Startup

"The Lean Startup provides a scientific approach to creating and managing startups and get a desired product to customers' hands faster"
Source: Principles of the Lean Startup
 I'm a big fan of the Lean Startup methodology. It proposes a simple Build-Measure-Learn process that allows you growing your business with maximum acceleration. This process should be executed in cycles and the main goal of a Lean Startup is to to reduce the time it takes to go through a whole cycle.

 The Lean Startup focuses on learning how to build a sustainable business. This learning has to be validated scientifically, by running experiments that test each element of the startup's vision. The results from these experiments are used to validate the experiment's hypotheses during the Measure step of the Build-Measure-Learn process.

The Measure step is the key to the Lean Startup process. Without a sound experimental design the lessons we learn from our measurements could be completely wrong. This observation applies not only to Lean Startups, but to any startup that uses some sort of metrics to measure and track its progress.

Enhancing Page Search Engines

Web search engines can deliver a list of relevant results within milliseconds. We know that the information we are looking for probably hides behind one of these search results, but the problem is how to extract it as quick as possible.  In our previous post we discussed the concept of a page search engine; a tool that basically helps you find information within a given web page.

Your browser's built-in search function is a good example of a page search engine. And it's also probably the only example, at least until browseye came into the picture.

A Page Search Engine: Do you Have to Read Lots of Online Documents?




Modern web search engines have succeeded at solving the underlying technical problems of getting access to an ever-increasing amount of information. Nowadays, we can get search results within milliseconds thanks to web search engines. Unfortunately, this is not the end of the story.

According to a comScore study (see figure below), Internet users spend from 4-5% of their time searching and navigating the Web. That means that around 95% of the time spent on the internet we are basically sitting in front of a web page

 


A Different Way of Developing Browser Extensions

Developing browser extensions can be a daunting task. If you want to release a version of your extension for each major browser in the market, you basically have to develop a different extension for each one of them. Sometimes you'll be able to reuse code (e.g., between Chrome and Safari extensions), and sometimes not (e.g., between Firefox and IE extensions).

Let's present three different ways of developing an extension for different browsers. Each way basically represents a different compromise between the amount of effort you have to put in the development of your extension and the control you get over its functionality. The first two ways are well-known, but I believe the third way is not, so let's compare the three of them.

Developing Browser Extensions: The Fast Way


Developing browser extensions can be a daunting task. If you want to release a version of your extension for each major browser in the market, you basically have to develop a different extension for each one of them. Sometimes you'll be able to reuse code (e.g., between Chrome and Safari extensions), and sometimes not (e.g., between Firefox and IE extensions).


In order to simplify this task you can rely on a cross-browser extension development framework. With this choice you'll be able to release a working extension for all major browsers at the cost of having limited control over what your extension can do. This is what I call the Fast Way of developing a browser extension. You may read about other alternatives here.

Developing Browser Extensions: The Normal Way

Developing browser extensions can be a daunting task. If you want to release a version of your extension for each major browser in the market, you basically have to develop a different extension for each one of them. Sometimes you'll be able to reuse code (e.g., between Chrome and Safari extensions), and sometimes not (e.g., between Firefox and IE extensions).

 If you want to develop browser extensions the way each browser meant you to develop them, then you'll have full control over your extension at the cost of the non negligible effort it will take you to learn each extension development framework. This is what I call the Normal Way of developing a browser extension. You may read about other alternatives here