How to choose the appropriate level of statistical significance for an AB-test

Nowadays a lot of product managers have to confirm most of their decisions with AB-tests. Yet, it is far not always clear how to choose the parameters for the test. A particularly difficult parameter to tune is often the level of statistical significance. If we choose too high level - tests will fail even though improvements do exist. If we choose too low level - we’ll be getting lots of “confirmations” of false improvements.

When we make decisions based on AB-tests, once in a while we’ll be making mistakes. We can limit the losses caused by such mistakes by choosing the appropriate level of statistical significance.

Signup with the phone number

Imagine that you are a product manager in a company running a typical web service.

You are working on the ways for increasing the amount of signups. You examine the signup process steps and notice that many visitors stop when they need to enter their email address. So you decide to try asking for a phone number instead.

Using phone number for identifying the users is not exactly a widely used practice yet and comes with the need to pay for a text message for activation. However, nowadays people aren’t that much into email anymore and do everything on the phone - so possibly phone number-based signups have the potential for bringing in more users.

VS