The following are the A/B testing stages, along with a detailed overview of each of them:
- Specify the vector of business growth and choose the right metrics.
- Formulate a hypothesis.
- Decide on a sample size.
- Examine the process of collecting data according to the determined metrics.
- Launch the test and register the results.
Specify the Vector of Business Growth and Choose a Metric
In order to specify the vector of business growth, you need to focus on the business aspects that need to be improved and determine which metrics should be used to measure that improvement. For example, you notice that customers seldomly open transactional emails that are sent after they have placed an order. In another situation, you may want to find out if you benefit from the product recommendations widget in a product card. In these cases, the possible metrics to measure improvement could include:
- number of orders
- average order value
- open rate
- repeat purchases
- number of line items in a receipt for an order
Formulate a Hypothesis
Once you have specified the course for business growth, you can determine what specifically needs to be improved. Certainly, a test wouldn’t be very useful without a hypothesis to go on. Normally, a hypothesis contains the expected improvement. For example, your hypothesis can lead you to test add-on widgets, headers, colors, text sizes, forms, and designs. See the examples of our clients’ hypotheses below.
|Open rate increases by 2% with the use of an emoji in an email
||Emoji vs no emoji
|The average order value will increase by 10% with a supplementary product widget in a product card
||With the supplementary product widget vs without it
|The order conversion rate will increase by 4%, with a free delivery pop-up shown on a website
||Pop-up vs no pop-up
||Number of orders and revenue
Determine a Sample Size
To receive a statistically significant result, a certain sample size is necessary every time you run a test. Statistical significance is an estimated measure of confidence in a result, ensuring its validity. A high measure of confidence is important because any random coincidence could be considered successful should statistical significance not be taken into account, risking a series of poor business decisions afterward.
Take, for example, that the present email open rate is 20%. Consider that you would like to make a change that would increase that rate to 25%. So, to do this, you will need a sample of at least 2000 people — the A/B test calculator helps to calculate the required sample size. (See more under the header «testing tool.»)
Examine the Process of Collecting Data According to Metrics
Before you launch the A/B test, make sure that your data collection is set up properly to a given metric. For example, if your goal data is set up in Google Analytics, and the experiment is launched in Google Optimize, while revenue data is collected in Mindboxs’ summary mailing report these numbers must relate to one another according to some given metric.
One important suggestion is to try A/A testing, particularly if you suspect that the test results are not representative. This could be that your sample features aren’t showing relevance to the tested variations.
The A/A Test as a Way to Check the Accuracy of Segmentation
An A/A test is a type of experiment that has identical variables. Therefore, there will be some error if the performance of these variables differ in spite of their similarity.
For example, the mistake may lie in the distribution of participants in the experiment. The participants from one segment may buy products more often than those from the other. There could also be a mistake in the process of data collection, where you may lose some data at a certain transfer stage. That’s where A/A testing comes in handy.
Register the Results
Register the results at the end of the test and calculate their statistical significance. The winning result is a statistically significant variation that costs less or brings in more money. Use an A/B testing calculator to compute the result. (See more below.)