My name is Olga Khomich, I am the head of CRM marketing at MIXIT. An important thing to note about our customers is that those who have an activated loyalty card purchase 1.5 times more frequently than those who do not, which is why we aim to increase this segment. In this article I will share with you: — How we built up this segment, resulting in a 90% increase […]
March 23, 2022
What is A/B Testing and how to do it Properly
A/B testing is an experiment used for choosing the most effective variation, usually two, from several. For example, you can test two variations of an email, a web page, or an online store product page. The test creates a seamless decision-making process, based on solid figures.
You can benefit from A/B testing when:
- Colleagues can’t come to an agreement, where the product owner prefers one variation and the designer and marketer prefer another.
- A costly change needs to be made.
- You need to choose a different service provider from many others that provide similar product recommendation algorithms.
- You hesitate to make some immediate changes that may involve some risk, and are asking yourself questions like, “what if it all goes wrong?”
This article exhibits some success stories, and guides you through the A/B test process step-by-step, highlighting the must-haves for a useful test, explaining how to avoid errors, showcase the tools to analyze results, and showing the value of A/A testing in finding errors.
A/B Testing Examples Valuable for Business
A successful test result, along with an improved conversion, can be seen as a winning variation. With A/B testing, Mindbox clients are guaranteed a measurable financial result. Clients such as:
- The marketer for AM Food & Wine, the chain of Food & Wine supermarkets, discovered that promo codes increased revenues no matter how large the discounts were.
- Furniture online store PM improved their mailing rates by 25%.
- Kari, the footwear retailer, saved 19% on their SMS channel budget.
You may find all these links listed at the end of this article.
Stages of an Experiment
The following are the A/B testing stages, along with a detailed overview of each of them:
- Specify the vector of business growth and choose the right metrics.
- Formulate a hypothesis.
- Decide on a sample size.
- Examine the process of collecting data according to the determined metrics.
- Launch the test and register the results.
Specify the Vector of Business Growth and Choose a Metric
In order to specify the vector of business growth, you need to focus on the business aspects that need to be improved and determine which metrics should be used to measure that improvement. For example, you notice that customers seldomly open transactional emails that are sent after they have placed an order. In another situation, you may want to find out if you benefit from the product recommendations widget in a product card. In these cases, the possible metrics to measure improvement could include:
- revenue
- number of orders
- average order value
- open rate
- repeat purchases
- number of line items in a receipt for an order
Formulate a Hypothesis
Once you have specified the course for business growth, you can determine what specifically needs to be improved. Certainly, a test wouldn’t be very useful without a hypothesis to go on. Normally, a hypothesis contains the expected improvement. For example, your hypothesis can lead you to test add-on widgets, headers, colors, text sizes, forms, and designs. See the examples of our clients’ hypotheses below.
| Hypothesis | Variations | Metrics |
| Open rate increases by 2% with the use of an emoji in an email | Emoji vs no emoji | Open rate |
| The average order value will increase by 10% with a supplementary product widget in a product card | With the supplementary product widget vs without it | Revenue |
| The order conversion rate will increase by 4%, with a free delivery pop-up shown on a website | Pop-up vs no pop-up | Number of orders and revenue |
Determine a Sample Size
To receive a statistically significant result, a certain sample size is necessary every time you run a test. Statistical significance is an estimated measure of confidence in a result, ensuring its validity. A high measure of confidence is important because any random coincidence could be considered successful should statistical significance not be taken into account, risking a series of poor business decisions afterward.
Take, for example, that the present email open rate is 20%. Consider that you would like to make a change that would increase that rate to 25%. So, to do this, you will need a sample of at least 2000 people — the A/B test calculator helps to calculate the required sample size. (See more under the header “testing tool.”)
Examine the Process of Collecting Data According to Metrics
Before you launch the A/B test, make sure that your data collection is set up properly to a given metric. For example, if your goal data is set up in Google Analytics, and the experiment is launched in Google Optimize, while revenue data is collected in Mindboxs’ summary mailing report these numbers must relate to one another according to some given metric.
One important suggestion is to try A/A testing, particularly if you suspect that the test results are not representative. This could be that your sample features aren’t showing relevance to the tested variations.
The A/A Test as a Way to Check the Accuracy of Segmentation
An A/A test is a type of experiment that has identical variables. Therefore, there will be some error if the performance of these variables differ in spite of their similarity.
For example, the mistake may lie in the distribution of participants in the experiment. The participants from one segment may buy products more often than those from the other. There could also be a mistake in the process of data collection, where you may lose some data at a certain transfer stage. That’s where A/A testing comes in handy.
Register the Results
Register the results at the end of the test and calculate their statistical significance. The winning result is a statistically significant variation that costs less or brings in more money. Use an A/B testing calculator to compute the result. (See more below.)
Tools to Set Up Tests
Let’s have a closer look at the A/B test tools:
A/B Testing Calculator
Use our free A/B Test Validity Calculator to calculate the necessary sample size for a statistically significant experiment and for a test result summary. Enter the figures from your experiment to find your results.
RFM-email with a discount promo code, sent every 90 days
Google Optimize
Google Optimize is a free tool offered by Google used for testing websites. You can set up several variations and launch testing. The tool is applied alongside Google Analytics that acts as a source of information for such data as revenues, volume of transactions, and more.
How to Set Up A/B Testing in Mindbox
In Mindbox, you can test mailings, promotions, and recommendation algorithms. A Mindbox team then sets up the test processes for you. In order to do this, you need to decide on a target action, sample size, and a desired conversion growth rate.
Summary
To confirm that your proposed changes would be efficient and to run a top-notch A/B test, you’ll need to:
- Specify in which direction the growth of the business is going.
- Formulate a hypothesis. It should include the expected increase in rates according to the required benchmarks.
- Decide on a sample size for testing.
- Make sure that the relevant data are collected properly, and use A/A testing if in doubt.
- Register the results, and use the validity calculator to check their statistical significance.
Useful links
The following case study is from Mindbox, the original brand behind Maestra’s technology