12 STOREEZ Generates Over 30% of Its Revenue Through Direct Communications. 6 A/B Tests and Their Results.

At the same time, our customers have the freedom of choice in that they can choose the topics they like (for example, new items, stories, or sales) on the page dedicated to subscriptions. This allows us keep the unsubscribe rate at 0.1% per month.

In August, we noticed a slight dip in customer activity and revenue from email campaigns. In response, we audited the campaigns and started working on improving the effectiveness of each one. Six A/B tests were launched to assess this. In the future, we plan to abandon bulk campaigns and turn to trigger-based sequences instead.

We use the A/B test reliability calculator before launching A/B tests. To determine the number of participants we need, we enter test parameters such as the expected uplift, number of variants, power, the desired realibility score and our current baseline in the metric that we're testing. On average, we need 17,000 people to test bulk campaigns.

The abandoned browse web push notification is sent if a customer viewed an item but didn’t add it to their cart and left the website. We were convinced that including customers’ names in push notification would increase the click rate.

This hypothesis failed.

Including customer names in the web push notification doesn’t affect the click rate.

I was surprised that the use of customer names didn’t have any influence on the click rate. This goes against our general understanding of the way customers respond to communications. This result may be due to the fact that we tested web push notifications and the situation may be different for other channels. In any case, we shouldn’t jump out of the frying pan into the fire and delete customers’ names from our communications after only one test. We'll need to carry out several tests for different channels.

We used to add emojis to every email's subject line – we believed that this could increase the open rate. So we decided to run an A/B test to confirm this.

This hypothesis failed.

The open rate is higher for emails without emojis.

It is possible that users perceive these emails as spam due to the fact that a lot of companies add emojis to their emails. An email without emojis seems to be of more importance.

We send our customers long emails with a selection of outfits created using our brand’s items. Under every outfit, there is a description of the items, ideas on how to mix and match items, and the best occasions to show off these looks. We decided to test whether showcasing one look at a time works better or if customers prefer a selection of several looks.

This hypothesis was confirmed.

The shorter email received almost twice as many clicks.

New customers are displayed a pop-up that collects email addresses. They are asked to subscribe and get an email with a playlist for their morning run. The first group was given the option to enter their email address straightaway. The second group was offered the playlist first and only then asked to leave their email addresses. An extra CTA button is the only difference between the pop-up types, whereas the text and image remain the same.

This hypothesis failed.

The pop-up with the extra button generated more contact information.

This A/B test was a revelation for me. Now I want to see whether the same principle applies to other pop-ups and campaigns. As marketers, we're taught that if you want to get something from a customer, you should ask for it right away and redirect them to your website or app. The test, however, proved that this is not always true.

The more "premium" a segment is, the more independent a customer wishes to be. These customers understand what they need, so it’s better to let them choose. This may have influenced the test results.

When a product becomes available again, a customer receives a "Back in Stock" notification after they’ve signed up for it. This could be a text message, an email, or a mobile push notification. We assumed customers would find it useful to be reminded of what the products look like. To verify the hypothesis, we sent 2 types of mobile push notifications that contained identical texts.

This hypothesis failed.

Customers are more keen to tap text-only mobile push notifications. Their click rate is 1.7% higher than that of push notifications that feature an image.

A/B tests helped us discover that:

— the conversion is not affected by the product recommendations widget in the abandoned cart email

— the click rate is not affected by inserting the customer’s name in web push notifications

— emails have a higher open rate if there are no emojis in the subject

— short emails have a higher click rate than longreads

— if there are several buttons in a pop-up, customers are more willing to share their emails

— text-only "Back in Stock" push notifications have a higher click rate than those with an image

We sent a questionnaire to customers who hadn’t opened our emails for half a year, asking them what was wrong. Customers were able to choose one of the five options in the questionnaire. On top of that, they could also leave a comment.

It was a surprise to receive 207 detailed answers from customers. They definitely wanted to help improve our campaigns. The questionnaire doesn’t redirect them to the website or the app. Nevertheless, we generated 28 orders from the segment of customers who hadn’t read our emails for half a year.

P.S. In this story, we talked about our Customer Data Platform (CDP). You can read about this module on our website or talk to someone from our team to learn more about the product.