Success stories
January 14, 2022

12 STOREEZ generates 30% of its revenue through direct communications. 6 A/B tests and their results.

  • GoalIncrease the share of revenue from the Abandoned Cart mechanics
  • What we didA/B tested email mailings, web push notifications, pop-ups, and mobile push notifications
  • Launched a survey for inactive subscribers
  • Results+1.21% to the direct communications share in total revenue
  • Participants12 STOREEZ CRM marketer, Mindbox manager
  • Timespan1 month
  • ITCustom online store developed by 12 STOREEZ, Back-office developed by RetailCRM, POS solutions developed by 1C:Retail, and the marketing automation platform
  • Business size44 brick-and-mortar stores, ~1 million customers in the database
  • Distinctive featureThe survey for inactive subscribers generated 905 reviews, 207 detailed responses, and 28 orders

One of the most fast-growing Russian clothing brands

Mariya Vesyolkina

Maria Vesyolkina, CRM Marketer, 12 STOREEZ

How we formulated our hypotheses, calculated the A/B testing validity, and what unexpected conclusions were made. Maria Vesyolkina, CRM Marketer, 12 STOREEZ.

Why Many A/B Tests Were Launched

12 STOREEZ applies 5 communication channels: email, mobile push, SMS, web push, and OSMI Cards push notifications. What’s more, marketers send 5 mailings a week. As a result, the customer communication load is high, averaging 35 communications per month (including transactional and service messages).

At the same time, our customers have the freedom of choice, which allows them to choose the topics they like (for example, New Items, Stories, or Sales) on the page dedicated to subscriptions. This lets us keep the Unsubscribe rate at 0.1% per month.

unsubscriptions per month

In August, we noticed a slight drop in customer activity and revenues from the mailings. In response, we audited the mailings and started working on improving the effectiveness of each of them. Six A/B tests were launched to assess this. In the future, we plan to abandon mass mailings and turn to trigger-based chains.

How we Carried out A/B Tests

The A/B Test Validity Calculator is applied to A/B testing. Test parameters are specified in the Calculator (expected absolute increase, number of variations, validity, power, and average rate). This shows the number of participants needed to achieve an informative result. On average, we need 17,000 people to test mass mailings.

The test duration depends on the mechanics being evaluated, the type of the mailing, and whether it is trigger-based or mass. The results of the mass mailing are checked in 3 hours, whereas the trigger-based mailing produces results in 7–14 days. When testing trigger emails in Mindbox, a notification is sent to the mailbox once the necessary amount of messages has been distributed.

We run every test 5–10 times in order to confirm the results.

What Hypotheses Were Tested

Hypothesis 1. The conversion rate drops if there is a product recommendation widget in the abandoned cart message

An Abandoned Cart email is sent if the customer has added items to the cart and abandoned the website. We doubted whether any recommendations should be added to such an email, as well as whether the conversion rate drops once there are too many buttons in the message.

Abandoned Cart
Variation 1. The email contains only a reminder about an item in the cart
Abandoned Cart
Variation 2. The email reminds customers about the items in the cart and proposes other items that may interest the customer and go great with their image

The hypothesis failed.

In this email, product recommendations don’t affect the order conversion rate:

Email variation Order conversion rate
No recommendations 0.8%
With recommendations 0.9%

Hypothesis 2. The click rate increases if the abandoned browse web push notification contains the customer’s name

The abandoned browse web push notification is sent once a customer viewed an item but didn’t add it to the cart and left the website. We were convinced that including customers’ names in a push notification would increase the click rate.

ONameless message
Variation 1. Nameless message
Customer addressed by name
Variation 2. Customer addressed by name

The hypothesis failed.

Including customer names in the web push notification doesn’t affect the click rate.

Email variation Click rate
Without name 2.3%
With name 2%

I was surprised that the use of customer names didn’t have any influence on the click rate. This goes against our general understanding of the mechanics. This result may be due to testing on web push notifications, however, the situation may be different for other channels. In any case, we shouldn’t jump out of the frying pan, and into the fire and delete customers’ names from our communications after only one test. We need to run several tests for different channels.

Hypothesis 3. The open rate increases when message subjects include emojis

We used to add emojis in every email. It was believed that emojis could increase the open rate. So, we decided to run an A/B test to confirm this.

Emojis in the subject
Variation 1. Emojis in the subject
No emojis in the title
Variation 2. No emojis in the title

The hypothesis failed.

The open rate is higher for emails without emojis.

Email variation Open rate
With emojis 11.4%
Without emojis 12.7%

It is possible that users perceive these emails as spam due to the fact that a lot of companies add emojis to their emails. An email without emojis seems to be of more importance.

Hypothesis 4. The shorter the email, the higher the click rate

We send our customers longread mailings with looks composed of the brand’s items. Under every look, there is a description of the clothes, ideas for combinations, and the best occasions to show off these outfits. We decided to check if one look worked better, or if we needed to put together several looks every time.

The email includes 5 outfits
Variation 1. The email includes 5 outfits. The customer simply needs to click the "Choose" button under the photo and a text will open up with a description of the outfit.

The hypothesis was verified.

A short email received almost twice as many clicks.

Email variation Click rate
Longread containing 5 looks 0.8%
Short email including one look 1.4%

Hypothesis 5. The shorter the journey to the target action, the higher the conversion rate

New customers can see the pop-up that collects emails. It suggests the customer to leave their email address to receive a playlist for their morning run. The first group of customers was asked to leave their email addresses straight away. The second group was offered to receive the playlist and only then to leave their email addresses. An extra CFA is the only difference between the pop-up types, whereas the text and picture remain the same.

We ask the customer to add their email straight away
Variation 1. We ask the customer to add their email straight away
We offer a playlist first
Variation 2. We offer a playlist first, then ask customers to subscribe to our mailing

The hypothesis failed.

The pop-up with the extra button generated more contact information.

Pop-up variation Conversion into shared emails
One button 0.98%
Two buttons 2.6%

The A/B test result was a revelation for me. Now I want to check whether it does the trick for other mechanics. If you want to get something from a customer, you should ask for it right away and redirect them to a website or an app. That’s what marketing courses teach us as a rule of thumb. The test, however, proved that this is not always true. The more premium a segment is, the more independent a customer wishes to be.

As a result, this may have influenced the test results. The customers understand what they need, so it’s better to let them choose.

Hypothesis 6. The click rate increases if we feature the product in the "Back in Stock" mobile push notification

When a product becomes available again, a customer receives a "Back in Stock" notification after they’ve signed up for it. This could be an SMS, an email, or a mobile push notification. We assumed customers would find it useful to be reminded of what the products look like. To verify the hypothesis, we sent 2 types of mobile push notifications that contained identical texts.

With a photo
Variation 1. With a photo
No item photo
Variation 2. No item photo

The hypothesis failed.

Customers tap text mobile push notifications more actively. Their click rate is 1.7% higher than that of push notifications that feature a photo.

Mobile push notification Click rate
With a photo 12.5%
Without a photo 14.2%

A/B Test Results

A/B tests helped us discover that:

— the conversion is not affected by the product recommendations widget in the Abandoned Cart mailing.

— the click rate is not affected by inserting the customer’s name in web push notifications.

— emails have a higher open rate if there are no emojis in the subject.

— short emails have a higher click rate than longreads.

— if there are several buttons in a pop-up, customers share their emails more actively.

— text-only "Back in Stock" push notifications have a higher click rate than those with a photo.

Questionnaire for Customers Who Stopped Opening Emails

We sent a questionnaire to the customers who hadn’t opened our emails for half a year. We directly asked them what was wrong. Customers were able to choose one of the five options in the questionnaire. On top of that, they could leave a text comment.

email was sent
The email was sent to the customers who hadn’t opened emails for 6 months
text answers
Open rate
Click rate

It was a surprise to receive 207 detailed answers from customers. They definitely wanted to help improve our mailings. The questionnaire doesn’t redirect to the website or the app. Nevertheless, we generated 28 orders from the segment of customers who hadn’t read our emails for half a year.

"You've Been With us for Half a Year" Trigger-Based Mailing

We are currently preparing to launch a "You've Been With us for Half a Year" reactivation trigger.

In such mailings, we tell the customer how much we have achieved together: how many emails were read, how many orders were delivered, and how many times the website was visited. We were inspired to create this mailing thanks to the Tinkoff and Yandex messages summarizing the year’s results.

The "You've Been With us for Half a Year" notification wasn’t designed to increase sales, but to stimulate customer loyalty, and to build warm friendly relationships with our customers.

You Have Been with Us for Half a Year Already
The number of emails read, orders delivered, and visits made over 6 months is what the customer will learn from the "You've Been With us for Half a Year" message

P.S. In this story, we’ve talked about our Customer Data Platform (CDP). Visit the product page or ask a consultant to learn more about the product.