How much money do messages actually generate?

5 Apr ‘17

There are a lot of ways to measure revenue from messages. The best way is to use control groups. We’ve implemented this feature in our product.


  • how to measure revenue from messages
  • why we chose this measurement tool and how it differs from GA
  • report details and quality assurance
  • what’s next

How to measure revenue from messages

We’ve added a new function that provides an objective measure of the revenue that all messages sent through Mindbox are generating: enabling a global control group. This approach has long been used in medicine and scientific research, and has now begun appearing in data-driven marketing.

How to enable a control group

Here is how to start measuring message impact on our platform:

Control group

When you start measuring message impact, 5% of your entire customer database (but not more than 110,000 people) will randomly be included in the control group each month. So customers will rotate into and out of the control group every month, and no customers will stop receiving messages for a long time.

You can also choose which clients to include in the control group and which to exclude by simply using the «Message impact measurement group» filter.

Got a control group? Here’s a report

We’ve created a special report in the dashboard: «Message impact. Turnover, conversions and purchases» («Admin – System settings – Message settings – Message restriction settings»):

Subscribe to report
 Subscribe to report

What the report shows

  • revenue share from all messages sent through Mindbox (in all channels, online and offline) and average monthly revenue for the selected period
  • rebuy rate
  • control group quality (to make sure that the customers in the control group did not receive the messages)

How to generate the report

All you have to do is select the reporting period and press «Generate». You can also subscribe to a report (press «Subscribe») to receive data at the necessary frequency by email (we suggest subscribing to monthly reporting, so that you can view the entire month at once).

Why we chose this measurement tool

Why control groups?

A control group is a subset of customers who receive no communications, so that you can see what they do when you do not influence their behaviour. Meanwhile, the rest of your database (we call this the main group) receives all messages.

CG ctrlg

The control group is an indicator of customers’ natural behaviour. This lets us know how many purchases would have been made if no messages had been sent at all.

Each month, you can compare the revenue from the main group with that from the control group. The incremental revenue from the main group reflects the impact of your messages.

Detsky Mir, Eldorado, Kari and many other retailers already know how much money their messages generate. Do you?

Won’t I lose the revenue from the 5% of my database that doesn’t receive messages?

This is the only way to measure message impact.

Imagine this: you’ve got a great site, awesome service, and customers really like your offerings: most of them are ready to purchase without additional stimulus. Are your messages generating additional revenue, or are the customers coming on their own and spending a lot? You can only find out by comparing with a control group.

Is a 50% rebuy rate good, or is there room for improvement? Don’t just guess, find out for sure with a control group.

This 5% is an investment in your communication strategy.

Google Analytics shows revenue and email share, why do I need something else?

The Google Analytics revenue data have several shortcomings that the control group method eliminates:

  • revenue can be overstated because of cannibalisation from other channels
  • revenue can be understated because the impact on offline sales is not accounted for

More about the report

You can measure:

  • message revenue share trends
  • average monthly revenue and revenue shares for the period
  • rebuy rate from all purchases made during the period

A very important indicator is «the probability that messages are having an impact». If it’s below 80-90%, then your messages are likely making zero impact. It’s vital to pay attention to this when analysing your results.

For reference, the vertical lines show the revenue share confidence interval (that is, the range of revenue share values). The green columns show the most likely revenue figure (the probability indicator, or «the probability that messages are having an impact»), which is worth taking a look at in your analysis.

Mailings revenue rate

You can also check the control group quality. There are two ways to do this:

  • view the number and share of messages sent to the main group and control group on the «Control group quality» tab
  • export the «Segment comparison» report

Future plans

Here are the planned updates to the report:

  • conversion to 1 and 2 purchases in comparison to the control group
  • lifetime value and churn
  • retention rate


We believe that using control groups as a measurement tool is an important and prudent step forward. This metric depends only on message quality, not on seasonality, different channels or other external factors. If you’ve managed to deliver growth in this metric, it will reflect in your overall revenue. We really want this approach to spread and will do all we can to help our clients boost their net profit from messages.

We’d be happy to discuss any questions or ideas regarding measuring revenue using control groups, write us at

Tell us a little about yourself

We’ll respond within 24 hours

Partnership request

Typically we’re answering within 24 hours