Highlights Testing communication chains helps determine the most efficient ways of interacting with the customer Previously, in order to test the workflow, it was necessary to add a flow split node, then unload actions and manually count statistics for each branch. Now all these actions can be completed via the new A/B test widget: split the branches and select the main test […]
Beethoven’s Story: Using a Neural Network to Forecast Best Sending Time and Reactivating an Invalid Database
Increase online store revenue
Switch to a new CDP
ERP, Mindbox marketing automation platform
Target revenue was exceeded by 30% and reached the pre-CDP switch level
Time to value.
The Next Best Action algorithm helped determine the best sending date for emails, which then became more effective than trigger-based campaigns
September 26, 2022
The following case study is from Mindbox, the original brand behind Maestra’s technology
- What triggered the decision to switch CDP;
- How the neural network’s LSTM model (the Next Best Action algorithm) increased the mailing quality, making them more efficient than trigger campaigns and helped increase mailing revenue by 8.5 times;
- How to segment the database, optimize bulk campaigns, and what to do with invalid email addresses;
- Future plans for improving the channel.
Results after Switching to the ew CDP
Target revenue was exceeded by30 %and reached the same level as before the implementation
The CDP now contains2.5 xmore valid customers thanks to the unified database and data cleansing
Revenue from emails increased by8.5 xwith the Next Best Action algorithm (based on last-click attribution in A/B tests)
The brand now sends23 xmore targeted emails (compared to trigger emails), aimed at boosting repeat purchases with the Next Best Action algorithm
Why Beethoven Decided to Switch to Mindbox
Beethoven decided to start developing direct marketing a year ago. The existing CDP could not ensure mailing growth because it didn’t allow for workflows to be customized or processes to be automated. For instance, segmentation was set up manually because it was impossible to change the selection parameters for product recommendations, and the overlapping of trigger campaigns wasn’t tracked. To better understand the needs of their customers and manage the settings themselves, Beethoven decided to switch to Mindbox.
At some point, I realized that if we want to grow, we need better partners, so we decided to switch to Mindbox. We’re launching the app soon, so I’m interested in working with recommendations for the app, and sending push notifications and triggers, among other features. It’s much easier to set it all up in Mindbox, and I’m sure we’ll be able to sort it all out simply using the platform.
So far, we have moved standard triggered flows like “Abandoned Cart” and “Abandoned Views, ” taken a look at how they work and set up triggers, time intervals and a priority sequence. I am grateful to the Mindbox manager who supervised these settings. The only thing that remains for us is to see how the recommendations work.
Before the switch to Mindbox
After the switch to Mindbox
There was no understanding of how workflows and algorithms worked
Transparent workflows that can be changed and customized according to the company’s needs
Pre-configured segmentation conditions, manual list uploads
The conditions and attributes for segmentation are set by the company; the platform runs segmentation and sends campaigns
Pre-configured algorithms with recommendations meant that it was impossible to analyze their operation or change the settings manually
Selecting products and recommendation algorithms is now possible. For example, if the email conversion is low, it’s possible to change the product and display only discounted or new products
Trigger Chain Management
The system didn’t track whether emails from different trigger chains were duplicated and still sent a similar email
If an email is sent to a customer and is duplicated in another trigger chain, a similar email won’t be sent
It was impossible to track the true effectiveness of the mailing because the system only counted clicks in the email itself
The platform attributes the customer’s purchases to the mailing list if the customer has made no other actions for 30 days. For example, if a customer received a mailing, and a couple of days later purchased the product directly through the website, the platform will attribute the purchase to the mailing
Using a Neural Network to Help Forecast the Best Sending Time for an Email with a Repeat Purchase Offer
We conducted an A/B test to evaluate the results of the algorithm. Some of the customers received emails based on the average time between purchases; others received emails based on the neural network’s forecast. The significance level of the A/B test results is 95%. The NBA algorithm produced higher results compared to the trigger, which sets a fixed time between email sendings:
23 times in absolute value
17 times in absolute value
Revenue for email sending
1.2 times in absolute value
1.5 times as a percentage
The revenue from the NBA algorithm reached
26.9 %of the total revenue generated by automated campaigns
In February, the NBA algorithm was launched for the entire database and, in just 2 weeks, it overtook all triggers in terms of efficiency and reached 26.9% of the total revenue received from automated campaigns. The only original trigger that remained competitive with the NBA algorithm was the one that sends an email with a promo code 15 days after a customer’s purchase. If the positive results we see from using the NBA algorithm persist, we will consider disabling the trigger with a promo code for a future order and replace it with an algorithm. This step will increase our margins without sacrificing any customer benefits. The customer will still receive reminders about orders.
An email with a “Repeat My Previous Order” button is sent when we predict the customer is about to run out of their previously ordered goods. The Best Sending Time is calculated by the NBA neural network algorithm.
The campaigns for the “Cats” and “Dogs” segments are sent before the “Cats and Dogs” segment, in order to avoid email duplication for those who have several pets.
To reactivate customers who have not made orders for a long time, in addition to new products, we also offer a discount.
Bulk Campaign Optimization
Bulk campaigns are sent
2 timesa month, which helps to save customers who are interested in communication in the database
The mailing list goes either through the entire database at once, or by segments for owners of cats or dogs. Because mass emails are rarely sent and trigger emails are sent only to those who visited the website, the company is able to avoid spamming customers. In addition, it helps gently remind churned customers about the company.
Bulk campaigns are sent by segments depending on the customer’s pet
Trigger campaigns overtook bulk campaigns in December 2020, which signified the growth of the active customer base. The channel grew due to the new CDP as well as due to additional advertising and affiliate programs.
Reactivating Customers with Invalid Email Addresses
The open rate of campaigns sent to invalid emails amounted to
At the same time, it is crucial not to send more than one email a month to invalid customer emails and exclude email addresses that are clearly invalid, such as those with non-existent domain names. Otherwise, it may have a negative effect on the spam status of the sender’s domain.
Developing Beethoven’s Digital Channel: Next Steps
- Increase revenue from campaigns by 15% in 2021;
- Launch end-to-end recommendations on the website and in the app;
- Launch AMP mailing lists;
- Transfer service emails to the platform.