Triple message impact with optimum timing

As part of our work with one of our early customers, we optimized the best timing of messages to increase user retention rates.

Message timing consists of two factors

1-  The time of day to send out a message. A simple and often best solution is to only send out messages to individual users in the same hours that they usually interact with the app.

2- The second issue, which is usually ignored, is how many days to wait to send a message since a user’s last interaction with an app or after they have triggered key events. We’ll call this delay.

Here we’ll take a look at delay and show that it is a very important factor in boosting the effectiveness of your messages.

Why is delay important?

The intuitive answer is that If the message is sent too late, it won’t be effective, since the customer has lost interest by then and will not come back even with a message.

On the other hand, if the message is sent too early, then it also won’t be effective, since the customer would have returned anyway and the message feels like spam to them, having little effect or even an adverse effect on retention.

As we analyze the results, you’ll see that the data backs up this intuition.

The problem with delay is that it’s a lot harder to guess at the right answer as there’s no easy way to predict its impact.

Case study context

The app we were working on was a freemium social game. The game is available for both iOS and Android devices and has had more than 2 million installs. The app’s average number of unique, active monthly users is about 120,000. Like most apps, it suffers from low retention rates. The name of the app and company have been anonymized so we can share real metrics with you.

In this case, the message being sent was simple; it told players that they’d received some free virtual currency in their account, and that new features were being added to the game.

Difference in delay results

We wanted to measure the effectiveness of our message in changing customer behaviour and increasing retention. To do this we set up control and test groups.

We sent messages, and compared people in the test group with people in the control group, matched by how long we’d waited since their last interaction with the app. The difference between groups indicates the increase in retention attributable to the messages.

 
 

Figure 1: Vertical lines represent inconsistency between users for a given delay. The curve is a possible pattern to the data underneath random noise.

As shown in Figure 1, the maximum effectiveness of the message was for customers that had not played for around five days (peak of our fitted curve, which attempts to ignore some random noise in the data) before receiving the message. Shorter delays and longer delays tended to lead to lesser retention benefits in this case, matching our intuitive hypotheses.

According to the data, the effectiveness of messages sent out immediately was 5% whereas those sent out on the optimal day had more than 15% effectiveness. That’s a tripling of impact!

Performance in different segments

Once we found the overall optimal contact time, we decided to investigate this effectiveness in two subgroups of players. Different groups of users can be targeted more precisely using additional information about them. In this example, we simply split users by the median in-game level of achievement, so that we had a low-level half and a high-level half, although more advanced methods can be used, such as clustering algorithms, multiple variables, etc.

As you would expect, higher level users had higher retention rates even before receiving a message (55% vs. 33%, not shown in the graph), since they were the select few who’d found value in the application and had remained loyal customers for months.

Lower level users benefitted more from messaging, however, as shown in figure 2. Perhaps due to higher level users already being more internally motivated to stay with the app:

 

 
 

In this case, messaging all users was acceptable and never appeared harmful, but in other situations or for other apps, messaging may only be worthwhile with a large enough expected retention benefit, and targeting the most efficient segment (like low level players here) for messaging may be optimal.

In a machine learning context, meaningful groups of users can be generated automatically by clustering algorithms, and hundreds of these analyses can be run to find the most efficient groups for messaging or other interventions to increase retention or revenue.

Closing

To summarize, we see that messaging in general can have a strong influence on user retention, but that it is critical to consider the delay of a user to maximize the benefit.

More specifically, we see that messaging has optimal timeframes. Too soon or too late can decrease efficiency, and careful analysis can uncover the best moment to send a message. As shown in Figure 1, a message sent out after an optimal delay of 5 days versus a suboptimal one day could triple the effectiveness of your messaging.

About DataEagle

DEagle is a plug and play solution that automatically finds the most effective set of options for messages, sends them out, and measures your resulting ROI.

Check out www.dataeagle.tech and sign up for free to start improving your key growth metrics.