We have previously demonstrated an advantage in messaging impact (increased user retention) by sending messages at optimal “delays,” or amounts of time after a user was last active: http://www.dataeagle.tech/articles/timing-study A message sent too soon after a user was active can annoy or do little to remind them while an app is already fresh in their memory. On the other hand sending a message after users have lost interest or have uninstalled the app would be too late.
In this case study, we demonstrate an additional layer of customization to messages: individually optimized timing by user segments. Different types of users respond best to different messaging times, so a better optimization is possible by testing different message delays per segment of users.
A customer asked us to optimize the timing of a message alerting users to a new feature. Segment-specific timing more than tripled the effectiveness of a message versus sending that message with no optimization, and increased the effectiveness versus finding a best time for all users without segmentation by 48%.
Finding Segments of Users
Traditionally, segments of users are defined manually, based on convenient variables like demographics or geography. These types of segments are useful for some questions specific to those variables. For example,when asking “Should we expand our company into Canada?” geographic user segments will be very useful.
When we don’t care about any particular subset of users, however, and simply want to divide all of them into some effective groups as naturally as possible, a systematic algorithm can yield more powerful segments. Automatic algorithms are not influenced by human biases or (possibly incorrect) assumptions. Automatic algorithms can also cheaply and easily update themselves on a regular basis, even daily or hourly, as an app’s user base slightly changes and evolves over time.
Many segmentation algorithms exist, from the simplest linear cutoff (“All users over age 30”) up to complicated deep neural networks. In this case study, we used an intermediate complexity algorithm called “k-means clustering.” The algorithm plots users in a multi-dimensional space for any number of measurement variables like amount spent, time zone, or session frequency (how active a user is on a daily basis). It then identifies natural clusters in this space.
Visualizing clusters in 5 or 6 dimensions is almost impossible, but below is an example of simplified imaginary data using only two measurement dimensions: spending and time spent in app. We applied our actual algorithm to the data, and the results are on the right:
The number of segments is also automatically chosen by the algorithm. In this case, it chose three segments which match the clusters that seem to naturally exist on the left. None of these clusters can be defined purely by a linear cutoff, and the boundaries of the cluster may not have been obvious intuitively to humans manually defining segments.
In our case study, the algorithm used six user measurements: language, best time to contact, spending, time zone, session count, and device model. It automatically chose four segments, none of which could be simply defined by one variable (similar to the above example), but which represented natural clusters in the userbase. We next optimized best time to message for each of these four segments.
Experiments and Results
After determining user segments, we ran a battery of A/B tests, trying messaging at various delays (amount of time since last user activity) from 1 to 18 days. This can be done almost instantly, since some users already exist at each of these delays. Every A/B test consisted of a group A (non-contacted users) from that segment and that delay, and a group B (contacted users) also from that segment and that delay.
Our outcome measure was daily retention rate improvement, or impact. Specifically, retention impact is the difference between the average daily retention rate over the first week after messaging in group B minus the average daily retention rate over the first week without messaging in group A, for each A/B test.
On smaller segments, fewer delays were tested. Too many experiments in a small segment does not leave enough users after the experiment to benefit from knowing the best delay.
We chose the best delay for each segment, with a small bias toward earlier delays (for faster turnaround of user messaging) when impacts were similar between different delays.
The following charts show retention impacts for users across segments. The chosen best delay is circled for each segment. These delays provided the advantages mentioned above: a tripling of impact versus a non-optimized message (sent to all users at once), and doubling of impact versus timing optimization without segments.
Many messaging variables can be optimized in increasing user retention or revenue, such as timing of messages, content of messages, promotions or other app content supplementing a messaging campaign. Like timing optimization, segment-specific customization is one method that can be fully automated by DataEagle, and it further increases retention impact. This method does not require creative content from developers or resubmission of a new version of an app.
DataEagle is a plug and play solution that automatically finds the most effective set of options for messages, sends them out, and measures your resulting ROI.
Check out www.dataeagle.tech and sign up for free to start improving your key growth metrics.