A core component of Spotify’s value proposal is personalization: They want to ensure that the content you receive is tailored to you based on the music you love. In this study, we partnered with Spotify to help them decide what would be the best algorithmic DJ to deploy for each user. Our solution consisted of using machine learning on data from a massive-scale experiment to predict what DJ would cause each user to listen to the most songs. Our findings show that choosing DJs based on this approach can increase the number of song streams by up to 3.7%, compared to simply choosing the DJ that performs best on average for everyone. The intuition behind the result is simple but powerful: different DJs work better for different users, so results are better when we use experimental data and machine learning to assign DJs in a personalized manner.

Our research also looked at three broad categories of algorithms to choose what DJ to deploy: The Outcome Learner (O-learner), which learns to predict the outcome of each DJ; the Effect Learner (E-learner), which learns to estimate the causal effect of each DJ; and the Assignment Learner (A-learner), which directly learns which DJ is more likely to have the best outcome. These general types of learners may look the same at first glance but actually differ in two important ways.

First, they differ in their level of generality. O-learners are the most general (and therefore useful for multiple purposes) and A-learners are the least general—only useful for predicting optimal DJ assignments. Second, the algorithms vary in how they learn. Specifically, when O-learners and E-learners attempt improvements based on predicted outcomes and effects, the improvements may occur at the expense of worse DJ assignments. Therefore, in our setup, we actually found that the A-learner produced the largest number of streams.

Overall, our study shows how large experimental data can provide substantial value for personalization, particularly when algorithms are specialized for the task at hand.