The Value of Sound Design in Audio Ads

Zhen Zhu
Algorithm and Blues
3 min readDec 1, 2017

--

Zhen Zhu, David Reiley and Kate Hawkins

As the United States’ largest publisher of digital audio ads, Pandora has been making investments in understanding how best to design the creative content of audio ads. Our audio design team has been developing best practices for years, and our science team has begun to conduct careful experiments to measure the impact of these practices.

We recently ran an experiment to measure the effectiveness of our audio advertising with different creative strategies. In this project, we were “eating our own dog food”, running audio ads to advertise our own ad-free subscription products, to measure whether those who receive the ads are more likely to sign up for a subscription than those who do not. We randomized listeners into different treatment groups receiving different ad campaigns. The first group received Pandora Plus audio ads with simple vocal narration. The second group received Pandora Plus audio ads with narration plus sound effects. The third group was a control group, receiving a placebo ad for an unrelated charity organization (Little Kids Rock). These placebo ads enabled us to know who would have received the Pandora Plus ads if they had been in the treatment group. The actual creatives are linked below.

Treatment Group 1: audio creative with simple voice narration.

Treatment Group 2: audio creative with narration plus sound effects.

Control Group: Little Kids Rock placebo ad.

Our outcome of interest is whether a listener signed up for a trial subscription for Pandora Plus. The experiment ran for 11 days in March and April 2017. To measure the impact on consumers, we find all listeners who started a free trial of Pandora Plus, and compare their numbers across treatment groups. We saw that not only do we get increased trials during the campaign, the benefits continue for three weeks past the end of the campaign as well. We therefore present the following results for the probability of a listener starting a trial subscription during the campaign:

Conversion rate by treatment groups

Given the large size of our test (more than four million people exposed to ads per treatment group), we have very precise estimates, with margins of error of approximately ±0.01% (a 95% confidence interval). Thus the observed differences are statistically significant.

Overall, we find that ads with intelligent sound design cause 30% more incremental conversions than ads without sound design.

Why is it important to run an experiment? We see that even the control group has a substantial number of people signing up for Pandora Plus during the campaign, so some people sign up for Pandora Plus even when they don’t receive our audio ads. However, the treatment groups show considerably more new trial subscriptions than the control group. Because we randomized people into different treatments, we know that there are no differences between the groups other than their Pandora Plus ad exposure, and therefore the increase must be caused by the ads. Our audio ads with voice narration increased subscriptions by 50%, while the ads with sound effects increased subscriptions by 65%.

In a future blog post, we use this same experiment to show that standard last-touch attribution models can substantially underestimate the causal effects of audio advertising. Here, we wish to emphasize the value of sound design in improving audio creatives. Pandora’s audio-design team stands ready to help clients create impactful creatives with good sound design.

Huge thanks to Dave Hardtke for reading drafts, editing, and comments.

--

--