Keeping Pace with New iOS Releases

How We Updated Pandora on iOS 12 Launch Day

Photo Credit: Stavros Constantinou

The Story That Shook the Press

The Pandora app was amongst the very few enterprise apps that successfully released an update for iOS 12 on Apple’s day one September 21 launch date, supporting the exciting new Siri Shortcuts feature.

Here are some notable quotes:

Engadget, “Music app Pandora is taking advantage of Shortcuts at iOS 12’s launch. Siri can learn your music preferences and suggest your most-listened-to Pandora stations, but you also can take the reins and add a custom Siri phrase to listen to your favorite artist, album or playlist.”

TechCrunch, “A new playlist isn’t the only update Pandora is rolling out today — it’s also one of the first apps to launch a Siri Shortcuts button. With the app’s iOS 12-optimized update, users can head to the Settings in the Pandora app and tap ‘Add to Siri.’ They can then choose a specific station, album, or playlist and record a custom phrase to say the next time they want to hear it.”

Apple Insider, “Finally, you can use Siri to play your favorite Pandora stations. These shortcuts can be set up for a specific station, album, or playlist. Side note: Pandora is gearing up to launch their brand new Apple Watch app!”

App Informers, “This is one of the most popular music streaming apps available and the new features really add to the iOS app. If you have already upgraded to iOS 12, then you should definitely check out Pandora. There is so much integration here with iOS 12, including all of the new Siri Shortcuts and more. It is so cool that you can now ask Siri to play your favorite artist, playlist, or even music station. These are all things that are going to make the Pandora iOS app easier to use and more hands-off than ever before.”

It’s a Southern Thing, “Siri is playing a lot nicer with Pandora. So, when you holler ‘Siri, play Patsy Cline Radio,’ she knows just what to do!”

Neowin, “Best of all, you can invoke these (Siri) shortcuts using your voice. Pandora is now equipped to take advantage of this, enabling users to access their ‘Recently Played’ tunes using Siri.”

Grinding in the Stone-Age

Getting to this point has been an exciting journey. A year ago, our macOS infrastructure consisted of aging Mac Minis and Mac Pros, and it was cumbersome for our Tech Ops, SRE and NetOps teams to host and maintain this infrastructure. Provisioning new macOS hardware inside our enterprise data centers is also a time-consuming (sometimes months long) and painful exercise that the Tech Ops org has little capacity to address.

Here’s what our mobile CI architecture looked like:

As you can see, we relied chiefly on VMware ESXi VMs for our builds and the dual-core Minis for running our automation tests. The 2014 quad-core Minis had already been discontinued by Apple.

The Road to Evolution

We required a solution which could provide us macOS servers in the cloud (internal or external) and provision them quickly on-demand based on our capacity needs. We achieved this by employing Anka Build software from Veertu Inc. to build and configure a macOS private cloud, running on Mac hardware hosted by MacStadium. We started by conducting a nearly half-year-long test of running our mobile iOS builds and unit tests on hosted Macs at MacStadium.

These hosted Macs solved our challenges of capacity demands, but they didn’t resolve the issue of provisioning our macOS images onto those servers in a scalable and consistent manner, which still took us weeks on-prem. The challenges faced were:

1. Provisioning through ESXi (an enterprise-class hypervisor developed by VMware) is not easily automatable and has a single point of failure due to shared storage.
2. The 2013 Mac Pro architecture is less performant than that of the Mac Mini or iMac Pro, so ESXi virtualization would result in degraded performance.
3. Nested virtualization is not supported in VMs created through VMware ESXi, so iOS UI tests and unit tests don’t run.
4. Environment rollback and horizontal scaling become an ongoing challenge.
5. It is not possible to support mono-repos or branches that require disparate configurations.
6. Environmental changes led to build instability.
7. Provisioning is hard due to network constraints, despite using Ansible.

At this time, we initiated a proof of concept with Anka virtualization, a brand new Docker-like service for macOS that can “install” an image in seconds, as well as switch between various stored images effortlessly.

Infrastructure on Demand

Environment rollback became as simple as provisioning an on-demand environment from a different image or previous image tag/version.

We were now able to create, manage, maintain and distribute multiple build environment configurations from a central Anka Registry. Then, from these primary build environment images, we used the Anka Build controller and Jenkins plugin to enable our build jobs to provision macOS environments on-demand on our Mac hardware. So, while we were maintaining the Pandora app on Xcode 9, we also started development build and test on environments provisioned from an image containing Xcode 10 on Anka Build cloud. We were able to get the Pandora app ready for the iOS 12 launch on very short notice. Developers used Anka Flow to build and test on an Xcode 10 Anka VM image locally, resulting in consistent environments across all stakeholders. Environment rollback became as simple as provisioning an on-demand environment from a different image or previous image tag/version.

Build instability was completely eliminated by executing build jobs in the immutable, sterile environments provisioned from an image stored in the central registry. Anka Build caches VM images on-demand on the host nodes and manages host storage availability, so there is no need to have VMs provisioned and served from a central ESXi shared storage.

The turnaround time for new hardware on MacStadium is extremely fast; half a dozen new machines can be stood up in a matter of hours.

We were able to completely abstract build and test environments from the underlying host configuration and scale simply by adding additional Mac machines hosted by MacStadium. The turnaround time for new hardware on MacStadium is extremely fast; half a dozen new machines can be stood up in a matter of hours.

Solving the conundrum: No Machine is alike

Next, it was time to turn our attention to the performance of the build/test farm. Tests determined that on-prem versus MacStadium per-server performance was for the most part on-par. We made the decision to go with faster hardware to get better performance. By running several performance tests, we discovered that the new 2017 iMac Pro was head and shoulders better than any other choice (note: 2018 Mac Minis were still not released at the time we conducted our bake-off); so much so, that the gains in productivity far outweighed the capital expense for these high-end servers.

We chose to procure the hosted iMac Pros from MacStadium, based on business justification analysis of cost/effort associated with the housing, managing and maintaining the hardware in our data center. Also, MacStadium boasts proprietary (patents-pending) technology for racking iMac Pros in a data center, which was not achievable in-house.

Architecture that scales

Here is what our new mobile build and test architecture looks like today:

Happy developers build solid products

Pandora is now on the leading edge in high-performance macOS cloud computing with Anka Build and hosted Mac hardware.

By performing CI build and test on the new iMac Pros using Anka Build, overall developer productivity nearly doubled, from ~39 minutes to turn around a build and test on the 2012 dual-core Mac Minis to ~22 minutes on the iMac Pros, running multiple builds in parallel. Considering the number of developers and number of builds they run, this will save a significant amount of developer-hours per year.

Pandora is now on the leading edge in high-performance macOS cloud computing with Anka Build and hosted Mac hardware. We can’t wait to see what great new iOS features we’ll be able to support in September 2019!

Pandora is hiring! We have open positions in Engineering, Product Management, Technical Program Management, and Data Science.