AndroidX Migration: A lesson in Git, Gradle, and Patience
It’s 9pm on Wednesday. I’ve been babysitting unit tests for most of the week. The original goal was to have this merged yesterday at noon.
The tests finish. 3 tests have failed.
I document the failures, fix them, and push the code.
But, in the time it took to fix those tests, a merge conflict has shown up in the main branch.
I don’t advocate working long hours, especially for days on end. Pandora doesn’t either: they encourage a very healthy work-life balance. In this case, though, this was a complex change that touched so many files that virtually every other change made to the main development branch caused a conflict. So, at 1am Thursday I submitted code that finally built and passed all the tests successfully. Success is sweet after a month of work migrating a codebase to AndroidX, including several aborted attempts which required starting over in a new branch; especially with so many lines of code, a large test suite, and dozens of engineers actively pushing code the whole time I’m working.
Let’s rewind to the beginning, two months before that first fully successful build. We needed to migrate the Pandora Android app from the (now deprecated) Android Support Libraries to AndroidX. If you haven’t migrated yet, you should definitely start the process soon. Google has stated that they aren’t adding new features to the old support libraries, which also means they probably won’t release a v29 of Android Support Libraries when the new version of Android is released this year. Before bringing it up to the team at large, I did a simple exploratory migration following the basic instructions from Google. That failed to do a lot of the heavy lifting, so I did some more research and found a great article from Dan Lew about more complex migrations.
With these references in hand, I took about a day to get the app built and running. But piles of new warnings had stacked up, and thousands of tests now failed. In better news, I had proved it could be done! So, I wrapped up the other work I was doing and approached the management team to determine when we should do the migration for real. We targeted a date immediately after an upcoming holiday weekend, when we would cut a release build on Tuesday, merge my code in, and have a minimal code freeze for the other Android developers.
If you’re paying attention, you’ll notice I missed that target. It happens. But, why did I miss it? Because the migration of the app code was just the beginning, and I ran up against a slew of issues, with our tests in particular.
Dan Lew mentions three specific points of failure he found when migrating:
- The Maven coordinates for the AndroidX artifacts were often out-of-date (referring to alpha builds instead of the latest stable builds).
- The Android Studio build-in migration tool did not actually find/replace all the package references that needed updating.
- It made extraneous changes to our codebase — it should’ve only touched import statements, but instead it unnecessarily modified other references as well.
With that knowledge in hand, plus his script, I started working on the migration in earnest about a month before that fateful Wednesday. Immediately, I realized I wanted to use only stable versions of AndroidX libraries. I dug in and found that the root of Google’s Maven Repository for AndroidX handily lists everything in tree form. I started with a fresh branch (again!) and applied the conversion script. Then I extracted a version of the script for Gradle dependencies. If you manage more than one module with shared dependencies, you are probably using common version variables at the project level. I took advantage of this for our migration, simply adding new variables for the AndroidX libraries and using the Maven repo list to ensure we were using the latest stable versions.
At this point the app built, ran, played music, and did all the usual things you’d expect from Pandora. But nearly half our tests were failing. It was time to dig into why. Nearly 75% of the failures were because a couple classes (notably LocalBroadcastManager) are final in the AndroidX libraries, and before they had been open classes. Since we’re using Mockito v1 for nearly all of our testing, the tests weren’t even running because that version can’t mock a final class.
I had solved so many problems, but I quickly realized I was just beginning. Mockito worked closely with PowerMock for v2 to include an opt-in feature to mock these classes. Sounds easy. Upgrade to Mockito 2, run tests, they’re certainly all gonna pass, right?
That opt-in alone fixed nearly half the failing tests. But, if you’ve already upgraded to Mockito 2, you have probably experienced one particular pain: One of the commonly used Matcher functions, any<T>(), went from matching with null to explicitly not matching with null. Suffice to say this caused failures in more than a few tests.
Retry? Continue?
As I worked through all the changes, I was saving patch files for each commit. Keeping a more robust log of each change made me more confident, in the fairly likely event that I’d need to replay them. I also took copious notes on the process, including: files with changes required by the update beyond the imports, missed imports, best library versions, and any place in the code that failed to compile at any point in the process. This turned out to be crucial to the process! Two weeks into the work, I was rebasing my work onto the main development branch, and something went sideways. It broke my branch so badly that I made the decision to abandon that branch and start over. (Again!) But all was not lost in the process. I had those scripts, the commit patches, all my notes, and added knowledge of the steps to take during a rebase.
Fresh branch, fresh perspective. Script all the things, and go! This time I had notes about where the code would need changes above and beyond the scripts, so I quickly knocked those out. Once I got back to where I was before the rebase that I couldn’t resolve easily, I went back to figure out what happened. If you aren’t familiar with git rebase, the short version is: I was re-applying my commits on top of the new commits since I branched.
I found that the failure occurred because several files had been deleted in somebody else’s commit, but because I had edited the imports on those files, git kept my version of those files around. This caused two distinct problems:
- Duplicate Classes. If the class was converted to Kotlin, then there would be a Java version that I had edited, and a Kotlin version side-by-side.
- We, as many Android Teams, are converting our code from Java to the more modern Kotlin.
- Kotlin is more concise, fully interoperable with Java, and adds many developer niceties.
- However, Kotlin compiles to .class files alongside the Java code. So there can be a conflict if both the original Java and the new Kotlin file are present in the codebase. - Invalid references. If the class was deleted completely, then the references could be incorrect all over the place.
The duplicates were easy to manage. The deleted/moved classes were much harder to identify, so I started a new process for rebasing:
- Make note of deleted/moved files in each commit.
- After the rebase, double-check references and delete any dangling classes/files.
Working again, I was fixing tests left and right. Finally under 1,000 failing tests.
But then I noticed that the assemble builds on the build server started failing. Oh no, what have I done? Digging in, the problem was not at all obvious. It was a compilation failure with no exception, no stacktrace, nothing.
Ok, what have I changed that would have broken the app? I’ve only been working on tests! I put Mockito v1 back in, and everything was fine. Ok, good to know… Knowing the root of the issue is tantamount to fixing the issue! But what exactly was wrong with the Mockito 2 dependency? I dug in and hunted the web for an answer of what could be going wrong. I couldn’t find anything. I was stabbing in the dark for answers, trying different combinations of dependencies, all of which we had been using previously, but several needed upgrades. After various attempts, I got a useful bit of knowledge from a Gradle scan. Mockito 2’s dependency on another library was causing a transitive failure. The issue was hidden a few layers deep, but now I was able to find a solution quickly!
We’re now at the Sunday before my hopeful Tuesday merge. You know: the one I’ll miss. Ok, builds are working again. To the tests! As I am cutting them down, I find one last gotcha: an old library was causing a couple packages of tests to fail, because of duplicate classes during dexing, again. This is becoming too easy! I feel like I’m gonna make it! I’m fixing tests, rebasing, getting closer and closer.
Tuesday comes and goes, and there are still ~100 failing tests. The long tail. Wednesday I worked through the last tests. Now we’re all caught up.
Even with all the planning and preparation, it still wasn’t the end of my troubles. I noted that deleted files in a rebase could cause problems; it turns out that so does migrating code to a new module. That fateful 9pm rebase included a merge that created a new module, and several files were now broken. It took three hours to fix, but I made it to the finish line and got all the builds working.
In an effort to make sure I didn’t break…you know, the whole app…we had been branch testing with a dedicated Quality Engineer for two weeks. Before I could merge my code, we needed them to sign off. Thursday I went into the office, and because I’m in Atlanta and the QE resource is in Oakland, all I could do was wait and hope. I announced in our Android developer Slack channel that this change was coming, per my manager’s request, but other developers kept merging code! Each one caused the build clock to restart. My manager and I realized that I was never going to get my branch merged if that kept happening, so management made a call to freeze merges until we had word from QE, and my branch was merged. But I was only given Thursday to make that happen.
The crunch was on! Everything was approved, and the merge build was started mid-afternoon! But mere minutes before it completed, an automated merge back from our release branch caused a conflict.
Ok, one last rebase, get the approvals again, request the merge.
The universe has a sense of humor. Three known flaky tests (in literally the last package to be tested) failed. At this point, you just have to laugh with the universe. Fix the tests, resubmit, get another round of approvals, and merge… and wait…
And now I’m babysitting the tests one more time after dinner. Just waiting to see if all the builds are green…
All the builds finish quickly, except unit tests. My lesson in patience. Finally, the unit test job finishes. I’m a little afraid to look.
Success! A green build, and a merge tag!
I learned a lot from this work. Lessons about git, the processes we have in place for branching and merging, and where those processes can break or not give you the result you expect. Some lessons from diving deep into the dependencies of the app and the libraries we use. If you don’t know how to use Gradle’s build scan feature, look into it, because it will likely save you hours or days at some point. Also, a great lesson in communicating with a team when you are ready to complete a change like this and merge it back to the main code branch. If we have to do any work that has this many moving parts in the future, we will have a more concrete plan when the time comes to merge, and we’ll make sure that the whole team is on board; that way we can move faster, and the interruption is limited.
These issues may have been magnified by the size and complexity of this migration, but they are issues that show up in smaller doses every day. And learning how to efficiently manage them will only improve our team and our workflow!
Have you migrated to AndroidX? What was your experience? Hopefully, you had a better time than I did! And if you haven’t migrated yet, I hope this can provide some helpful insight into managing the process.
