Python 3 Retrospective

At the beginning of the year, I introduced Pebble’s long-running project to migrate our systems to Python 3. At the end of 2017, how did we do?

Background

At Pebble, we built our product line on Python and Django. With the pending release of Django 2.0 and the biggest frameworks removing support for Python 2, we decided now was the time to start moving to Python 3 in earnest.

To get the migration underway, we knew we needed a way to keep ourselves accountable to our end goals. We started by setting up a monthly meeting within the development team to review our past month’s actions and to set actions for the next month to complete.

Due to our size as a business and a dev team, we knew that we’d be unable to stop product development to just update our systems to Python 3. This led to us working to update our systems in a more piecemeal fashion.

Review

As with any project, we had our ups and downs. Some things went really well, some things didn’t go so well, and there were some things we simply failed to achieve.

The Good

One of our biggest wins was tracking and improving our code coverage and lint scores across our systems. A major benefit of this is that we’re now able to track these metrics as KPIs to measure the overall health of our development process.

Splitting our codebases out, especially our Fund Manager and Arro products, made it possible to update the systems separately more easily. We were able to perform this split during a weekend — where downtime doesn’t affect our users — with almost no user-facing impact.

Most of our systems are either running Python 3 now or will be updated to Python 3 within the first quarter of 2018. Constant work on this throughout the year ensured that, by the end of December 2017, we only needed to spend a couple of days each working through our remaining codebases to get tests passing in Python 3. With 90% coverage across our systems, we have a good level of confidence on our migration in the next stage.

We tracked the list of actions we needed to complete to make the migration to Python 3 and, where possible, married this up to our product roadmap. Doing this gave us a good picture of when things could be accomplished and even allowed us to remove tricky dependencies (i.e. suds) we were projected to no longer need. We assigned each item on this list to an individual who would either complete or oversee its completion.

The Bad

We split our codebases out more than we probably should have. Whilst this made it easier to update individual systems, it also makes keeping the library versions on each project up-to-date more labour-intensive.

Our monthly meetings kept a good track on what we committed to doing but too many items remained incomplete for far too long. I believe part of this was caused by this list being maintained separately from our usual product development roadmap.

The Ugly

One of our systems remains on Python 2.7 and this is unlikely to change in the near future. We will be working across the business in the next 6–12 months to figure out what we need from this system. Using this, we’ll be able to decide whether we maintain and upgrade the system in-place or whether we scrap and rewrite the codebase.

Lessons Learned

This project was pretty interesting and, while not technically challenging, provides some interesting business concerns that a newly-minted project manager or lead developer could trip over. This is the sort of project that has no immediate customer-facing benefits but will improve your product development processes in future.

Communicate at a High Level

Before starting a project like this, make sure the teams that could be impacted broadly understand what you’re doing and why. Keep this high level and in a language your audience understands.

So, when explaining the benefits of this move to your support team, the following isn’t going to help:

The superior exception handling in Python 3 will make it easier to track down bugs.

Instead, take those benefits a step further and tie it to your support team’s goals:

We’ll be able to resolve customer issues faster and to a greater degree of confidence.

Also, be honest about the potential impact! The wider business is there to help you deliver great products to your customers but that can’t do this if you don’t be up-front with what you believe could go wrong. Depending on your company culture, they may even offer to help test, they can communicate with customers in advance and help keep the pressure off.

Keep and Maintain One List

A major factor that slowed our development was the fact we maintained this list separately. I do strongly believe that keeping a copy of this job list as meeting minutes & outcomes was the right thing for referencing back. I think the mistake we made was not adding these actions into our product backlog. This meant discussions between the Product Manager and I weren’t incorporating this necessary work during the planning stages.

What I should have done was move all the tasks into our regular product backlog and maintain the assignees. We can then note the ticket numbers back on the outcomes section of our meeting minutes. Doing this would have meant the regular job list that the development team work from includes the agreed tasks and we would have a clear endpoint for completing the migration.

This doesn’t mean we throw away the meeting minute structure — far from it! Assigning the ticket references to the minutes would have made it even easier to agree completion at the following meeting!

Use Status Update Meetings

When doing long-running projects like this, schedule specific status update meetings. Put this meeting on a recurring Calendar event every week or month and make time for it. If you try to slide it into a general company meeting, or other meetings, you’ll find the project never gains momentum.

Periodically Test Python 3

What your system will look like the first time you test Python 3

While most of the syntax changes are pretty straightforward, your biggest challenge is going to be handling Python’s changes to how it handles strings. You’re going to find code paths that fall over only once you start trying to test them with input. Make the time to run your test suite on areas that interact with the wider world. Make sure you’re running some integration tests here!

Include it in Code Review

Depending on the size of your codebase, this is going to take some time. The rest of the business may not agree to stop feature development while the development teams work on this, so you’ll need to reach a compromise.

On our end, we incorporated the Python 3 project into our product development. To keep the two aligned, we agreed to enforce usage of Python 3 idioms, six etc. as part of our code review process. In short, the reviewer would look at every file edited and look for __future__ imports, str and unicode being replaced with six.binary_type and six.text_type, and a bunch of other changes.

The reality is that, as you code, you’re focused on solving the task at hand and not Python 3. By giving the reviewer permission to look for Python 3 compatibility, we were able to upgrade our most commonly used code very quickly and update some areas that we’d not have otherwise thought to look!

Summary

So how did we do? We still have one outstanding codebase that will miss to Python 3 goal, however our key product lines will be on Python 3 before the end of February. I’d say our approach worked pretty well.

If you’re working on upgrading your codebases to Python 3, let me know whether this helped! If you have any tips of your own, please share them below.