How to best handle large migration projects?

Large migrations often create tension between engineering, product, and leadership in an organization. They are difficult to get right but even more difficult to avoid: technical debt tends to accumulate incrementally over time until it becomes a major liability to development velocity and a large migration is inevitable. A common indicator of too much technical debt is that teams become change-averse: since the system is now too complicated, no-one wants to make changes to it since no-one can predict the effect of a change.

From my perspective, there are three critical things to get right when handling a large migration: 

1 – having an explicitly appointed migration program owner 

2 – securing initial support from senior leadership in the organization

3 – regular progress reviews with the same set of leaders

Without a single fully accountable owner, migrations lose steam very quickly. Depending on the structure and culture of the organization, this owner can be a senior engineer, a program manager, or an engineering manager. The owner is usually responsible for securing initial support for the migration from senior leadership but should always be responsible for making sure that there is continuous progress on the migration until it is fully complete.

Why is it critical to secure support for the migration from senior leadership? Well, migrations tend to take much longer than initially expected. So unless senior leadership recognizes the strategic importance of a large migration, work on the migration tends to be prioritized below development of new features or products and the migration runs even slower or grinds completely to a halt. The only way, in my experience, for an engineering team to secure support from senior leadership in the first place is through a very clear narrative around the business value of the migration. For instance, if the migration enables building new features in a few weeks instead of the multiple quarters it takes now, that could be a narrative that resonates with senior leadership. Or if the migration removes modules that have been responsible for multiple recent customer-facing outages, that could be a strong narrative. Having an initial review with senior leadership acts as a forcing function on the engineering team to think through what the value of the migration actually is and to consider potential alternatives to a large migration.

Now, once the large migration is underway, regular progress reviews with senior leadership helps leadership recognize that progress is being made, but it also acts as a forcing function on the team to think through their priorities. Often, a large migration is not an all-or-nothing event but rather a sequence of smaller migrations of parts of the system being migrated from. Presenting completion of these intermediate goals to senior leadership helps them understand that progress is being made and also presents opportunities for leadership to reward and recognize the team as it hits critical milestones. Since large migrations are usually multi-year efforts, the surrounding business landscape changes quite a bit as the migration progresses. Regular reviews of progress to date and plans for, say, next quarter forces the team to regularly think through their priorities. Senior leadership can help by providing a broader business context to the engineering team in these reviews. If business priorities change, it is sometimes the right decision to step away from some parts of the initial plan.

To round up, it is also important to not lose track of the end goal. A large migration is not done until all clients are completely off the old system and the old system has been removed from the code base. It is the responsibility of the migration program owner to continue to apply pressure on the team doing the migration until the migration is indeed fully complete.