Plan, Build, & Ship: How to Execute a Data Migration
Migrating to new technology is a reality that all companies face from time to time. While migrations may appear daunting, when they’re well thought out and effectively executed, performing migrations at the right time and for the right reasons can be a competitive advantage. Whether modernizing your infrastructure, adjusting to new regulatory standards, or embracing emerging technologies, migrating to new systems is vital to keep your organization competitive and innovative.
With the increasing costs of Modern Data Stack solutions, particularly those offering software as a service (SaaS), the rapid evolution of tools, and changing data best practices, organizations are grappling with when and how to migrate their data platforms.
Data migrations are unique in many ways since organizations must consider specific challenges such as data validation, data availability during migration, and the change management needed to roll out new organizational workflows, among others. Due to these challenges, migrations can be long and expensive initiatives, often requiring assistance from external resources with experience in the new technologies to help ensure the existing team remains focused on their roadmap. Moreover, migrations sometimes lack a clear ROI, since quantifying and communicating the value proposition may take a long time, and different stakeholders will value different things.
Because planning and execution are so important to data migrations, we’ve created this blog series to highlight our approach to performing migrations. Our overarching framework is based on a tried-and-true product development cycle: Plan — Build — Ship. We’ll highlight the pillars of this cycle as it relates to migrations, and in future posts, we’ll go into greater detail on each one.
Plan
The first phase of data migration is planning. A well-thought-out plan should be able to address questions and cater to different types of audiences in the organization such as:
- Why are we performing this data migration?
- What is the scope of this migration?
- What are the migration’s constraints and requirements?
- Who is involved in this migration, and how?
- What does success look like when we’ve finished the migration?
- What are the risks involved and how are we managing them?
Much of the value in answering these questions is in the process. Understanding why you’re undertaking this project, going through the process of assembling the right team, ensuring the right workflows are in place, and getting stakeholders involved from the beginning are core elements to a successful migration. Ultimately, these and other considerations should be documented, socialized, and aligned on before continuing. The output of this phase will be, minimally, a single document that doubles as both (1) a roadmap with answers to the above, and (2) a tool that you can use to project manage the migration. Keeping these artifacts tightly coupled ensures fewer duplicative efforts and a single source of truth for the migration.
Consider:
|
Build
After planning is complete and the roadmap has been socialized and aligned upon, ideally by a steering committee, you’re ready to enter the next phase of the migration — Build. This is the longest phase in the process and it's where the magic happens. The first step in the Build phase is setting up a “landing zone” to determine your data's destination. Then you move the code and data, followed by thorough testing and validation, a critical piece involving numerous feedback loops across your teams. After passing QA, you deploy to a staging environment before proceeding to the Ship phase where your migration gets fully completed.
Consider:
|
Ship
After the Build phase is completed, you’re onto the final stage — Ship. Shipping changes with extra TLC is important due to the potential impact of data migration work. As your teams look to promote the migrated system to production and begin deprecating old workloads, they should do so gradually.
While the benefits of a technology migration are often clear to data teams, your consumers still rely on the old system and may not realize the benefits of the migration until long after it’s completed. With this imbalance in mind, it’s integral to provide minimal disruption to existing operations and maintain hard-won trust between data and the rest of the organization. It’s also important to maintain existing operations while the migration is underway.
Consider:
|
The Importance of Artifacts and Documentation
Lastly, from beginning to end, it’s critical to create usable artifacts and documentation at each stage of the migration process. Documentation enables the migration team to scale beyond itself, by socializing their work and helping teams onboard themselves. Artifacts, such as training videos, best practices, a data changelog, and validation reports, demonstrate that care was taken during the migration to mitigate operational disruptions. This promotes trust and visibility into the project, which is critical when the change management happens in the back half of the migration process.
As with every project, there are nuances and best practices to each phase which we’re excited to share with you as this series goes on. Stay tuned for more details on how to succeed through each phase of a data migration. Set yourself and your team up for success the next time you upgrade from a legacy system, refactor your data models, or consolidate tooling.
Want help planning your data migration? Contact us. Our data experts can share details more about our Plan—Build—Ship approach to data migrations and use it to ensure a successful migration for your organization.