Got New Tech? Don’t Forget Your Old Data!

Every month we explore the various tools and technologies that impact individuals and organizations. In Part Two of our three-part “New Tech, New Landscape” series (“Choosing Technology That Fits Your Business”), my colleague provided helpful guidance for organizations looking to assess their current information system solutions to select technology that meets their unique needs. In Part Three of the series “Got New Tech? Don’t Forget Your Old Data!”, I outline factors to consider during the planning phase of an information system change or update and provide direction for putting together a data migration plan aimed at ensuring continuity of operations and avoiding common migration pitfalls.

New System…Check! Now What?

So, you’ve assessed your organizational needs and strategically selected a new information system you’d like to replace your legacy (i.e., old) system(s)—that’s wonderful! Your new system comes with the potential to enable significant improvements in service delivery and organizational performance, while reducing operating costs. Unfortunately, your strategy and planning work isn’t yet over; when switching over to a new information system (i.e., migrating to a new system), there are many factors to consider and make decisions on. One of these factors is whether, and how and when, to “flip” over your data streams and/or historical data from the legacy system to the new system. Information systems function on and produce data that feeds into your business operations and these processes almost invariably require continuity if the organization is to continue seamlessly operating during the tech switch. As a result, thoughtfully producing a thorough data migration strategy and plan is an essential step in the information system migration process.

Data Migration

Data Migration is the process of preparing and transporting data between locations, formats, or systems. At a minimum, your new system implementation plan will schedule when the old system is turned off and the new system is turned on. However, to the degree that your business operations rely on your information system data to function, you can’t simply leave your current and historical data in your legacy system and use only day 0 data in the new system—this makes the data migration strategy and planning effort a crucial step for a successful new system implementation. At a high-level, a basic data migration will involve extracting, transforming (as required), transferring, and loading legacy data into a new system. Although this sounds straightforward enough, there are considerable complicating challenges to overcome and strategic opportunities to understand that allow you to refine your business processes and underlying data.

Project Management

While technical in nature, a data migration project is still an organizational project. As a result, the starting place for a data migration strategy and plan lies in proven project management principles. In particular, the following project management practices can support the production of a solid migration project plan:

• Scope the migration project: Before you begin the migration project, you should have a comprehensive understanding of the scope of the project. Examples of scoping activities include identifying what the project requirements and objectives are, as well as the project budget and schedule.

• Include cross-functional stakeholders: The project requirements and objectives development process should include meetings with stakeholders across the functions that rely on or contribute to system data. The involvement of the stakeholders early in the planning process is critical for establishing a more complete understanding of your data ecosystem.

• Determine migration activity owners: Data migration projects are often large, multi-activity projects, requiring several related and disparate processes. As with any large project, it’s important to clarify project roles and responsibilities for each migration activity in the project plan, ideally at the task-level. This step serves to formalize the impact each activity has on the overall migration, while allowing for a decision-making paper trail.  

Current State Assessment

In order to strategically set your organization up for success in your new system, it’s important to engage in strategic planning. To identify which migration activities are value-added, you’ll need to thoroughly assess the state of your legacy system and data processes, the new system and its data rules, and the business processes your systems support. Consider conducting the following activities as part of your strategic migration planning process:

Landscape Analysis

Landscape analysis involves cataloging the data elements and related models, dictionaries, schemas, and business processes in the organization. Conducting this analysis is critical because it allows for a review of the entirety of your legacy system data and the processes it feeds into or relies on. It provides an understanding of the relationships between data, enabling strategic planning and later standardizing, mapping and cleaning activities.

Data Mapping

Data mapping is the process of identifying where data exists in your legacy system and determining where to locate it in your new system. Data mapping is essentially the principal process that allows you to take inventory of all the disparate data elements in your system to understand them in relation to your business processes. One major benefit of conducting data mapping is this process will identify data and system quality issues like duplications and inconsistencies in the data feeds your processes rely on. Illuminating these inefficiencies now allows you to clear up the issues in your data map prior to migration, resulting in more concise, efficient business processes in your new system. Automated data mapping is typically performed using software that maps data between the systems automatically. The main benefits of automating data mapping include reduced costs due to a quicker migration (e.g., reduced downtime) and less manual work, which can minimize potential errors.

Data Profiling & Quality Check

Another major data migration process is data profiling, which involves reviewing and configuring data between the legacy and new systems. Data profiling can also provide information on the accuracy and validity of the structure and content of, and relationships among, your current data. The concept behind the data profiling activity is that data and operations that function well in your legacy system in one format, or according to legacy system rules, may not work the same way in the new system. Data profiling then allows you to evaluate and account for system differences and data quality issues to ensure processes won’t be interrupted during migration (to the degree possible).

Data Cleaning

Data profiling enables data cleaning like a doctor’s exam enables medical treatment—data profiling diagnoses the problems; data cleaning addresses them. Thus, before migrating your legacy data into the new system, the legacy data should be reviewed for accuracy and cleaned. At a high level, the data cleaning activity improves data quality by removing obsolete, inaccurate, or duplicate data and is arguably the most impactful activity on the success of your migration project. The data cleaning activity typically involves the following tasks:

• Data de-duplication: Involves identifying and removing duplicate data elements in your system. Increases system processes parsimony.

• Data validation/verification: Involves replicating system outputs manually or using automated software to verify the system operates on clean, correct data.  

• Data parsing/concatenation: Involves reformatting legacy system data elements to either be broken into smaller chunks or combined into larger chunks to better fit new system rules or processes.

While every project will be constrained by time, scope, and cost, it is not advisable to avoid addressing data quality issues during your migration project. It may seem easy to punt this task into the future, but by doing so you’ll miss an opportunity to correct what is often years of data management issues that negatively affect your organization all the time. Legacy data can be cleaned in the legacy environment, in the new system, or in a staging system during migration, with each option having strategic strengths and weaknesses. Be sure to consider the details of your systems and processes to determine the ideal time to clean your legacy data, as opposed to making a blanket decision out of convenience.

Go-Live Validation & Rollback Plan

Before executing your migration, you’ll want to be certain everything occurs according to plan. You’ve got some options for testing or validating the data migration prior to your go-live date. One typical testing activity is parallel testing, where processes using the same data are run simultaneously on both the legacy and new system and outputs compared. After testing is conducted according to specifications and standards, involving stakeholders to engage in acceptance testing provides an opportunity for users to interact with legacy data in the new system. Engaging stakeholders through acceptance testing also enables additional opportunities to identify issues and can lead to improved user adoption and new system satisfaction/engagement.

Even though you’ve planned everything meticulously, unexpected or unplanned errors still occur. Certainly, a surprise error that leads to your new system going down will be quite disruptive and should be avoided to the extent possible. Having a rollback plan is an effective “plan b” should one or more of these unexpected system errors occur. A rollback plan essentially returns the system to a pre-determined state, allowing for the restoration of previous processes. These plans are clearly very important for business continuity, but they’re also helpful for affording some precious time to identify and resolve system migration errors, should they arise.

Closing Thoughts

A data migration project is essential for a seamless transition to your new IT system. As an added benefit, strategically planning your data migration effort allows you a rarely afforded opportunity to review your existing processes and data, identify and remedy data quality issues, and establish new maintenance and monitoring procedures for the new environment—it’s the Spring cleaning of your data house! Consider taking advantage of the opportunity to initiate new processes and tools aimed at making your organization more efficient and agile.

So, there you have it, folks! We covered quite a bit in this three-part blog series: New Tech, New Landscapes. In Part One, “A Morphing Landscape: Technical Desires”, Dominic Jeter focused on how the changing technical landscape came to fruition through the cloud movement. In Part Two, “Choosing Technology That Fits Your Business”, Shrish Willett shared strategies organizations can take to ensure they are making as informed a decision as possible when choose technology that fits your business. This third and final blog wraps up the series by outlining important data considerations when migrating systems.

Interested in learning more about one of these tech topics? Let us know at bd@fmpconsulting.com!

James Wilcox joined FMP in May 2019 and works in the Technology and Tools Center of Excellence mostly on Human Capital IT Transformation and Analytics. James is from Orlando, Florida and most enjoys learning from others and exploring new places.