Your complete guide to a successful data migration

If you’re reading this guide, chances are you’ve already decided to initiate a data migration project and are looking for additional guidance.

Cutting right to the chase, this guide will help you create a data migration strategy and follow a framework that will reduce your chances of making costly mistakes. It will also help you understand where and how you can use a data quality solution to optimize the process and make your data migration a success.

Let’s get started!

Understanding the challenges you may have to face

According to industry analysts, at any given time, around Fortune 1000 companies are engaged in data migration or data conversion projects. However, nearly 70 – 80% of these projects fail, translating into a loss of billions of dollars.

Yes, the statistics may sound alarming, but if you really take into consideration the complex and time-consuming process of data migration, it’s easy to see how there is a higher failure ratio.

Data migration is inherently a high-risk project. Moving your data from a legacy or mainframe environment to a new environment requires more than just technical expertise. You would need a foolproof data migration strategy, a plan, and checklists, all while fixing data quality issues, hiring/retaining talent, team collaboration, and conducting trials and tests among many other tasks.

In our experience working with 4,500+ clients, we’ve noticed several major challenges that eventually become the reasons for data migration failure.

Here’s a quick heads-up on potential challenges you may face and one that you have to be prepared for:

1. Neglecting the Importance of Data Quality:

Ample studies and research indicate that poor data quality costs businesses millions of dollars, yet, businesses tend to overlook the importance of data quality. The result? Poor data becomes a bottleneck that prevents successful data migration. If data is being moved from a legacy system to a new system, chances are the system may even outright reject the data, especially if the newer system has stricter standards and data quality governance.

Data quality refers to the health of your company’s data. If your data suffers from:

• Inaccurate information
• Invalid and incomplete information
• Typos, character errors, punctuation issues
• Duplicate data that affects data quality
• Incorrect formatting and messy data (upper/lower case, inconsistencies, etc)

… you may have a data quality crisis.

If these issues are not resolved, your data migration process is bound to be a failure.

2. Business users and high-level executives oblivious to Data Quality Issues

Data quality is seldom considered as a business problem especially since it’s managed in a database and that is under the domain of IT. Business leaders of various departments are hardly aware of the issues with their data so when data migration initiatives take place, they are oblivious to data quality issues.

Business leaders need to be aware of their data collection process and the resulting issues caused at the data collection stage. Unless business leaders know and deal with these issues, any data migration, any data analysis, and any reporting will be flawed.

3. Lack of a concrete migration plan that binds everything and everyone together

Most companies restrict their migration plan to technical requirements, budget, timeline, and costs. A robust migration plan focuses on processes, planning, resource management, workload management, change management, and much more. Moving data from one environment to another can take months of intensely focused work that takes into account:

• Data auditing.
• Data cleanup.
• Data quality controls.
• Monitoring and managing business activities.
• Setting of expectations.
• Resolving conflicts.

4. Poor execution

Data migration is not something that can be accomplished in-house on a whim. You will need extensive expert guidance, attention, and consultation – not to mention combining several external tools and applications to make the migration a success.

Most organizations overestimate their team’s capacity for an in-house, DIY data cleaning and data migration process. Ultimately, this results in costly expenses, loss of talent and failed projects. Because data migration is a one-time effort, it is always wise to work with experts than to attempt to task your IT team to the project.

Here’s a quick breakdown of the costs you will incur if you get your team to even do basic data quality checks in-house. The cost multiplies exponentially if you get your team to also perform a data migration process.

5. Poor organizational planning

Data migration will be a tough journey, hitting hard on your organization’s resources. Companies that fail to clearly define deliverables, timelines, job responsibilities, and expectations will find it hard to succeed. At the end of the day, any activity within the organization is dependent on its people. While businesses focus on the technology part, little do they realize their people part is missing.

This is why it’s important to create a structured task workflow so members are aware of their job role and are able to perform it without any confusion disrupting the process.

How does one ensure a successful data migration?

An organization’s business model, size, resources, ambitions, goals, revenue, etc. affect the scope of their migration process and creates unique challenges. Therefore, it would not be fair to impose a one-size-fits-all answer or success recipe.

That said, over the years, we’ve gathered a key understanding of factors that impact the success rate of a data migration process and we’ve had clients who successfully pulled off a major migration project because they addressed these factors and did it the right way.

Here’s what you should do to ensure you’re on the right track.

Creating a plan and answering key questions

Despite its challenges and problems, data migration can be successful if a robust plan is put in place. We’d say you must spend at least 3 months in simply creating a plan. The plan must effectively consider the following:

• Why you want to migrate and what goals do you want to achieve with the migration?
• Is your data suitable enough to be migrated?
• What type of data migration methodology will you follow? (more on this below)
• Is your data quality up to the mark?
• How is data stored in source systems?
• What are the possible risks involved in the process?
• Who would you be choosing as your migration service provider?
• The tools, consultants or experts you will be using for the job
• The resources you have for the job and their respective expertise
• Do you have a cost list – i.e the costs associated with each step of the process?
• Will your employees be able to readily adopt and adapt to new processes and systems?
• Would you be needing additional talent?
• How much are you willing to give to the migration process?
• What controls do you have in place to ensure that the process will not cause regular business
• activities to suffer?
• What workflows and processes do you have in place to ensure everything is in sync?
• Do you have a set of data migration policy documents?

Dealing with scope creep

In migration projects, it is not uncommon to deviate away from the scope. Known as, ‘scope creep,’ this is the phase when multiple change requests, adjustments, and fixes begin affecting the original scope or intent of the project. It also happens when more tasks and responsibilities are added or when new issues are discovered that were unexpected or undiscovered during the planning phase.

It is therefore imperative that the initial business goals, planning, and evaluation of records in the old system are effectively taken into consideration to avoid a scope creep that can become a potential problem for the smooth execution of the project. It’s bad for your project and for everyone involved in your team.

Here’s how you can avoid scope creep from causing your project to fail:

Be Very Specific With Your Project Goals from the Start: It’s “cool” to add a new feature or a new tool, but if it’s not part of your project goals or if it doesn’t impact the migration in any way, it’s not required to be part of your project. Managing scope creep starts from ensuring that your team is on track with the project and goals are communicated and approved right from the start.

Document requirements: Set requirements and clearly define the timelines, budgets, and responsibilities of your team members. The more thorough you are with requirements, the better your chances of avoiding new requirements creeping up on your initial scope.

Use Task Management Tools to Keep Everyone On Track: Your schedule, deliverables, tasks, and goals should all be documented on a project or task management software to keep everyone on track. This also helps you identify any red flags that may potentially become a bottleneck and may need instant attention.

Implement a Change Management Plan: It’s not to say that you should entirely ignore any new issues rather, you have to be smart about it and implement a change management plan to keep the changes in control. Make sure any change is analyzed thoroughly in terms of its impact on your business and your migration project. Additionally, you will have to go through the whole decision maker/client/stakeholder approval process to ensure the change is approved and is in the pipeline to be executed along with other tasks.

DME is specifically dedicated to providing a flexible data-matching solution that gives the user the ability to do much more than just basic matching.

Whether you like it or not, scope creep will possibly happen but it’s necessary not to let it dictate your core process flow or project management plan. You will have to find a smart way to incorporate new changes with your existing plans.

Focusing on data quality

A lot has been said about data quality, but how do you go about practically implementing it? Here are things you can do:

Question the reliability & quality of your data: Issues with data quality may seem so deceptively simple that you may end up neglecting it completely. Problems like typos, spelling mistakes, and invalid or incomplete addresses are not noticed until they become the cause of flawed reports and analytics. In the case of data migration, poor data quality can result in complete failure. Don’t assume your data is perfect by looking at it superficially. Ask your teams to create data quality reports and identify commonly faced issues.

• Do they have complete physical addresses?
• Do they have the right unique identifiers?
• Is the data duplicated?
• Is the data suffering from the lack of standardization (upper and lower case issues)
• Are phone numbers valid
• Are physical addresses validated with the government database?
• Is redundant data removed?
• Is there a data governance policy in place?

Examples of common data issues would be:

Invest in a Data Cleaning and Matching Tool

Data migration will require you to invest in multiple tools to fix existing issues. One of these being a
data cleaning and matching tool. Why? Because this tool will help you:

• Match and consolidate data from disparate sources. All that data you have siloed away will need to be merged and consolidated into a single source of truth before you attempt to move them into the new system.
• Remove duplicates and ensure consistency of data.
• Profile data and lets you discover errors plaguing your data including typos, case issues, format issues, etc.
• Validate addresses against government databases.
• Helps create a master record which you can eventually use for data migration.

Regardless of structure, type, or format, source data intended for migration should be validated in terms of the following key attributes:

• Relevance: Is it relevant to its intended purpose?
• Accuracy: Is it correct and objective, and can it be validated?
• Integrity: Does it have a coherent, logical structure?
• Consistency: Is it consistent and easy to understand?
• Completeness: Does it provide all the information required?
• Validity: Is it within acceptable parameters for the business?
• Timeliness: Is it up to date and available whenever required?
• Accessibility: Can it be easily accessed and exported to the target application?
• Compliance: Does it comply with regulatory standards?

The biggest and most drastic mistake you would make would be to migrate your data untreated.

Choosing a migration strategy

There are multiple ways to build a data migration strategy. Your organization’s specific business needs and requirements will establish the most appropriate strategy. However, most migration strategies are bundled into two categories – “Big Bang” and “Trickle.”

Big bang data migration

This is a common strategy, but one that is performed under immense pressure. The company’s resource is shut down for a limited timeframe, a period within which the data goes through the ETL process (Extract, Transform, Load) and transitions into the new database.

There is a limited time-frame. Teams are under pressure to pull off the transition without any failure and must be achieved to avoid maximum business loss.

While this is an easy, do-it-once strategy, it will require the process to be executed flawlessly. You could try emulating the process through a simulation before the actual event.

The trickle migration

This approach is less intensive and is a gradual process where the migration is done in phases. During this implementation, the old and new systems run in parallel, eliminating the pressure of downtimes and operational interruptions. Real-time processes are not affected and data can be migrated smoothly.

Compared to the big bang approach, this method is more complex and requires consistent effort, however, because it is low-risk, it’s more preferable.

The use of both of these approaches depends on your business size, requirements, resources and most importantly on your timeline. If data migration is an immediate, top-priority need, you may want to use the big bang approach, provided you are aware of the risks involved. Have Data Controls in Place: Fix data quality, create a backup of your fresh data. You cannot afford to lose data in case the operation goes wrong.

Analyzing, Fixing, Optimizing Source Data: We’ve mentioned this before, but we can’t help reiterating how important this is. Take a deeper look at your data. Evaluate quality. Get the right software to fix issues. Create a clean record.

Map Source Data: Before actually migrating data, mapping source data to the target system will help you identify how the records and fields in your existing databases correspond to the new target data fields.

Start Building: Now that you have a clean data source, have mapped it to your target data-set, start building the migration logic and test with sample data using a production environment.

UAT: This is where the pre-involvement of business users is most helpful. They will be instrumental in testing out the new environment from a business user perspective allowing you to identify and fix any potential risks or conflicts.

Production Build: This is the final phase of the process where you execute the full data migration, provided you are absolutely sure all risks were assessed and potential hiccups have a valid resolution.

At each stage of the data migration process, you will be employing a combination of tools to help you progress to the next stage. For example, in the first stage, you’ll need to invest in a data quality tool to clean your data and make it suitable for migration. The data quality tool will be helpful at the start as well as at the end of the project when you will need to constantly monitor and maintain data quality in the new system. Here is where Data Ladder’s DataMatch Enterprise solution will be your strongest ally against data issues.

How to Make DataMatch Enterprise An Important Part of Your Data Migration Process

Data Ladder’s DataMatch Enterprise is a data quality tool that will be instrumental in the first phase of your data migration process. The tool follows the data quality framework, allowing businesses to:

• Integrate and Import Data From Any Data Source.
• Profile data to get a bird’s eye view of quality problems.
• Verify and validate addresses against government databases.
• Clean and standardize data.
• Match and merge data to remove duplicates & ensure consistency
• Create a master data record
• Automate cleaning schedule

Data Ladder’s solution is designed to be used by any business size for any data size at an unmatchable price plan, unparalleled speed, and accuracy. Unlike other data quality solutions, Data Ladder does not keep a multitude of separate tools for resolving data problems. Whether it’s cleaning address data or deduplicating list data, you get everything in one on-premises solution that can be deployed both on the cloud and on your private server.

Clean and sanitize data source: Connect over 150+ data sources for thorough cleansing and data sanitization including important functions as address verification and validation. You can even standardize data according to pre-defined business rules, specific delimiters (such as identifying fields with punctuation marks) and much more.

Match data sources to remove duplicates: Data stored in legacy systems that have not been cleaned or updated for a long time will require more than just data cleaning. It will require thorough checks for duplicates and redundant information. For example, one entity with five addresses is data duplication that can cause significant problems for your new database.

Implement data management standards: Once you know the source of your data problems and the kind of issues that plague them, you will be in a better position to implement data management standards. When your business and team leaders understand the value of data and the steps they need to take to ensure data quality, your migration project will have a higher success rate.

Conclusion

Whatever the reason or goal for data migration, one thing remains clear – it must be given the due diligence it needs to be successful. Fancy tools, expensive teams, and complicated processes will not help in making a data migration successful. What you need are:

• A powerful, cohesive, well-documented, approved PLAN.
• A data migration strategy that takes into account your business nature and size, your timeline, your budget, and available resources.
• A data-centric approach where your priority must be to have clean, usable, reliable data before you even plan for a migration.
• A commitment to protect your data quality, ensuring that you will keep data updated and prevent it from degradation caused by unchecked errors or duplication.

Following a governed strategy and methodology will reduce the risks and pain of managing a data migration. You will need to invest in a range of software designed to meet multiple purposes of the migration. Data cleaning and matching tools for data quality, project management tools, task management tools and many more.

If you follow the guide given in this whitepaper and adopt a single focus to manage data migration, such as on data quality, your migration project will have it’s most critical challenge resolved.