How legacy systems and bad data quality hinders a digital transformation plan

Introduction

Cloud. Automation. Analytics. With the rapid influx of tech developments and trends that affect business, organizations are doing everything they can to embrace digital transformation to meet the demands of a fast-changing business landscape and new regulatory standards.

Unfortunately, the focus remains on superficial digital transformation aspects: New customer service tools. New automation tools for work processes. Fancy mobile apps and web interfaces. It is only after the initial excitement dies down that companies realize these digital initiatives did little to actually transform their business.

Digital transformation isn’t just about experimenting with technology – it’s about rebooting the organization’s systems and transforming the business itself. But as companies implement digital transformation initiatives, they come to the realization that they cannot let go of legacy systems.

As if legacy systems were not enough pain, companies also begin to realize they have poor data quality. They cannot replace legacy systems or migrate from an old application to a new CRM until they have clean master data. When companies are hit with this realization, they either act fast to get it sorted or keep the whole transformation idea on the back burner.

Working with private and government organizations for nearly two decades, we’ve seen how poor master data quality residing in outdated legacy systems can potentially destroy a business. Loss of revenue and business credibility, employee fatigue, talent drain, and stressed management are all problems that can drag a business down.

This whitepaper is an attempt at finding the middle-ground solution – giving up a legacy system and improving master data quality is a huge and expensive undertaking, one that doesn’t happen on a whim. Keeping this limitation in mind, there ought to be a middle-ground solution, one that enables a business to optimize its data while making the gradual move from legacy to modern.

Why Legacy Systems are Still Being Used in Organizations

Legacy systems refer to any computer system, programming language, application, or process that can no longer receive support and maintenance; however, since they’ve been woven into the very foundation of the organization, they are not easily replaced or updated for several reasons. While the system may still be functioning well, it does not support growth. As an example, a legacy CRM may not offer integrations with third-party marketing platforms or tools like Google or Facebook.

As technology advances, most organizations find themselves dealing with the oddities of a legacy system. Even with updates, some systems keep the company in a rut, preventing growth or scaling opportunities.

Despite being a bottle-neck, legacy systems are still being used in organizations. There are several varying reasons for this:

High Overhaul Costs: Imagine a system that has been holding decades of data, a system that a business’s model depends on, and a system that is being used by hundreds of employees. Overhauling a legacy system at this level is not only expensive but also detrimental to business operations. Moreover, upgrading to a new system would require an up-front investment, both in money and in talent.

Staffing Challenges: Legacy systems are often built on programming languages that are no longer in use. It’s hard and even expensive to find new talent with the skills to migrate from the legacy system to the new one while retaining the original functionality and scope. For enterprises that have been running on decades-old systems, there may be little to no documentation about the system. Original developers may no longer be employed with the company. In such cases, even the idea of migrating may seem overwhelming.

Uncertainty & Doubt: Companies want systems they know, understand, and can rely on. Legacy systems like the IBM or IBM mainframes offered bulletproof reliability and their characteristics, as well as their limitations, are well understood by organizations. Legacy systems remain in use because of their reputation for reliability and although modern systems offer much more flexibility, some of them are not as reliable or sturdy as legacy systems. Companies are aware of this so they would rather risk being outdated than use a system that could open the doors to vulnerability and instability.

Tailor-Made Systems: Many enterprises operate on systems that have been tailor-made (or tailor-coded) to meet their requirements. Some of these systems even operate with external stakeholders and vendors who are equally comfortable with using the platform. This becomes a challenge for organizations especially since migrating would affect not only business performance but also internal/external operations.

What are the Risks Associated with Legacy Systems?

Although legacy systems are integral to business operations, they can cause a myriad of problems, including the most critical issue – exorbitant maintenance and update costs. As much as organizations are afraid of the up-front costs of upgrading, they are paying even more than the cost of maintenance. The main difference? That cost is a gradual expense that does not alter the functionality of the business.

Multiple Data Silos: Legacy systems were never designed to have integration features, and many legacy solutions are built on frameworks that can’t integrate with newer systems. This means that each system in the organization has its silos leading to disparate data sources that can never be used to obtain a unified single source of truth.

Data Quality Issues: Data quality remains a consistent challenge with legacy systems. Organizations have years and years of master data that have not been structured, optimized, or cleaned. It’s only when companies are beginning to invest in data analysis and data intelligence programs do they find it critical to improve data quality. Again, they meet with another challenge – it’s not easy to fix data when it’s stored in multiple legacy systems and when those systems do not allow the flexibility of data matching and cleansing.

Compliance Issues: Compliance regulations like the GDPR require companies to be accountable for customer data. With increasingly complex compliance regulations, companies cannot risk being careless with data. The problem? Legacy systems are not equipped to meet these compliance challenges especially if they are records stored in outdated, siloed systems.

Vulnerabilities & Security Issues: When legacy systems were built, they did not have to face the constant threat of security and data breaches. Hackers and digital criminals are becoming increasingly malevolent posing a serious threat to businesses. Organizations running on legacy systems have outdated data security measures that put the entire organization at risk.

Inability to Meet with Current Digital Demands: Today and shortly, systems are expected to be hyper-connected meaning, multiple data sources, processes, apps, and systems are expected to collaborate in real time. A legacy system’s inability to integrate with other technologies or ERPs prevents the organization from growing with confidence.

There is only so much you can do in a legacy system. It’s like forcing an old horse to win a race you may make it run, but you can never make it outperform other horses.

Don’t Let Lagging Legacy Systems Affect Your Business

Additional risks we’ve seen some of our clients suffer from include:

• Unstable and often incompatible with new OS, browsers, or other applications.
• Companies may stall upgrades but they always have to keep improving one component or the other.
• Sometimes fixing one component causes other components to malfunction.
• The more patches these systems have, the more susceptible they are to security issues.
• As the years go by, performance will be slower, failure will be more frequent and efficiency will take a big hit. Considering the pace at which technology is moving, legacy systems will fail in the next five to ten years.
• Talent with legacy system knowledge and expertise will no longer be available to manage and sustain the system.
• Data from legacy systems will no longer be reliable or good enough for business intelligence analysis.

So while organizations may stall an upgrade of their legacy system or a critical migration, they will have to realize that eventually, the risks with outdated systems far outweigh the current benefits of using them

Legacy Systems and Poor Data Quality Hindering Digital Transformation

Master data is an organization’s raw data that remains unchanged over a period of time. Every organization will have master data on its products, customers, finances, operations, processes, and other operational areas. This data, while previously ignored in the depths of oblivion has immense value in today’s age. This data plays a crucial role in a digital transformation strategy – it is only unfortunate that master data in most organizations is not robust, reliable, or up-to-date.

It seems obvious that companies should already be ramping up their data management game plan if they haven’t yet done so in 2020. But while companies invested in fancy technologies, they neglected to upgrade their data management processes, causing their data to become a significant roadblock in any digital transformation initiative they try to take on.

Data is central to the operation of a business, however, in today’s digitally empowered world, data is more than just a component of a business. It is the lifeline of a business. The quality of the underlying data is paramount in ensuring the making of correct business decisions.

But what do we specifically mean by poor data quality? And how does a legacy system affect data quality?

Here’s what we found out after years of working with our clients.

Understanding Poor Data Quality

When we talk about data quality, we’re talking about the accuracy, consistency, reliability, usefulness, and completeness of data. Poor data quality refers to data that is incorrect, invalid, irrelevant, and outdated.

Examples of Good Data Quality:

Reason for Maintaining High Quality Data

Examples of BAD Data Quality:

Although these examples may sound pretty basic, in the long run, they become a nightmare for IT teams and data analysts hired to optimize or upgrade the company’s digital infrastructure.

In fact, when data is migrated from one existing database to another, it presents a host of quality issues.

Let’s see some situations where data quality can be a significant roadblock to a transformation plan.

Situations that Create or Exacerbate Data Quality Issues

There are many other instances where processes and operational changes within an organization can have a direct or indirect impact on data quality. With organizations focusing on the automation of internal processes, high volumes of data are at risk of being either neglected or mishandled by team members who may not necessarily be data experts.

In conclusion, data quality may be lost due to processes that bring data into the system, or by processes that seek to enhance existing systems in the organization.

Successful digital transformations, therefore, rely on replacing existing legacy systems, provided it is done with a data management plan in place. As obvious as it is, only a few organizations have a data strategy in place when embarking on an ambitious digital transformation initiative.

Needless to say, data transformation is the first step to a successful digital transformation, but for data transformation to happen, it’s necessary for the organization to set clear goals and vision for the value it expects both transformations to create.

Organizations need to identify the obstacles that are holding them back – from the lack of front-office controls that are the root cause of poor data input to inefficient management of data within multiple legacy IT systems, to a lack of business support and technical talent and finally the unclear goals and ambitions that stall a data & digital transformation initiative.

Solution – What Can Be Done to Improve Data Quality & Update Legacy Systems While Keeping in Mind Existing Challenges?

Digital transformation is not something you can achieve in a month or a year. It takes months and years of planning ahead, assessing risks and challenges, and trying to find a middle-ground. Go too fast and you disrupt a functional business – go too slow and you lose out on growth, opportunities & business value.

So let’s boil it down to the basics.

Data Strategy as the Key to Updating Legacy Systems

Consider digital transformation as the ultimate destination. To get there, legacy systems have to be updated. To update legacy systems, one has to start from data.

Therefore, creating a data transformation strategy is the key to updating legacy systems and ensuring a successful digital transformation.

Here’s what can be done at the most basic level:

1. Extract Data from the System: This might sound rudimentary, but even at an enterprise level, data must be safely extracted from the system and backed up before any other processes occur. Migrating data out of a legacy system starts with ensuring it can be extracted safely.

2. Mapping the Data: When migrating to a new system, old data needs to be mapped onto the new system. The mapping process is to analyze whether the data in the old system is compatible or can be mapped to the new system. This step is important for the new system to understand data from the legacy system.

3. Cleanse & Optimize Data: Data from a legacy system are bound to have quality issues. Before mapping or moving to a new system, old data must be cleansed and rid of duplications, incomplete data, and unstructured data. For example, a legacy system with dashes in phone numbers will not work with a new system that doesn’t allow or use dashes.

4. Cleanse & Optimize Data: Once data is cleansed, a sample set of data is imported into the new system to test for potential problems and errors. This weeds out potential issues before the new system goes live. The validation and standardization procedures ensure that old data is standardized according to the new system’s features.

5. Cleanse & Optimize Data: It is here that staff training and front-end data input procedures must be conducted. Often organizations end up committing the same mistakes with the new organization by overlooking the human factor in data input. For example, addresses must be written in its full form. Names must not be abbreviated and so on. Set standards across the organization ensures everyone’s on board with best practices. While this doesn’t completely prevent errors, it does give employees much-needed awareness.

Once data has been cleansed, standardized, and stored in a centralized system, it becomes easier to manage and analyze. A robust backup and recovery plan, a scalability plan in terms of performance and capacity, a storage location plan must all be part of this data strategy.

Creating a Single Customer View to Optimize Business Processes

The whole purpose of digital transformation is to meet customer expectations and enhance customer experience.

But digital improvements are not a shot in the dark. Companies have to get accurate insight from their customer data to know what to improve. This insight can only be obtained if the data is reliable and can provide a unified, single-customer view.

A single customer view is a holistic, consolidated, consistent representation of an organization’s customer data. Companies may not need to make drastic overhauls to their data or digital platforms to achieve a single customer view. Instead, they can make small steps to clean, dedupe, match, and consolidate customer data held in more than one system – resolving customer entity discrepancies both within and between systems.

Sales, marketing, billing, shipping, and customer service are some of the core departments that share the same customer info. If these departments are each using their own CRM or data input methods, the same data is being recorded, used, and distributed in multiple ways.

This means a company could potentially have dozens of records across data sources for each customer. Some records may contain their buying preferences, some may have demographic data, others may have more complete contact data, etc. And in each of these records, the key identifier is a little different. Perhaps the customer used a nickname when filling out a loyalty rewards program card, or they could’ve used their email address when signing up on the website. Regardless of the reason, there is now a variety of representations for each customer stored across a variety of systems.

Take for example the case of a retail food distributor that does not have a unified source of data. Their
data problems:

• The [phone] token has missing city codes in the number.
• The [address] field has missing zip codes.
• The [name] token has typos, abbreviations, nicknames, fake names, missing first/last names, and son.
• Spellings are inaccurate and contact info has been updated multiple times by different people across different departments.

When executives want to analyze their data to make a key business expansion decision, they can’t seem to get reliable statistics from their data.

Billing data gives a different story than customer service data. Feedback and reviews from customers are random and inconsistent.

External stakeholders or third-party distributors send in their data which doesn’t map with the retailer’s existing data pattern. Because retailers send local information, they don’t record phone numbers with city codes.

When marketing obtains this information, they are dealing with broken email records, missing phone numbers, typos in spelling, and much more.

Data duplication and data inconsistency spell disaster for this company.

How Do You Obtain a Single Customer View?

Data Integration:

The first step is obviously to identify and integrate first-party, second-party, and third-party data sources that contain the customer information to bring together. Which format is the company receiving data from partners, vendors, or suppliers? Which applications or databases are in place internally? What public data sources are connected or used?

Customer data sources needed to build a true Single Customer View can include:

Make a list of data sources and create a strategy for the integration of these sources. Now cross-check this list against your legacy system to identify which of them can be integrated perfectly into the system and which would require external options.

Data Profiling: When integrating data from multiple sources at scale, data quality, like typos, irrelevancy, incompleteness, inaccuracy, and lack of standardization needs to be sorted. Profiling data after integration helps give an overview of quality issues such as punctuation in names, missing zip codes in addresses, etc. Fixing these issues gives better results when matching different representations of the same customer in the final step of the process.

Data Cleansing: Once quality issues are identified, it’s time to scrub it clean and standardize it to get the best results. The cleansing step can be very time-consuming and requires considerable attention to detail. For faster, accurate results, investing in data cleansing software is a good choice.

Data Matching: This is the most important part of the process where data is matched to identify and remove duplicates, compare data, identify patterns, and enable greater standards. For best data-matching results, data must be standardized to minimize false negatives. Ideally, a good data quality solution software will also allow for visual data matching where the user can define their match conditions.

Data Cleansing: Once the data is cleansed, deduped, and standardized, it’s time for the final step in the process: Choosing a master record. After all, the purpose is to build a Single Customer View that contains the cleanest, most complete record for every customer. Now that you’ve matched all of your data sources that contain customer data, it’s time to bring all of that information together. DataMatch Enterprise, the data quality software of Data Ladder allows for a Merge & the Survivorship step where the user can merge data into a single, comprehensive record.

Obtaining a single customer view will not only help the business identify opportunities but will also assist them in upgrading from legacy systems without disrupting the customer experience or the core operations of the business.

It must be noted once again that digital transformation is not about uprooting existing systems – it’s about how well you optimize business processes to meet customer demands while keeping in mind budget and operational restraints. With the experience we’ve had with hundreds of clients over the years, we know for certain that the most successful transformations always start with data. New systems and processes are a later investment.

Case Study: Maxeda DIY Group

Data Challenge

Like many large enterprises, Maxeda had lots of legacy system issues that were affecting their digital transformation plan. Their data was split not just between different systems, but also between different countries. Additional challenges included non-unified methods of collecting and storing data. All this disparity eventually led to numerous data challenges such as the lack of unified systems, unorganized data, and old and dirty data that needed cleaning and standardizing. These challenges were significant roadblocks to Maxeda’s digital plans. Before they could move into a new CRM or PIM or implement a marketing plan, they had to ensure optimal data quality.

DataMatch Enterprise Was Used to Provide a Single Customer View

Benefits

Conclusion

Aging systems and infrastructure is bogging down processes and hurting company revenues. But it’s not just about a slow system – it’s the data in that system that has not been brought up to par with modern digital needs. So while organizations aim for augmented or virtual reality, blockchain, cloud computation, data lakes, and many other digital initiatives, they will find it hard to succeed unless they don’t find a way to unleash their data.

Essentially then, the true essence of transformation is not systems, but data, which is the soul of a system.

Image taken from: https://www.britannica.com/biography/Tim-Berners-Lee