Data Migration Testing Tutorial: A Complete Guide

Introduction to Data Migration Testing:
Oftentimes, we hear about applications being transferred to different servers, undergoing technology changes, being updated to the next version, or moved to different database servers, among other scenarios.

• What does all of this actually mean?
• What is expected from the testing team in these situations?

From a testing perspective, it means that the application needs to be thoroughly tested end-to-end, including the successful migration from the existing system to the new system.
This tutorial is part of a series:

• Data Migration Testing Part 1
• Types of Migration Testing Part 2

In this case, system testing must be performed with all the data used in the old application as well as the new data. Existing functionality and new/modified functionality need to be verified.

Instead of just migration testing, it can also be referred to as data migration testing, where the entire user data is migrated to a new system.
So, migration testing includes testing with old data, new data, or a combination of both, as well as testing old features (unchanged features) and new features.
The old application is usually referred to as the ‘legacy’ application. Testing of the legacy application needs to be continued alongside the testing of the new/upgraded application until the new/upgraded one becomes stable and consistent. Performing extensive migration tests on the new application will reveal new issues that were not found in the legacy application.

What is Migration Testing?

Migration Testing is a verification process that involves the migration of the legacy system to a new system with minimal disruption or downtime. It ensures data integrity, no loss of data, and that all specified functional and non-functional aspects of the application are met after migration.
Simple Representation of a Migration System:

Why perform Migration Testing?

When an application is migrated to a new system, it could be for various reasons such as system consolidation, obsolete technology, optimization, or other reasons.
Therefore, when migrating the system in use to a new system, it is essential to ensure the following:

1. Avoid or minimize any disruption or inconvenience caused to the user due to migration, such as downtime or data loss.
2. Ensure that the user can continue to use all the features of the software with minimal or no damage during migration, such as changes in functionality.
3. Anticipate and rule out any possible glitches or hindrances that might occur during the actual migration of the live system.

Hence, to ensure a smooth migration of the live system by eliminating defects, it is essential to carry out Migration Testing in a lab environment.
This testing is important and plays a vital role when data is involved in the migration process.
Technically, it is also required to be executed for the following purposes:

• Ensure compatibility of the new/upgraded application with all possible hardware and software supported by the legacy application. Test compatibility of the new hardware and software platforms as well.
• Ensure that all existing functionalities work as they did in the legacy application. There should be no change in the way the application works compared to the legacy version.
• Identify and fix any defects related to data that may arise during the migration process. These defects can cause a large number of issues and affect data integrity.
• Verify that the system response time of the new/upgraded application is the same or better than the legacy application.
• Ensure that the connection between servers, hardware, and software components is intact and does not break during testing. Data flow between different components should not be disrupted under any condition.

When is this Testing Required?

Testing needs to be performed both before and after migration.
The different phases of Migration testing to be carried out within the Test Lab can be classified as follows:

1. Pre-Migration Testing
2. Migration Testing
3. Post Migration Testing

In addition to the above, the following tests are also executed as part of the entire Migration activity:

1. Backward Compatibility Verification
2. Rollback Testing

Before performing this Testing, it is essential for any Tester to clearly understand the following:

1. The changes happening as a part of the new system, such as server, front end, DB, schema, data flow, or functionality changes.
2. The actual migration strategy laid out by the team, including the step-by-step changes in the backend of the system and the scripts responsible for these changes.

Hence, it is essential to thoroughly study the old and new systems, and then plan and design the test cases and test scenarios to be covered in the testing phases accordingly. Prepare the testing strategy based on this analysis.

Data Migration Testing Strategy

Designing the test strategy for migration includes a set of activities to be performed and a few aspects to be considered. This ensures minimal errors and risks during migration testing, as well as effective migration testing.
The activities involved in this Testing include:

#1) Specialized team formation:
Form a testing team consisting of members with the required knowledge and experience, and provide training related to the system being migrated.
#2) Business risk analysis and possible errors analysis:
Conduct ‘Business Risk Analysis’ meetings with the right stakeholders (Test Manager, Business Analysts, Architects, Product Owners, Business Owners, etc.) to identify risks, implement mitigations, and ensure business continuity after migration. Identify scenarios that uncover these risks and verify the implementation of mitigations during testing.
Perform ‘Possible Error Analysis’ using appropriate ‘Error Guessing Approaches’ and design tests around these errors to identify them during testing.
#3) Migration scope analysis and identification:
Analyze the clear scope of the migration test, including what needs to be tested, when, and how.
#4) Identification of the appropriate Tool for Migration:
Identify the tools to be used for migration testing, whether automated or manual. For example, an automated tool can be used to compare source and destination data.
#5) Identification of the appropriate Test Environment for Migration:
Identify separate environments for pre and post migration testing. Understand and document the technical aspects of the legacy and new systems to ensure that the test environment aligns with them.
#6) Migration Test Specification Document and review:
Prepare a Migration Test Specification document that clearly describes the test approach, areas of testing, testing methods (automated or manual), testing methodology (black box or white box testing), number of test cycles, testing schedule, data creation and usage approach (sensitive information should be masked), test environment specification, and tester qualifications. Review the document with stakeholders.
#7) Production launch of the migrated system:
Analyze and document the steps required for the production migration and share this information well in advance.

Different Phases of Migration

The various phases of Migration are as follows:

Phase #1: Pre-Migration Testing

Prior to data migration, a set of testing activities are performed as part of the Pre-Migration test phase. Although these activities may be overlooked in simpler applications, they are crucial for complex applications.
The following actions are taken during this phase:

• Determine the scope of the data to be included or excluded, as well as any necessary transformations or conversions.
• Perform data mapping between the legacy and new applications, comparing the relevant data types in each.
• Ensure that mandatory fields in the new application are not null in the legacy application.
• Study the data schema in the new application, including field names, types, constraints, validations, etc.
• Identify and verify any changes to the tables and views between the legacy and new systems.
• Record the number of records in each table and view in the legacy application.
• Analyze the interfaces in the new application and their connections, ensuring that data flow is secure and uninterrupted.
• Create and execute test cases, scenarios, and use cases for new features or conditions in the new application.
• Execute test cases and scenarios with a set of users to verify that legacy data and functionality remain intact after migration.

Phase #2: Migration Testing

‘Migration Guide’ prepared by the migration team should be strictly followed during the migration activity. This activity typically begins with data backup so that the legacy system can always be restored if needed.
Verification of the documentation in the ‘Migration Guide’ is also a part of data Migration Testing. The document should be clear, easy to follow, and without any ambiguity. Any errors or mismatches in the order of steps should be reported and fixed.
The migration scripts, guide, and other relevant information should be retrieved from the version control repository for execution.
The actual migration activity takes place on the legacy system.
During this testing, all components of the environment will usually be brought down and removed from the network to carry out the migration activities. The downtime required for the migration will be recorded in the test environment and used to estimate the approximate downtime in the live system.
The migration activity defined in the ‘Migration Guide’ document typically includes the following:

• Actual migration of the application
• Modification of firewalls, ports, hosts, hardware, and software configurations according to the new system
• Perform data security checks
• Verify connectivity between all application components

The tests that verify these activities can be performed in the backend of the system or by conducting white-box testing.
After completing the migration activity specified in the guide, all servers are brought up and basic tests are conducted to verify the successful migration. These tests ensure that all systems are appropriately connected, components are functioning as expected, the database is operational, and the front-end is communicating with the back-end successfully. These tests should be identified and documented in the Migration Test Specification document.
If the software supports multiple platforms, migration should be verified on each platform separately.
Verification of migration scripts is also part of Migration Testing. Sometimes individual migration scripts are verified using white-box testing in a standalone testing environment.
Migration testing is a combination of both white-box and black-box testing.
Once the migration-related verification is complete and the corresponding tests pass, the team can proceed with the Post-Migration testing activities.

Phase #3: Post-Migration Testing

Post-Migration testing takes place after the application has been migrated successfully.
End-to-end system testing is performed in the testing environment, covering identified test cases, test scenarios, and use cases with legacy data as well as new data.
In addition to the general post-migration tests, there are specific items to be verified in the migrated environments:
All of these items are documented as test cases and included in the Test Specification document for execution.

1. Verify that all data in the legacy system has been migrated to the new application within the planned downtime. Compare the number of records between the legacy and new application for each table and view in the database. Also, record the time taken to move a specific number of records.
2. Verify that all schema changes have been implemented in the new system.
3. Ensure that data migrated from the legacy to the new application retains its value and format, unless specified otherwise. Compare data values between the legacy and new application’s database.
4. Test the migrated data against the new application, covering a variety of cases. To ensure complete coverage of data migration verification, automated testing tools can be used.
5. Check for database security.
6. Check data integrity for sample records.
7. Verify that existing functionality in the legacy system works as expected in the new system.
8. Verify data flow within the application, covering most components.
9. The interface between components should be extensively tested to ensure data is not modified, lost, or corrupted. Integration test cases can be used for verification.
10. Check for redundancy in the legacy data after migration.
11. Check for data mismatches, such as changes in data types or storing formats.
12. Ensure all field-level checks in the legacy application are covered in the new application.
13. Verify that any data added to the new application does not affect the legacy data.
14. Test updating legacy application data through the new application, ensuring it does not affect the legacy data.
15. Test deleting legacy application data in the new application, ensuring it does not delete the data in the legacy system.
16. Verify that changes made to the legacy system support the new functionality delivered in the new system.
17. Verify that users from the legacy system can continue to use both the old and new functionality, especially those that have undergone changes. Execute test cases and use the test results from Pre-migration testing.
18. Create new users on the system and test if they are supported by both the legacy and new applications.
19. Perform functionality-related tests with a variety of data samples.
20. Verify if ‘Feature Flags’ are enabled for new features and if switching them on/off activates or deactivates the features.
21. Perform performance testing to ensure the migration has not degraded system performance.
22. Perform load and stress tests to ensure system stability.
23. Conduct security testing to ensure there are no vulnerabilities introduced by the software upgrade, especially in areas that have undergone changes during migration.
24. Verify usability, especially if the GUI layout or front-end system has changed. Evaluate the ease of use compared to the legacy system.

Considering the extensive scope of Post-Migration testing, it is ideal to prioritize important tests that qualify the migration as successful, and then perform the remaining tests later. Automating end-to-end functional tests and other applicable tests can help reduce testing time and provide quicker results.
Tips for writing test cases for post-migration execution:

• Reuse existing test cases for the new application, as they should still be valid for the legacy application. Modify or create new test cases as needed for any changed features.
• Ensure reliability and consistency in test cases. Verify critical data to avoid missing any required verification during the execution.
• If the new application’s design (UI) differs from the legacy application, modify UI-related test cases to adapt to the new design. Decide whether to update existing test cases or create new ones based on the extent of change.

Backward Compatibility Testing

Migration of the system requires verification of ‘Backward Compatibility’, ensuring that the new system is compatible with the old system (at least two previous versions) and functions correctly with those versions.
Backward compatibility testing ensures:

1. Support for functionality provided by the previous two versions, along with the new functionality.
2. Successful migration from the previous two versions without any issues.

Therefore, it is essential to specifically test backward compatibility during migration and include relevant test cases in the Test Specification document.

Rollback Testing

If any issues arise during the migration process or if there is a migration failure, the system should be able to roll back to the legacy system and resume its functions quickly without impacting users and previous functionality.
Therefore, migration failure test scenarios need to be designed for negative testing, and the rollback mechanism should be tested. The time required to resume the legacy system should also be recorded and reported in the test results.
After the rollback, main functionality and regression testing (automated) should be performed to ensure that the migration has not affected anything and that the rollback was successful in restoring the legacy system.

Migration Test Summary Report

A test summary report should be produced after completing the testing. This report should provide a summary of the various tests and scenarios conducted during each phase of migration, along with their pass/fail status and test logs.
The following activities should be recorded:

1. Total migration time
2. Downtime of the applications
3. Time taken to migrate a specific number of records
4. Time taken for rollback

Additionally, any observations or recommendations can be included in the report.

Challenges in Data Migration Testing

Data migration testing poses various challenges, mainly related to data. Some of these challenges include:
#1) Data Quality:
Poor data quality in the legacy application may result in poor data quality in the new/upgraded application. This can be caused by assumptions, data conversions, or invalid data entered in the legacy application. Poor data quality leads to increased operational costs, higher data integration risks, and deviations from business objectives.
#2) Data Mismatch:
Data migrated from the legacy to the new/upgraded application may not match in the new system. This can occur due to changes in data types, storage formats, or redefined data purposes. Addressing this requires effort to correct or accept the mismatched data and adjust it as needed.
#3) Data Loss:
Data loss can occur during migration, whether for mandatory or non-mandatory fields. Loss of non-mandatory field data can be resolved by updating the data again. However, loss of data in mandatory fields renders the record void and requires retrieval from backup or audit logs if captured correctly.
#4) Data Volume:
Migrating large amounts of data requires an extended timeframe within the planned downtime. For instance, scratch cards in the telecom industry or users on an intelligent network platform can present challenges as the amount of data grows during migration. Automation can be a solution for handling massive data migration.
#5) Simulation of a Real-Time Environment:
Simulating a real-time environment in the testing lab poses challenges, as testers encounter different types of issues with real data and systems that are not experienced during testing. Therefore, data sampling, replication of the real environment, and identification of the volume of data involved are crucial in data Migration Testing.
#6) Simulation of Data Volume:
Test teams need to study live system data carefully and come up with analyses and samples to approximate the real environment. Automated tools can be used to generate large volumes of data. In cases where data volume cannot be simulated, extrapolation can be considered.

Tips to Minimize Data Migration Risks

Here are some tips to smooth the risks associated with data migration:

• Standardize data used in the legacy system to ensure consistent data in the new system after migration.
• Enhance data quality to meet business standards, facilitating testing as an end-user.
• Clean data before migration to prevent duplicate data in the new system.
• Verify constraints, stored procedures, and complex queries to ensure accurate data results in the new system.
• Select an appropriate automation tool for data and record checks in the new system, comparing them with the legacy system.

Conclusion

Given the complexity involved in data Migration Testing and the risks associated with any verification oversight during testing that can lead to production failures, it is crucial to carefully study and analyze the system before and after migration. Planning and designing an effective migration strategy with robust tools and skilled testers is vital.
Migration has a significant impact on application quality, and the entire team must make a substantial effort to verify the entire system in terms of functionality, performance, security, usability, availability, reliability, compatibility, etc. This effort will ensure successful Migration Testing.
The next tutorial in this series will briefly explain different types of migrations that commonly occur and testing approaches specific to each type.
About the Authors: This guide was written by STH Author Nandini, who has 7+ years of experience in software testing. Thanks also to STH Author Gayathri S. for reviewing and providing valuable suggestions to improve this series. Gayathri has 18+ years of experience in software development and testing services.
We welcome your comments and suggestions regarding this tutorial.