Category: Test Data Management

Every aspect of our daily lives involves the usage of data. Be it our social media, banking account, or even while using an e-commerce site, we use data everywhere. This data may range from our names and contact information to our banking and credit card details.

The personal data of a user is quite sensitive. In general, all users expect a company to protect their sensitive data. But there is always a slight chance that the app or service you are using might face a data breach. In that case, the question that comes to mind is how the company or app will keep your data safe.

The answer is data privacy regulations. Nowadays, most countries have their individual data privacy laws, and companies operating in those countries generally follow these laws. Data privacy laws protect a customer’s data in production. But did you ever think about whether your dev or testing environment is safe and secure?

In this post, we’ll discuss why you must follow data privacy regulations in a non-production environment. We’ll take a look at the challenges faced while complying with privacy rules, solutions to these challenges, and strategies to follow while implementing privacy laws in non-production. But before that, we’ll discuss a bit about privacy regulations. So, let’s buckle up our seat belts and take a deep dive.

What Do You Mean by Privacy Regulations?

Data privacy regulations ordata compliance is a series of rules that companies must abide by to ensure that they’re following all the legal procedures while collecting a user’s data. Not only that, but it’s also the company’s job to keep the user’s data safe and prevent any misuse.

There are various data privacy laws. For instance, companies operating under the European Union follow GDPR. On the other hand, the United States has several laws like HIPAA, ECPA, and FCRA. Failing to follow these rules results in potential lawsuits or penalties. The goal of these rules is to keep a user’s sensitive data safe and secure from malicious activities.

Now that we know what data privacy regulation is, let’s discuss why we need to follow these rules in non-production.

Why Privacy Regulations in Non-Production Are Important

While deploying an app or a site in production, we add various security protocols. But often, the environment where we develop or test our apps is not that secure. In 2005 and 2006, Walmart faced a security breach when hackers targeted the dev team and transferred sensitive data and source code to somewhere in Eastern Europe.

This kind of incident can happen to any company. Currently, many companies use production data for in-house testing or development. So, how does a company ensure that a user’s sensitive data is safe? The answer is data masking, which is one of the mandatory rules of data privacy regulations.

However, implementing data privacy rules comes with many challenges. Let’s explore some of them and the ways to resolve these challenges.

Challenges Faced While Complying With Privacy Rules

Adapting to something new always comes with certain challenges, be it some new tool, technology, or regulation. Data privacy is no exception. However, the challenges are not that complicated. With proper planning, overcoming them is quite straightforward.

Adapting to New Requirements

Data privacy regulations are generally process-driven. While implementing privacy rules in non-production, your team must welcome changes in the way they do things. This may involve data masking, generating synthetic data, etc. Your team will take some time to adapt to the new processes.

Chalk out a plan before the transition. Train your team and explain why they need to follow these regulations. With proper training and clarification of individual roles, adapting to the new changes won’t take much time.

New Rules of Test Data

If your testing team is using real user data for testing the essential features of your product, beware. The process is going to change. As per data privacy regulations, you cannot use real user data for testing, so the challenge comes while rearranging or recreating your test data.

However, with a proper test data management suite, the task becomes a lot easier than doing the entire thing manually.

Adjusting Your Budget Plan

Implementing any new process often involves spending a lot of money. While implementing privacy laws, you have to think about factors like

• the research your teams need to do
• the purchase and implementation of data compliance tools that will help you generate privacy-compliant test data
• the arrangement of training sessions for your team
• the hiring of resources to monitor or enforce compliance laws

All of the above and more will affect your budget, so it’s best to have a discussion with your finance and technical team. Figure out the zones where you should focus spending and calculate an approximate amount. Planning is beneficial if you want to avoid overspending. On that note, in the following section, we’ll discuss some strategies to follow while implementing privacy regulations in non-production.

Strategies to Implement Privacy Regulations in Dev and Testing

Although there is no end to planning strategies while implementing data privacy regulations, there are some important steps that we can’t miss.

Sorting Data

Before following privacy laws, you must know everything about your data. If the project is at a starting phase, there will be a lot of customer data. Discuss this with your team to categorize the data and clarify what data is sensitive to the user. Once you categorize the data and separate sensitive data from general data, it’s time for the next steps.

Encrypting Sensitive and Personal Data

GDPR and other data privacy laws make it mandatory for you to secure any sensitive data. Ensure that if you have any such data in a non-production environment, it’s secured by layers of encryption. Even if you’re not using the data, you must still secure it in your database. This is because no matter how strong your firewall is, hackers can always breach it. So it’s wise to protect sensitive data with layers of encryption apart from just a firewall.

Restricting Access to Database

As per most data privacy rules, your database should not provide overall access to all users. Since a database has multiple types of data, you must create roles and grant specific permission to each role. For instance, a tester should have access to test data only and not production data. Imagine if a fresher on your team deletes a table from the production database. The incident may happen by mistake, but it will cost the company a lot. Enforce these rules to prevent similar unfortunate mishaps.

Change the Policies of Cookies

If you’re developing a site, you’ll need to think about how your cookies work and whether they comply with the data privacy law you’re following. For instance, what if your website is operating outside the EU and the target audience is in the EU? In that case, apart from standard compliance, you need to comply with GDPR as well. As per GDPR, a website should collect a user’s personal data only after they agree to cookie consent. That means you should inform the user about the data used by your site’s cookies to perform specific functions. The information must be clear, and your cookies can collect data only after the user gives permission.

Use of a Compliance Monitoring Solution

Generally, companies often appoint a data protection officer (DPO) whose job is to monitor the processes, analyze the risk, and suggest measures so that your company never fails to comply with privacy laws. But a DPO is a normal human being. When it comes to large data sets, a human mind can always miss something. The solution? Provide your DPO with a compliance monitoring solution.

Enov8 provides such a solution that addresses the needs of compliance managers. The tool monitors your data and identity risks. Not only that, but the tool also helps you to find compliance breaches and points out processes that you need to optimize in order to protect the data.

Disclose Important Information to Users

Data privacy laws ensure that users should have all the knowledge about how companies are using their data. You must disclose everything about data usage while signing the agreements. Situations may arise later for which you may need to revise the agreement. For instance, suppose you’re monitoring the logs of a system that’s connected with the customer’s network. If the logs contain the user’s IP address or other sensitive data, inform the customer.

Synthetic Test Data Generation and Data Masking

There are some cases where you need real data to develop or test something. But what if the data compliance standard that your company follows prohibits you from using real data? Don’t worry. Synthetic data is the next best thing. Synthetic data is data generated by an algorithm and closely imitates the original data. You can also use data masking, where sensitive data is hidden and replaced by similar dummy data. The advantage? You can continue your work without any risk of failing to comply with privacy laws.

Train Your Team on Privacy Regulations

When it comes to complying with privacy laws, there is no end to learning and adapting to new things. It’ll be quite hectic for your team if you enforce a lot of rules on your team all of a sudden. Make the transition smooth by arranging training sessions for your employees to explain the need for compliance with privacy laws and the consequences if they fail to abide by these laws. In addition, train them on using data compliance suites. You can take a look at Enov8’s data compliance suite, which monitors your data and ensures you’re compliant with GDPR, FCRA, ECPA, and multiple other standards.

Keeping your test and dev data compliant with privacy laws may prove to be a little challenging at first. But if planned and executed in a phased manner, your team will adapt easily.

Author

This post was written by Arnab Roy Chowdhury. Arnab is a UI developer by profession and a blogging enthusiast. He has strong expertise in the latest UI/UX trends, project methodologies, testing, and scripting.

Updated November 2026.

A sound QA strategy depends on good test data, and good test data depends on a proper test data management (TDM) process. Copying production in full is the usual shortcut, and it quickly runs into storage costs, slow refreshes and long-running test suites.

This is a hands-on guide to how to subset a database for testing. It walks through the referential integrity problem, a step-by-step process, working SQL examples, and when to move from scripts to a build, buy or open source tool.

Data Subsetting in a Nutshell

Subsetting means taking a small, self-contained slice of production, such as 5% of customers plus every account, order and transaction that belongs to them, so tests still behave as they would against the full database. If you want the full definition, methods and a worked example first, read Enov8’s data subsetting guide. This article focuses on the practical work of building subsets.

Why Not Just Copy Production?

To see why subsetting is needed, start with the problem it solves. One of the most popular ways to get test data is to copy it from production, often called production cloning. Nothing is more realistic than real data, but full clones bring two kinds of problems. Security and privacy risks are addressed with techniques like data masking. This guide focuses on the other problem: infrastructure.

The Infrastructure Cost of Production Cloning

Copying 100% of production into a test environment means paying for production-scale storage and compute outside production. Most organizations also run multiple QA environments, so that cost is multiplied three, four or five times.

There are indirect costs too. Large datasets take longer to copy, load and refresh, which slows test cycles and delays feedback to developers.

Decide Your Selection Strategy First

Before writing any code, decide how rows will be chosen: a percentage of an anchor table, business criteria such as region or status, a list of specific entities, a time window, or a mix. The Enov8 guide compares these approaches. Whichever you pick, the hard part is the same: keeping every relationship intact.

The Core Challenge: Referential Integrity

Subsetting brings its own difficulties, and the biggest is referential integrity. Imagine you work for a social network with one million users, and you want 100,000 of them in your test database. Taking 100,000 rows from the users table is easy. The hard part is fetching only the posts, pictures and friendships that belong to those users, so every foreign-key relationship stays intact and no rows are left orphaned.

Some relationships make this harder than it first looks. A friendship row links two users, so it can only be kept if both users are in the subset. Lookup and reference tables usually need to be copied in full. Self-referencing tables, such as employees and their managers, have to be followed recursively.

The problem grows when relationships span multiple databases. Nothing stops an organization from storing user profiles in PostgreSQL and posts in Oracle. It grows again across different kinds of data sources, with some data in relational databases, some in CSV files and some in a NoSQL document store. Keeping a consistent slice across all of them is one of the main reasons teams move from scripts to purpose-built subsetting tools.

How to Subset a Database, Step by Step

1. Work from a copy. Never run reduction logic against production. Start from a backup, clone or gold copy.
2. Map the relationships. Document foreign keys, including implied relationships the schema does not enforce.
3. Choose anchor tables. These are the driving entities, such as customers or users, that define what is in the subset.
4. Define the criteria. Set the target size, filters and any must-include records for edge cases.
5. Traverse the relationships. Follow foreign keys down to child tables and up to required parent and reference tables.
6. Extract or reduce. Copy the selected rows to the target, or delete everything else from the cloned copy.
7. Validate. Check row counts and confirm there are no orphaned rows or broken constraints.
8. Mask sensitive data. A subset is still real data, so apply masking before it reaches non-production users.
9. Automate and version. Store subsetting rules in version control so refreshes are repeatable.

Four Ways to Subset a Database

There are four main ways to implement data subsetting, from hand-written SQL scripts to dedicated tools.

1. Using SQL Queries

The simplest approach is plain SQL. Using the social network example, suppose there are 10,000 users and you want 100 of them. In PostgreSQL you could take the first 100:

SELECT * FROM users ORDER BY id LIMIT 100;

Taking the first rows by ID tends to favor the oldest accounts, though. For a more representative sample, use random sampling instead:

CREATE TABLE subset_users AS
SELECT * FROM users TABLESAMPLE SYSTEM (1);

Then fetch the related rows. Posts follow directly from the selected users, but friendships must have both users in the subset:

SELECT p.* FROM posts p
WHERE p.user_id IN (SELECT id FROM subset_users);

SELECT f.* FROM friendships f
WHERE f.user_id   IN (SELECT id FROM subset_users)
  AND f.friend_id IN (SELECT id FROM subset_users);

In practice, these queries live in script files that should be checked into version control so you can track changes.

The advantage of SQL is that it is easy to get started, and most organizations already have people with SQL skills. The drawback is that it doesn’t scale. As the schema grows, the scripts become harder to understand, change and maintain. A change to the root query, such as the one that selects users, cascades down to every child query. Poorly optimized queries against large tables can also become too slow to use.

2. Building a Custom Solution

The second approach is to build an in-house subsetting application, using whatever languages and frameworks your engineers already know. This can work for small and medium-sized teams, especially on cost, but the downsides usually outweigh the benefits.

Building the tool takes development time that could be spent elsewhere, and it requires deep database knowledge. Once it is built, you also have to maintain it for as long as it is in use, including every time the schema changes.

3. Adopting Commercial Tools

Why build when you can buy? That’s the reasoning behind the third approach. Commercial data subsetting tools offer robustness, scalability and support for many databases and data sources. Many also combine subsetting with masking, synthetic data generation and automation through APIs.

The main downside is cost. That includes not only licensing but total cost of ownership, including the learning curve, which can be steeper than expected. When evaluating tools, check database coverage, how relationships are discovered, whether the source is left untouched, and how output is delivered.

4. Using Open Source Tools

Open source subsetting tools offer something of both worlds: you don’t have to build from scratch, and you can read and extend the source code if you need more capabilities. The downside is again total cost of ownership, which can still be high depending on the learning curve, the maturity of the project and the support available.

Automating Data Subsetting with Enov8 Prune

Hand-written subsetting scripts break down as schemas grow. Enov8 Prune, the subsetting engine in Enov8 DataFactory, automates the process. It cuts a large database down to anywhere from 1% to 100% of its size, with every foreign-key relationship intact, so nothing is left orphaned or broken.

You choose anchor tables, set a target size and optionally add row filters by value, text, year, number range or empty/not empty. Prune treats the source as read-only and never modifies it. It can output CSV or JSON per table, schema DDL in your database’s dialect, or a reduction script that shrinks a cloned copy in place. Supported sources include MySQL, SQL Server, Oracle, PostgreSQL, Db2 for LUW and SAP HANA.

Because Prune’s output is real data, pair it with masking where needed. When you need data that doesn’t exist in production yet, DataFactory’s Bloom engine generates synthetic data from the schema. For a deeper walkthrough of Prune, including a worked retail banking example, read Enov8’s data subsetting guide.

If your team already provisions database clones, there is another route: dynamic data subsetting, which uses data cloning to give each environment a lightweight copy.

Where Subsetting Fits with Masking and Synthetic Data

Subsetting reduces volume, but the rows it keeps are still real. Most teams subset first and then mask, use synthetic data where production can’t be used, and use database virtualization when environments need full-size copies quickly. For more on balancing these, see To Mask, or Not to Mask.

Data Subsetting Best Practices

• Never run subsetting or reduction scripts against production.
• Mask every subset that contains personal or sensitive data.
• Include known edge cases explicitly rather than relying on random samples to find them.
• Validate every run for orphaned rows, broken constraints and expected row counts.
• Keep subsetting rules in version control and review them when the schema changes.
• Automate refreshes so test environments stay current without manual effort.

Data Subsetting FAQ

Can I subset a database with SQL alone?

Yes, for small and stable schemas. As the number of tables and relationships grows, SQL scripts become hard to maintain, because every change to the anchor query cascades through the child queries. That is usually the point to consider a tool.

What percentage of data should I subset?

It depends on the test. Functional testing often works well with 1% to 10% of production, while performance testing usually needs much larger volumes. Start small, then increase the size if tests miss important scenarios.

Does data subsetting remove personal data?

No. A subset contains fewer records, but those records are still real. Apply masking or use synthetic data before the subset reaches non-production environments.

Can you subset across multiple databases?

Yes, but it is harder. You need to keep the same set of business entities, such as the same customers, across every database so cross-system relationships still line up. This is where scripts usually struggle and dedicated tools help most.

Should I build or buy a subsetting tool?

Building gives full control but costs development time and ongoing maintenance whenever the schema changes. Commercial tools add database coverage, relationship discovery and support, at a licensing cost. Open source sits in between. Compare total cost of ownership, not just the price tag.

Conclusion: Subset Your Way to Better Test Data

Copying production data for testing is an old technique, and it gives tests realistic data. But full copies bring security and privacy risks, high infrastructure costs and slow test cycles.

Data subsetting addresses the cost and speed problems, and combined with masking it addresses privacy too. Start with SQL if your needs are small, and move to a tool such as Enov8 Prune when referential integrity, scale and repeatability start to matter.

Thanks for reading, and until next time.

Author

This post was written by Carlos Schults. Carlos is a .NET software developer with experience in both desktop and web development, and he’s now trying his hand at mobile. He has a passion for writing clean and concise code, and he’s interested in practices that help you improve app health, such as code review, automated testing, and continuous build.

You may have heard of code smells or even smelly test environments.

But, what about Data Smells?

In this post we discuss top smells that indicate you need Test Data Management.

Foreword

In computer programming, a code smell is any characteristic in the source code of a program that possibly indicates a deeper problem.^[1][2] Determining what is and is not a code smell is subjective, and varies by language, developer, and development methodology.

The term was popularised by Kent Beck on WardsWiki in the late 1990s. Ref Wikipedia.

Invariably every IT problem has a symptom that we call smells.

In this post lets focus on the most popular ones associated with Test Data Management

Top 15 TDM Smells

1. Testers waste large amount of time creating data rather than testing the application.
2. Data provides doesn’t meet the requirements for testing (has incorrect mix of data).
3. DevTest cycle /and project slippage to data unreadiness.
4. Dependency on other experts to provide the Test Data (experts that may have other priorities).
5. System Data lacks integrity (is incomplete) and limits System Testing.
6. Up or Down stream data hasn’t been prepared in similar fashion, causing E2E integrity issues.
7. Data Related defects caused by data being in unrealistic state i.e. False Positives.
8. Test Data Creation is (or is deemed) too complicated or time consuming.
9. Test Data is too large causing refreshes to take too long.
10. Test Data size (production size) causes performance bottlenecks & broken batch processing in smaller test environments.
11. Data has been copied from production and has PII data i.e. Data is insecure.
12. Testers (& developers) don’t understand what the platform data looks like. Resulting in the engineers fumbling in the dark as they try to exercise it effectively.
13. Due to Data complexity, Testers can’t easily find (mine) the data sets once it has been deployed.
14. Test results are being corrupted by testers “only” using/reusing the same small data sets  (data contention).
15. Lack of data reuse (or automation) resulting in continuous reinvention and repeated mistakes.

In Conclusion

Test Data is an essential, if not somewhat complicated, and often ignored, aspect of effective Devops & Quality engineering. However treating it as an after thought will invariably result in the smells described above. Smells that will introduce suboptimal DevOps/DataOps operations, unwanted project delays and poor testing.

Do you have other ideas on Test Data Management Smells, then please let us know?

Post By Jane Temov

Jane is an experienced IT Environments Management & Resilience Evangelist. Areas of specialism include IT & Test Environment Management, Disaster Recovery, Release Management, Service Resilience, Configuration Management, DevOps &Infra/Cloud Migration.