Updated November 2026.
A sound QA strategy depends on good test data, and good test data depends on a proper test data management (TDM) process. Copying production in full is the usual shortcut, and it quickly runs into storage costs, slow refreshes and long-running test suites.
This is a hands-on guide to how to subset a database for testing. It walks through the referential integrity problem, a step-by-step process, working SQL examples, and when to move from scripts to a build, buy or open source tool.
Data Subsetting in a Nutshell
Subsetting means taking a small, self-contained slice of production, such as 5% of customers plus every account, order and transaction that belongs to them, so tests still behave as they would against the full database. If you want the full definition, methods and a worked example first, read Enov8’s data subsetting guide. This article focuses on the practical work of building subsets.
Why Not Just Copy Production?
To see why subsetting is needed, start with the problem it solves. One of the most popular ways to get test data is to copy it from production, often called production cloning. Nothing is more realistic than real data, but full clones bring two kinds of problems. Security and privacy risks are addressed with techniques like data masking. This guide focuses on the other problem: infrastructure.
The Infrastructure Cost of Production Cloning
Copying 100% of production into a test environment means paying for production-scale storage and compute outside production. Most organizations also run multiple QA environments, so that cost is multiplied three, four or five times.
There are indirect costs too. Large datasets take longer to copy, load and refresh, which slows test cycles and delays feedback to developers.
Decide Your Selection Strategy First
Before writing any code, decide how rows will be chosen: a percentage of an anchor table, business criteria such as region or status, a list of specific entities, a time window, or a mix. The Enov8 guide compares these approaches. Whichever you pick, the hard part is the same: keeping every relationship intact.
The Core Challenge: Referential Integrity
Subsetting brings its own difficulties, and the biggest is referential integrity. Imagine you work for a social network with one million users, and you want 100,000 of them in your test database. Taking 100,000 rows from the users table is easy. The hard part is fetching only the posts, pictures and friendships that belong to those users, so every foreign-key relationship stays intact and no rows are left orphaned.
Some relationships make this harder than it first looks. A friendship row links two users, so it can only be kept if both users are in the subset. Lookup and reference tables usually need to be copied in full. Self-referencing tables, such as employees and their managers, have to be followed recursively.
The problem grows when relationships span multiple databases. Nothing stops an organization from storing user profiles in PostgreSQL and posts in Oracle. It grows again across different kinds of data sources, with some data in relational databases, some in CSV files and some in a NoSQL document store. Keeping a consistent slice across all of them is one of the main reasons teams move from scripts to purpose-built subsetting tools.
How to Subset a Database, Step by Step
- Work from a copy. Never run reduction logic against production. Start from a backup, clone or gold copy.
- Map the relationships. Document foreign keys, including implied relationships the schema does not enforce.
- Choose anchor tables. These are the driving entities, such as customers or users, that define what is in the subset.
- Define the criteria. Set the target size, filters and any must-include records for edge cases.
- Traverse the relationships. Follow foreign keys down to child tables and up to required parent and reference tables.
- Extract or reduce. Copy the selected rows to the target, or delete everything else from the cloned copy.
- Validate. Check row counts and confirm there are no orphaned rows or broken constraints.
- Mask sensitive data. A subset is still real data, so apply masking before it reaches non-production users.
- Automate and version. Store subsetting rules in version control so refreshes are repeatable.
Four Ways to Subset a Database
There are four main ways to implement data subsetting, from hand-written SQL scripts to dedicated tools.
1. Using SQL Queries
The simplest approach is plain SQL. Using the social network example, suppose there are 10,000 users and you want 100 of them. In PostgreSQL you could take the first 100:
SELECT * FROM users ORDER BY id LIMIT 100;Taking the first rows by ID tends to favor the oldest accounts, though. For a more representative sample, use random sampling instead:
CREATE TABLE subset_users AS
SELECT * FROM users TABLESAMPLE SYSTEM (1);Then fetch the related rows. Posts follow directly from the selected users, but friendships must have both users in the subset:
SELECT p.* FROM posts p
WHERE p.user_id IN (SELECT id FROM subset_users);
SELECT f.* FROM friendships f
WHERE f.user_id IN (SELECT id FROM subset_users)
AND f.friend_id IN (SELECT id FROM subset_users);In practice, these queries live in script files that should be checked into version control so you can track changes.
The advantage of SQL is that it is easy to get started, and most organizations already have people with SQL skills. The drawback is that it doesn’t scale. As the schema grows, the scripts become harder to understand, change and maintain. A change to the root query, such as the one that selects users, cascades down to every child query. Poorly optimized queries against large tables can also become too slow to use.
2. Building a Custom Solution
The second approach is to build an in-house subsetting application, using whatever languages and frameworks your engineers already know. This can work for small and medium-sized teams, especially on cost, but the downsides usually outweigh the benefits.
Building the tool takes development time that could be spent elsewhere, and it requires deep database knowledge. Once it is built, you also have to maintain it for as long as it is in use, including every time the schema changes.
3. Adopting Commercial Tools
Why build when you can buy? That’s the reasoning behind the third approach. Commercial data subsetting tools offer robustness, scalability and support for many databases and data sources. Many also combine subsetting with masking, synthetic data generation and automation through APIs.
The main downside is cost. That includes not only licensing but total cost of ownership, including the learning curve, which can be steeper than expected. When evaluating tools, check database coverage, how relationships are discovered, whether the source is left untouched, and how output is delivered.
4. Using Open Source Tools
Open source subsetting tools offer something of both worlds: you don’t have to build from scratch, and you can read and extend the source code if you need more capabilities. The downside is again total cost of ownership, which can still be high depending on the learning curve, the maturity of the project and the support available.
Automating Data Subsetting with Enov8 Prune
Hand-written subsetting scripts break down as schemas grow. Enov8 Prune, the subsetting engine in Enov8 DataFactory, automates the process. It cuts a large database down to anywhere from 1% to 100% of its size, with every foreign-key relationship intact, so nothing is left orphaned or broken.
You choose anchor tables, set a target size and optionally add row filters by value, text, year, number range or empty/not empty. Prune treats the source as read-only and never modifies it. It can output CSV or JSON per table, schema DDL in your database’s dialect, or a reduction script that shrinks a cloned copy in place. Supported sources include MySQL, SQL Server, Oracle, PostgreSQL, Db2 for LUW and SAP HANA.
Because Prune’s output is real data, pair it with masking where needed. When you need data that doesn’t exist in production yet, DataFactory’s Bloom engine generates synthetic data from the schema. For a deeper walkthrough of Prune, including a worked retail banking example, read Enov8’s data subsetting guide.
If your team already provisions database clones, there is another route: dynamic data subsetting, which uses data cloning to give each environment a lightweight copy.
Where Subsetting Fits with Masking and Synthetic Data
Subsetting reduces volume, but the rows it keeps are still real. Most teams subset first and then mask, use synthetic data where production can’t be used, and use database virtualization when environments need full-size copies quickly. For more on balancing these, see To Mask, or Not to Mask.
Data Subsetting Best Practices
- Never run subsetting or reduction scripts against production.
- Mask every subset that contains personal or sensitive data.
- Include known edge cases explicitly rather than relying on random samples to find them.
- Validate every run for orphaned rows, broken constraints and expected row counts.
- Keep subsetting rules in version control and review them when the schema changes.
- Automate refreshes so test environments stay current without manual effort.
Data Subsetting FAQ
Can I subset a database with SQL alone?
Yes, for small and stable schemas. As the number of tables and relationships grows, SQL scripts become hard to maintain, because every change to the anchor query cascades through the child queries. That is usually the point to consider a tool.
What percentage of data should I subset?
It depends on the test. Functional testing often works well with 1% to 10% of production, while performance testing usually needs much larger volumes. Start small, then increase the size if tests miss important scenarios.
Does data subsetting remove personal data?
No. A subset contains fewer records, but those records are still real. Apply masking or use synthetic data before the subset reaches non-production environments.
Can you subset across multiple databases?
Yes, but it is harder. You need to keep the same set of business entities, such as the same customers, across every database so cross-system relationships still line up. This is where scripts usually struggle and dedicated tools help most.
Should I build or buy a subsetting tool?
Building gives full control but costs development time and ongoing maintenance whenever the schema changes. Commercial tools add database coverage, relationship discovery and support, at a licensing cost. Open source sits in between. Compare total cost of ownership, not just the price tag.
Conclusion: Subset Your Way to Better Test Data
Copying production data for testing is an old technique, and it gives tests realistic data. But full copies bring security and privacy risks, high infrastructure costs and slow test cycles.
Data subsetting addresses the cost and speed problems, and combined with masking it addresses privacy too. Start with SQL if your needs are small, and move to a tool such as Enov8 Prune when referential integrity, scale and repeatability start to matter.
Thanks for reading, and until next time.
Author
This post was written by Carlos Schults. Carlos is a .NET software developer with experience in both desktop and web development, and he’s now trying his hand at mobile. He has a passion for writing clean and concise code, and he’s interested in practices that help you improve app health, such as code review, automated testing, and continuous build.
