Category: Test Data Management

Once you’ve decided to tokenize sensitive data, the next decision is architectural. Do you keep a vault that maps every token to its original value, or generate tokens with a cryptographic process so there is nothing to store? Both approaches are reversible, both can preserve the format of the original data, and both appear in production payment and data platforms. They fail in different ways, scale in different ways and carry different compliance implications.

This guide explains how vaulted and vaultless tokenization work, compares them on performance, security and compliance, walks through a worked example and gives you a framework for choosing. If you need a refresher on the fundamentals first, start with what data tokenization is.

The Short Answer

Vaulted tokenization generates a random token and stores it against the original value in a secure database, the vault. Reversing a token means looking it up. Vaultless tokenization generates tokens with a cryptographic process, such as format-preserving encryption or pre-generated token tables, so nothing is stored per value. Reversing a token means running the process backwards with the same key.

Vaulted gives the cleanest separation between token and data and makes individual records easy to revoke. Vaultless scales and distributes more easily, but concentrates risk in the key.

How Vaulted Tokenization Works

In a vaulted model, the tokenization service sits in front of an encrypted database that holds every token-to-value mapping.

1. An application sends a sensitive value, such as a card number, to the tokenization service.
2. The service checks whether the value has already been tokenized. If it has and the system issues consistent tokens, it returns the existing token.
3. If not, it generates a random token, often in the same format as the original and sometimes keeping the last four digits, and confirms it is unique.
4. It stores the mapping in the vault, with the original value encrypted, and returns the token.
5. When an authorized system needs the original, it calls the service, which checks permissions, looks up the token and returns the value.

The defining property is that the token has no mathematical relationship to the original value. Someone who steals tokens, or the algorithm that produced them, learns nothing. The trade-off is that the vault becomes a high-value target. It has to be encrypted, backed up, replicated and access-controlled to the same standard as the data it replaces, and it grows with every distinct value you tokenize.

How Vaultless Tokenization Works

Vaultless tokenization removes the per-value database. The tokenization service derives each token from the original value and a secret, and reverses it with the same secret. There are two common approaches.

• Format-preserving encryption (FPE). The token is the original value encrypted with an algorithm that keeps its length and character set, such as the FF1 mode defined in NIST SP 800-38G. Reversing the token means decrypting it with the same key.
• Static token tables. The service uses fixed tables of random substitutions, generated in advance and distributed to each node. The value is split into pieces and each piece is substituted through the tables. The tables stay the same size however much data you tokenize. Implementations vary by vendor.

Because the same value and the same key always produce the same token, vaultless tokenization is deterministic by default. That keeps joins working across systems, much like deterministic masking does for masked data. Any node that holds the key or tables can tokenize and detokenize locally, with no central database to call.

The trade-off is that the key, or the token tables, become the crown jewels. Anyone who obtains them can reverse every token they produced. Algorithm choice also matters. Short fields give FPE a small domain to work with, which weakens it, and NIST revised its FF3 mode after published attacks. Use a vetted mode such as FF1, follow current NIST guidance on minimum domain size, and keep keys in an HSM or key management service.

Vaulted vs Vaultless Tokenization: Side-by-Side

Criteria	Vaulted	Vaultless
How tokens are created	Random value, checked for uniqueness	Derived from the value and a secret key or token tables
How tokens are reversed	Lookup in the vault	Cryptographic reversal with the same key or tables
Link between token and value	None; the token is random	Mathematical, protected by the key
What you must protect most	The vault, its backups and replicas	The key or token tables
Storage growth	Grows with every distinct value	Fixed; nothing stored per value
Latency	Network call plus database read or write	Local computation; typically lower
Multi-region scaling	Needs consistent replication of the vault	Distribute keys or tables; nodes work independently
Consistent tokens for joins	Optional; requires a lookup before issuing	Built in for a given key
Revoking one person’s data	Delete the mapping and tokens become unresolvable	Harder; delete the tokens or use per-record keys
Key or secret rotation	Re-encrypt the vault; tokens can stay the same	New key changes tokens; stored tokens must be re-tokenized
Compliance scope	Vault and tokenization service stay in scope	Keys, tables and tokenization service stay in scope

Performance and Scaling Trade-Offs

In a vaulted system, every new value means a uniqueness check and a database write, and every detokenization means a read. At high volumes the vault becomes the bottleneck. Running in more than one region adds a harder problem: replicas must stay consistent so two regions never issue different tokens for the same value, or the same token for different values. Caching helps with latency, but a cache of token mappings is sensitive data in its own right.

Vaultless systems are compute-bound instead. Tokenization can run as a local library, a sidecar or a service in each region, and it scales roughly in line with the compute you add. Only keys or tables need distributing, not data. That makes vaultless the common choice for high-throughput pipelines, data warehouses and globally distributed applications.

Security Model

The two models put the critical secret in different places. With a vault, an attacker who steals tokens has nothing; they need the vault or access to the detokenization service. With vaultless tokenization, an attacker who obtains the key can reverse every token it produced, wherever those tokens have been copied.

Rotation is where this shows up operationally. Rotating the encryption key on a vault re-encrypts the stored values while the tokens themselves stay the same. Rotating a vaultless key changes every token, so stored tokens must be re-tokenized or the service must support versioned keys.

In both models, the control that matters most is who can detokenize. Most real-world exposure comes from service accounts and applications with broader detokenization rights than they need, not from breaking the tokenization itself. Grant detokenization narrowly, log every request and review the logs.

Compliance Implications

This section is general guidance, not compliance or legal advice. Confirm your approach with your assessor or privacy team.

• PCI DSS. Tokenization can reduce the scope of the cardholder data environment, because systems that only see tokens and cannot detokenize may fall outside it. The vault or the keys and tables, and the tokenization service, remain in scope. PCI SSC guidance also distinguishes tokens that can be reversed cryptographically from those that can only be resolved by lookup, which can affect how assessors treat systems holding them.
• GDPR. Reversible tokens are generally pseudonymised data, not anonymous data, under either model. The right to erasure is easier with a vault: delete one mapping and that person’s tokens become unresolvable everywhere. With vaultless tokenization, you need to delete the tokens themselves or design per-record keys.
• Data residency. With a vault, the location of the vault and its replicas decides where the original data lives. With vaultless tokenization, the original data is not stored by the service at all, but the location of keys and who can use them still matters.

Worked Example: Tokenizing a Card Number

A retailer runs checkout in two regions, analyses orders nightly and needs to process refunds. It tokenizes card numbers at checkout. The card number below is a standard test number and the tokens are illustrative.

With a vault

1. Checkout in region A sends 4111 1111 1111 1111 to the tokenization service.
2. The service checks its index, finds no existing token, generates the random token 8264 7302 9846 1111 and stores the encrypted card number against it.
3. The order database and the nightly analytics only ever see the token, and repeat customers can be matched because the service returns the same token for the same card.
4. For a refund, the payments service, which has detokenization rights, requests the original. The lookup is checked and logged.
5. Region B must reach the same vault, or a consistent replica. If it issued its own tokens, the same card could end up with two different tokens.

Without a vault (FPE)

1. Checkout in either region sends the same card number to a local tokenization service holding key K, retrieved from the key manager.
2. The service encrypts the digits with FF1 and returns 5930 2817 6604 1111. Both regions produce the same token without talking to each other or storing anything.
3. For a refund, the payments service asks the tokenization service to decrypt the token with K.
4. If K is ever exposed, every token produced with it can be reversed. If K is rotated, the stored tokens must be re-tokenized.

When to Choose Each

Choose vaulted tokenization when:

• You need to revoke or erase individual records cleanly.
• You want tokens with no mathematical link to the original value.
• Volumes are moderate and the vault can sit close to the applications that use it.
• A payment processor or provider already runs the vault for you.

Choose vaultless tokenization when:

• You need high throughput and low latency, for example in data pipelines or warehouses.
• Applications run across several regions or at the edge.
• You want to avoid operating a growing database of sensitive values.
• You need consistent tokens for joins across many systems.

Many platforms support both, and some organisations use a vault for payment data and vaultless tokenization for analytics. And if nobody ever needs the original value back, as is usually the case for test and development data, tokenization may be more than you need. See how tokenization compares with masking before you commit to running token infrastructure.

FAQ

Is vaultless tokenization just encryption?

FPE-based vaultless tokenization is, technically, format-preserving encryption. The difference is mostly in how it is packaged and controlled: tokens replace values across systems, and a dedicated service governs who can reverse them. Table-based vaultless approaches work differently and are not standard encryption.

Which is more secure, vaulted or vaultless?

Neither is inherently more secure. Vaulted tokenization concentrates risk in the vault; vaultless concentrates it in the key or tables. The quality of key management, access control and detokenization auditing matters more than the model you pick.

Can vaultless tokens collide?

An FPE mode such as FF1 is a permutation, so for a given key and domain each input maps to a unique token. Vaulted systems must check for collisions when they generate random tokens.

Does vaultless tokenization remove systems from PCI DSS scope?

It can reduce scope for systems that only handle tokens and cannot detokenize. The tokenization service and the keys or tables remain in scope.

DataOpsZone is an Enov8 resource. Links to Enov8 pages are included where they add detail.

In contemporary IT environments, development speed, security, and operational efficiency are under constant pressure. One concept gaining significant traction in response to these pressures is ephemeral data, a modern approach to delivering fast, compliant, and disposable data for development, testing, and analytics.

This post explains what ephemeral data is, why it matters, and how leading organisations are using it to improve agility while reducing cost and risk.

Defining Ephemeral Data

Ephemeral data refers to data that is short-lived, on-demand, and discarded once its immediate purpose is served. Unlike traditional datasets that are stored, shared, and retained across environments, ephemeral data exists only for the duration of a task, test cycle, or session.

In practice, ephemeral data is:

• Temporary — created just-in-time and removed when no longer needed.
• Non-persistent — not stored long-term or reused across cycles.
• Automated — provisioned programmatically through pipelines or tooling.
• Isolated — delivered to a specific environment without shared dependency.
• Secure and compliant — typically masked or virtualized to reduce exposure.

This model aligns directly with modern DevOps, CI/CD, and cloud-native development patterns.

Why Ephemeral Data Matters

Organisations are moving toward ephemeral environments and ephemeral data for several compelling reasons:

1. Faster Development & Testing

Ephemeral data supports rapid iteration by providing developers and testers with instant, production-like datasets without waiting days for database refreshes. When environments can be provisioned and destroyed automatically, delivery cycles accelerate dramatically.

2. Reduced Storage & Infrastructure Costs

Traditional test databases are often multi-terabyte, persistent, and duplicated across multiple environments. Ephemeral data eliminates these heavy copies, lowering storage consumption and associated infrastructure overhead.

3. Improved Security & Compliance

Short-lived datasets reduce the exposure window for sensitive information. When paired with masking or synthetic data generation, ephemeral data helps organisations maintain compliance with regulations such as GDPR, HIPAA, or PCI-DSS.

4. Elimination of Environment Drift

Long-running non-production environments tend to accumulate configuration drift, creating inconsistent testing outcomes. Ephemeral environments are provisioned cleanly every time, ensuring repeatability and reliability.

5. Scalable Parallel Testing

Because ephemeral data is lightweight and fast to provision, teams can run multiple test cycles or pipelines concurrently — a necessity for high-frequency release models.

Ephemeral Data vs Persistent Data

It’s important to recognise that ephemeral data supplements, rather than replaces, persistent data.

• Persistent data is essential for production, audit, compliance, and long-term storage.
• Ephemeral data is designed for short-lived operational tasks across development, testing, and analytics.

The key is selecting the right model based on purpose and lifecycle.

Delivering Ephemeral Data Through Modern Tooling

To fully realise the benefits of ephemeral data, organisations require automation that supports rapid provisioning, masking, and controlled disposal. This is where dedicated virtualisation and data-provisioning platforms come into play.

One example of such a solution is Enov8 VirtualizeMe (VME), an enterprise-grade platform that delivers lightweight, masked, and disposable database environments in minutes, not hours.

You can learn more about VME here:
https://www.enov8.com/virtualizeme-vme-data-cloning-and-provisioning/

VME enables teams to:

• Create ephemeral database instances and datasets on demand
• Integrate data provisioning into CI/CD pipelines
• Deliver masked or anonymised data for compliance
• Scale parallel test environments without infrastructure sprawl
• Retire environments automatically to avoid drift and reduce cost

This aligns perfectly with the principles of ephemeral data outlined above.

Conclusion

Ephemeral data is becoming a foundational practice for modern IT organisations seeking faster delivery, improved quality, stronger compliance, and reduced operational overhead. By shifting from large, persistent data copies to on-demand, short-lived datasets, organisations can streamline development, reduce environmental risk, and modernise their testing and delivery processes.

With the right platform — such as Enov8 VirtualizeMe — the transition to ephemeral data becomes both achievable and highly beneficial.

Databases can feel complicated, especially when terms like referential integrity start popping up. But the concept is actually quite simple, and understanding it is key to keeping data accurate, consistent, and trustworthy. In this guide, we’ll break down what referential integrity is, why it matters, and how it works—without drowning in technical jargon.

What Is Referential Integrity?

At its core, referential integrity is about making sure relationships between pieces of data stay valid. Think of it as a promise the database makes: if one piece of data refers to another, that other piece of data must actually exist.

A common example comes from everyday life:

• Imagine you’re filling out an online form to book a flight. You pick your departure city, your destination, and the airline.
• The system has a master list of airlines (Qantas, Emirates, Singapore Airlines, etc.).
• Referential integrity ensures that when you choose “Qantas,” your booking system doesn’t accidentally store “Qantaz” or some airline that doesn’t exist in the master list.

In database terms, this is usually enforced through primary keys and foreign keys.

Primary Keys and Foreign Keys

Let’s simplify:

• A primary key is a unique identifier for each record in a table. Example: Customer ID in a Customers table.
• A foreign key is a reference to that identifier from another table. Example: The Orders table stores a Customer ID to show who placed the order.

If a foreign key points to a primary key, the database enforces the relationship. You can’t create an order for a customer who doesn’t exist in the Customers table. That’s referential integrity at work.

Why Is Referential Integrity Important?

Without referential integrity, databases can turn into chaos. Here are three risks:

1. Orphan Records
   • Example: An order exists for a customer who has been deleted. You now have an “orphan” order with no parent record.
2. Inconsistent Data
   • Example: The Orders table says a booking was with “Qantas,” but the Airlines table doesn’t have Qantas listed anymore. Reports and analytics now show unreliable results.
3. Broken Processes
   • Example: A billing system tries to send an invoice but can’t find the customer details. The whole process fails.

By enforcing referential integrity, databases prevent these issues and keep data reliable.

How Databases Enforce It

Most database systems (like Oracle, SQL Server, PostgreSQL, or MySQL) offer rules to enforce referential integrity. Here’s how they typically work:

1. Prevent Invalid Inserts
   • You cannot insert an order with a Customer ID that doesn’t exist in the Customers table.
2. Prevent Invalid Deletes
   • If you try to delete a customer who still has existing orders, the database will block you (unless you handle it properly).
3. Handle Updates Safely
   • If a primary key is changed (rare, but possible), the foreign key values linked to it must also be updated to keep relationships intact.

Options for Managing Relationships

When designing databases, you often need to decide what happens when related data changes. Common strategies include:

• Restrict/Delete Block: Don’t allow deleting a customer if they still have orders.
• Cascade Delete: If you delete a customer, automatically delete all their orders. (This is powerful but dangerous if done carelessly.)
• Set Null: If a customer is deleted, update the Customer ID in orders to NULL. (Useful in some cases, but may create ambiguity.)

Each approach has pros and cons depending on your business rules.

Real-World Analogy

Think of referential integrity like family records:

• A child’s birth certificate lists the parents.
• If the government deletes the parents’ records but leaves the child’s, you now have a document pointing to people who officially don’t exist. That’s a broken reference.
• A good record-keeping system ensures the references always stay valid, just like a database does with referential integrity.

Common Pitfalls

Even with rules in place, mistakes happen. Some common challenges include:

• Disabling Constraints: Developers sometimes temporarily turn off integrity checks for bulk loads and forget to turn them back on. This leads to bad data.
• Poor Design: If relationships aren’t defined properly at the start, the database can’t enforce them later.
• Manual Workarounds: Users bypass rules by editing raw data, creating mismatches.

These pitfalls remind us that referential integrity is not just a technical safeguard—it’s also about discipline in how teams manage data.

Why Should Non-Tech People Care?

If you’re a manager, business user, or executive, here’s why referential integrity matters to you:

• Trustworthy Reporting: Analytics and dashboards rely on accurate data relationships.
• Operational Efficiency: Broken references cause system errors, delays, and extra costs.
• Regulatory Compliance: In industries like finance or healthcare, bad data relationships can mean legal trouble.

Put simply: without referential integrity, your data becomes unreliable, and unreliable data leads to bad decisions.

Referential Integrity and Test Data

When teams copy production data into development and test environments, referential integrity is one of the first things to break. Two common test data operations put it at risk:

• Data subsetting: Taking a smaller slice of a database (for example, 5% of customers) only works if every related order, payment and address comes with it. Copy the parent rows without their children, or the children without their parents, and you get orphan records and failing tests.
• Data masking: If a customer ID is masked in one table but not in another, the foreign key no longer matches. Deterministic masking solves this by always turning the same input into the same masked value, so relationships survive across tables and even across databases.

The rule of thumb: whatever you do to test data, the keys that link tables must stay consistent. When choosing a subsetting tool, check that it follows foreign-key relationships rather than slicing tables in isolation. Enov8 DataFactory includes Prune, a subsetting feature for building smaller, production-like datasets for test environments.

Final Thoughts

Referential integrity may sound like a niche database term, but it’s the backbone of trustworthy information systems. By ensuring relationships between tables remain consistent, businesses avoid orphan records, reduce system errors, and keep their data foundation strong.

So next time you hear “referential integrity,” just think: it’s about keeping the links in the chain unbroken. Without it, the whole system risks falling apart.