Category: Test Data Management

Guides to test data management (TDM): data masking, tokenization, synthetic data, subsetting, provisioning, database virtualization and keeping non-production data secure and compliant.

Posted on

Why Virtualize your Data?

Why Virtualize Data

1: Introduction

Data is increasingly recognized as one of the most valuable assets of modern organizations. It can help businesses make better decisions, improve customer experiences, optimize operations, and gain competitive advantages. However, managing and leveraging data can also be a complex and challenging task, especially when dealing with multiple data sources, formats, and systems.

Traditional approaches to data integration, such as extract-transform-load (ETL) or data warehousing, can be time-consuming, costly, and inflexible. In recent years, a new technology called data virtualization has emerged as a promising alternative to these traditional methods. This technology enables organizations to access and integrate data from various sources in a faster, more flexible, and cost-effective way.

In this post, we will explore the benefits of data virtualization and why organizations should consider adopting it as part of their data management strategy.

2: Benefits of Data Virtualization

Data virtualization offers several benefits over traditional data integration methods, including:

  1. Improved data agility: Data virtualization allows for faster and easier data access, integration, and delivery. With data virtualization, organizations can create a single virtual view of their data without physically moving or duplicating it. This means that changes to the underlying data sources can be reflected immediately in the virtual view, without requiring time-consuming ETL processes or disrupting downstream applications.
  2. Cost savings: Data virtualization can help organizations reduce the need for physical data copies, manual ETL processes, and maintenance of multiple systems. By consolidating data sources into a virtual view, organizations can reduce storage costs, simplify data management, and streamline development and testing processes. Additionally, data virtualization can help organizations avoid the costs associated with building and maintaining complex data warehouses or data marts.
  3. Better data governance and security: Data virtualization can help organizations enforce data policies, track data lineage, and secure sensitive data. With data virtualization, organizations can create a unified data access layer that can be centrally managed and controlled. This enables organizations to define and enforce data access rules, audit data usage, and monitor data quality across all data sources.
  4. Enhanced analytics and insights: Data virtualization enables organizations to access and analyze data in real-time, providing more accurate and timely insights. By combining data from multiple sources, organizations can gain a more complete view of their operations, customers, and markets. This can lead to better decision-making, improved customer experiences, and increased operational efficiency.

Overall, data virtualization can help organizations achieve greater agility, cost savings, data governance, security, and insights. In the next section, we will look at some real-world examples of how organizations have used data virtualization to achieve these benefits.

3: Use cases and Examples

Data virtualization has been used successfully in a variety of industries and applications. Here are some real-world examples:

  1. Financial services: A large financial institution was struggling to integrate data from multiple legacy systems, which were siloed and had different data structures. By using data virtualization, the organization was able to create a unified view of its data without physically moving or copying it. This enabled the organization to deliver real-time data to its front-office applications, improve risk management, and reduce operational costs.
  2. Healthcare: A healthcare provider needed to integrate patient data from multiple electronic health record (EHR) systems to improve clinical decision-making and patient outcomes. By using data virtualization, the organization was able to create a real-time, unified view of patient data, which could be accessed by clinicians across different locations and systems. This enabled the organization to improve patient care coordination, reduce medical errors, and increase efficiency.
  3. Retail: A large retailer wanted to improve its customer experience by creating a 360-degree view of its customers, which included data from online and offline channels, social media, and loyalty programs. By using data virtualization, the organization was able to create a single, real-time view of customer data without duplicating it. This enabled the organization to personalize customer interactions, optimize marketing campaigns, and increase sales.
  4. Manufacturing: A manufacturing company needed to integrate data from multiple systems, such as ERP, CRM, and supply chain management, to improve production planning and quality control. By using data virtualization, the organization was able to create a unified view of its data, which could be accessed by different departments and stakeholders. This enabled the organization to optimize its production processes, reduce waste, and improve product quality.

These examples demonstrate how data virtualization can help organizations achieve their data integration and management goals in a faster, more flexible, and cost-effective way. In the next section, we will look at some of the potential challenges and limitations of data virtualization.

4: Potential challenges and limitations

While data virtualization offers many benefits, it also has some potential challenges and limitations that organizations should be aware of. Here are a few:

  1. Performance issues: Depending on the complexity and size of the data sources, data virtualization can sometimes suffer from performance issues such as latency or slow response times. This can be especially true when dealing with real-time data sources. To mitigate this challenge, organizations can implement caching or other optimization techniques.
  2. Complexity: Data virtualization can be a complex technology that requires specialized skills and knowledge to implement and maintain. Organizations may need to invest in training or hiring skilled professionals to ensure that data virtualization is implemented effectively.
  3. Data latency: Depending on the nature of the data sources and virtualization technology used, there may be some latency or delay in the delivery of data. This can impact real-time applications or those that require up-to-the-minute data.
  4. Vendor lock-in: Some data virtualization vendors may require organizations to use their proprietary software or platforms, which can lead to vendor lock-in and limit flexibility.
  5. Security and compliance: Data virtualization can introduce new security and compliance challenges, such as data privacy, access control, and auditing. Organizations should carefully consider how to ensure data security and compliance when implementing data virtualization.

While these challenges should not discourage organizations from adopting data virtualization, they should be taken into account when planning and implementing data virtualization. With proper planning and implementation, organizations can mitigate these challenges and reap the benefits of data virtualization.

5: Conclusion

In conclusion, data virtualization is a technology that can help organizations improve their data integration, management, and analytics capabilities. By providing a layer of abstraction between the physical data sources and the end-users, data virtualization enables faster and easier data access, integration, and delivery. It can also help organizations save costs, improve data governance and security, and gain more accurate and timely insights.

However, data virtualization also has some potential challenges and limitations that organizations should be aware of, such as performance issues, complexity, data latency, vendor lock-in, and security and compliance risks. These challenges can be mitigated through proper planning, implementation, and monitoring.

Overall, data virtualization is a promising technology that can help organizations achieve their data-related goals in a more agile, cost-effective, and secure way. By adopting data virtualization as part of their data management strategy, organizations can stay competitive and make better use of their valuable data assets.

Posted on

How to secure non-production data: A Guide.

Secure Non Production Data

INTRODUCTION

Production data and non-production data are very important to an organization of any size. And sometimes real production data makes it to non-production data in databases. That is one of the reasons why securing the data is so crucial. In this post, we are going to explain what production data and non-production data are. Then we’ll show you how to secure non-production data.

PRODUCTION DATA VS. NON-PRODUCTION DATA

Before we talk about securing non-production data, let’s discuss what production data and non-production data are and how they differ. 

PRODUCTION DATA

Production data is the data that is the business. Every organization, whether a startup or a big multi-national company, has critical data. For a bank, the customer data and the transactional data are production data. And for an e-commerce giant, the production data is the product catalog, the user information, and the transaction. This kind of data is secured with the best systems available. But any data taken by a hacker can cause both reputational and financial losses. 

NON-PRODUCTION DATA

Non-production data is generally used for testing and development purposes. In an ideal scenario, it should be fake data, but it should emulate real data. Suppose your production database contains 10 million records. That means the test database should also contain 10 million records. One thing to note is that only the action load and performance testing can be done. But sometimes the developers and testers require real production data. And in such cases, they are given a subset of the data, which is generally replicated. This is the reason why securing non-production data is so important. Even if this subset of production data is stolen by a hacker, it can cause havoc in the organization. 

DIFFERENCES BETWEEN PRODUCTION AND NON-PRODUCTION DATA

Every organization uses databases. The data is the main part of the business, and in most cases, it’s the business itself. Now, whether this data is stored for internal purposes (like data on all the employees) or for external purposes (like the catalog of an e-commerce site), it is all considered production data. Every developer needs to work with databases to develop applications. They cannot work on production data, as they can corrupt it or, in worst-case scenarios, delete it. So, all developers work on non-production data from non-production databases. This data generally consists of fake records that replicate the original production database. But in some cases, it contains some real data, as the developers need to check the real structure of the records. The testers testing the database or application work on both production and non-production data because they need to test the application before it goes live in production to real-world users. They also test the production data the way an end user experiences it once the release goes through to production. 

SECURING NON-PRODUCTION DATA

As discussed earlier, the non-production data used by developers can also contain sensitive production data. These records can be sensitive records like credit card details, bank details, and even Social Security numbers. The exact data is not required by developers, but they at least need the structure of the database and the schema of the record. Now, before giving the data to developers or testers, it’s important to mask sensitive data through data masking. 

DATA MASKING

As the name suggests, we mask the original data before handing it to the developers or testers. In this process, the company first decides on the sensitive data that cannot go to the non-production database. The perfect masking needs to be done in a manner where the original data doesn’t go to the developer. But the data should have some meaning—a zip code should be a valid one. Some of the methods used for masking are shuffling and multiplier. In shuffling, the names are changed, so John becomes David and vice versa. And in multiplier, a random number is added to numeric data like dates. So, 12/31/2010 becomes 13/11/2019. Data masking is generally done with the help of tools, which we will look into in the next section. These tools mask data in two ways: static masking and dynamic masking. In static masking, the production database is used to create a static database, which contains masked data. This masked data is then used by developers and testers. In dynamic masking, whenever the developer or tester makes some query to the production database, a proxy service receives the request. It gets the real data from the production database but converts it to dummy data by masking it. And then it returns this masked data to the developer or tester. 

DATA MASKING TOOLS

Here are some of the top data masking tools available. 

ENOV8 TEST DATA MANAGER

The Enov8 Test Data Management platform speeds up your development & testing process by identifying where data security vulnerabilities reside inside your databases, rapidly remediating those risks, through masking, to avoid breaches and automatically validate PII compliance success. It also comes with IT delivery accelerators for example: Data provisioning (DataOps) automation, Data Mining & Test Data Booking features. Enov8, geared for the larger enterprise, is probably the most “holistic” or feature rich solution.

ORACLE DATA MASKING AND SUBSETTING

Oracle Data Masking and Subsetting is a solution from a top provider that also runs on non-Oracle databases. It completes the masking in very little time. Besides masking, it also helps remove duplicate data in testing and development databases. The only drawback is that since it comes from a top vendor, it’s costly. For pricing details, you need to contact Oracle directly. 

INFORMATICA PERSISTENT DATA MASKING

Informatica’s persistent data masking tool is again a solution from a top vendor. It is created with big enterprises in mind and helps set data masking from a single location. That means the administrator can set the masking from a single place. It also supports a huge volume of data to mask, which is not possible with small solutions. It is again costly because it is an enterprise product. But Informatica offers a 30-day trial period. 

K2VIEW DATA PRODUCT PLATFORM

K2View’s Data Product Platform is one of the top data masking products on the market, and it does both static and dynamic masking. K2View not only masks traditional data but also records PDFs and images. In fact, it even masks the original image by blurring it. Because of the cost, it is most suitable for large organizations. 

DATPROF

DATPROF’s data masking tool has a state-of-the-art algorithm, which not only masks the data but can also generate a lot of dummy data from it. Besides traditional data, it also supports XML and CSV files. It has an easy-to-use interface and can create templates, which can be used later. The drawback of these templates is that they can be created on a Windows machine only. It does support a large number of records. 

ACCUTIVE DATA DISCOVERY AND MASKING

Accutive Data Discovery and Masking is a top tool that also does data discovery of sensitive data. This is done automatically and can use preconfigured keywords. Or keywords like “credit card” or “Social Security numbers” can be added by the administrator. Besides this feature, the masked data is consistent across multiple destinations. Like if the masking of Rohit is done to John in the development database, then it is the same in the testing database. Also, data can be moved between multiple kinds of databases. It can be moved from an Oracle database to a MySQL database, or from a flat file to a MySQL database. The UI is very easy to use, and they have one of the most cost-effective products.

CONCLUSION

In this post, we first discussed production data and non-production data, as well as the differences between them. Then we reviewed how to secure non-production data through the process of data masking. This process masks sensitive data from the users of non-production data. We also looked into the top tools available for data masking. 

AUTHOR

This post was written by Nabendu Biswas. Nabendu has been working in the software industry for the past 15 years, starting as a C++ developer, then moving on to databases. For the past six years he’s been working as a web-developer working in the JavaScript ecosystem, and developing web-apps in ReactJS, NodeJS, GraphQL. He loves to blog about what he learns and what he’s up to.

Posted on

What is Data Cloning? A Beginners Guide

What is Data Cloning

Data Cloning, sometimes called Database Virtualization, is a method of snapshotting real data and creating tiny “fully functional” copies for the purpose of rapid provisioning into your Development & Test Environments.

The Cloning Workflow

There are four primary Steps

  1. Load / Ingest the Source Data
  2. Snapshot the Data
  3. Clone / Replicate the Data
  4. Provision of the Data to DevTest Environments

Under the Hood

Cloning is typically achieved/built using ZFS or HyperV technologies and allows you to move away from the traditional backup & restore methods, which can take hours.

By using ZFS or HyperV you can provision databases x100 quicker and x10 smaller.

What is ZFS?

  • ZFS is a file system that provides for data integrity and Snapshotting. It is available for most if not all major OS platforms.

What is HyperV?

  • HyperV is a Microsoft virtualization platform that can be used to create and manage virtual machines. It supports Snapshotting as well.

Problem Statement

Backups are often taken manually and can take hours or days to complete. This means that the data isn’t available for use during this time period, which can be problematic if you need access to your data immediately.

There is also a secondary issue with storage. A backup & restore is, by its nature, a 100% copy of the original source. So if you started with a 5 TB database and wanted x3 restores then you are up for another 15 TB in disk space.

What are the Benefits of Data Cloning?

Data cloning is the process of creating a copy, or snapshot, of data for backup, analysis, or engineering purposes. This can be done in real-time or as part of a scheduled routine. Data clones can be used to provision new databases and test changes to production systems without affecting the live dataset.

Advantages

– Clones can be used for development and testing without affecting production data

– Clones use little storage, on average about 40 MB, even if the source was 1 TB

– The Snapshot & Cloning process takes seconds, not hours

– You can restore a Clone to any point in time by bookmarking

– Simplifies your End to End Data Management

Disadvantages

– The underlying technology to achieve cloning can be complex.

However, there are various cool tools on the market that remove this complexity.

What Tools are available to support Data Cloning?

In addition to building your own from scratch, commercial cloning solutions include:

Each is powerful and has its own set of features and benefits. The key is to understand your data environment and what you’re trying to achieve before making that final decision.

Common Use Cases for Data Cloning

  • DevOps: Data cloning is the process of creating an exact copy of a dataset. This can be useful for several reasons, such as creating backups or replicating test data, into Test Environments, for development and testing purposes.
  • Cloud Migration: Data cloning provides a secure and efficient way to move TB-size datasets from on-premises to the cloud. This technology can create space-efficient data environments needed for testing and cutover rehearsal.
  • Platform Upgrades: A large majority of projects end up going over the set schedule and budget. The primary reason for this is because setting up and refreshing project environments is slow and complicated. Database virtualization can cut down on complexity, lower the total cost of ownership, and accelerate projects by delivering virtual data copies to platform teams more efficiently than legacy processes allow.
  • Analytics: Data clones can provide a space for designing queries and reports, as well as on-demand access to data across sources for BI projects that require data integration. This makes it easier to work with large amounts of data without damaging the original dataset.
  • Production Support: Data cloning can help teams identify and resolve production issues by providing complete virtual data environments. This allows for root cause analysis and validation of changes to ensure that they do not cause further problems.

To Conclude

Data cloning is the process of creating an exact copy of a dataset (database). This can be useful for many reasons, such as creating backups or replicating data for development and testing purposes. Data clones can be used to quickly provision new databases and test changes to production systems without affecting the live dataset.

This article provides a brief overview of data cloning, including its advantages, disadvantages, common use cases, and available tools. It is intended as a starting point for those who are new to the topic. Further research is recommended to identify the best solution for your specific needs. Thanks for reading!