Data quality is the usefulness of a specific dataset towards a particular goal. It can be measured in terms of accuracy, completeness, reliability, legitimacy, uniqueness, relevance, and availability.
Whether you work in development, sales, management, or acquisition, utilizing quality data is essential for day-to-day business operations. At the same time, in most cases, you will not be ensuring data quality yourself. You will probably get access to a dataset prepared by the data analytics team or buy a cleaned dataset. I wrote this article to ensure that you know the vocabulary for ensuring data quality, tracking, the potential issues with data quality, and the questions to ask before you start looking for insights.
Data is abundant and comes in many forms (quantitative, qualitative) and formats (JSON, CSV, XML). However, obtaining and maintaining quality data can be challenging. Understanding data quality can help with this challenge.
Due to its vast usage, data quality has many definitions and can be improved in various ways. In this article, I will explore how data quality is defined, use cases for data quality, and six methods to improve your company's overall quality of data.
Key takeaways
If you don't have time to read the entire article (it's pretty comprehensive!), I've prepared a list of things you must remember to improve your data quality. While you will not do the data quality checks yourself in most cases, knowing what to discuss with a data analyst or data provider and how to approach the topic is essential.
Here are key topics to address before you start using a dataset:
- Data quality dimensions. Is the data accurate, complete, consistent, fresh, uniform, and unique? A high-quality dataset will check all of these categories.
- Data quality issues. Raw data might have problems, such as missing values, duplicates, incorrect information, or misaligned formats. Before using a dataset, you must address all of these issues.
- Dataset parameters. How many items are in the dataset, when were they collected, and what are the top categories? What are the highest and lowest values? Answering these questions will help you spot any inconsistencies in the dataset.
- Industry context. While the data must speak for itself, it will be heard in context. Before making any announcements, double-check similar sources to ensure your insights are unique but at least somewhat match the existing research.
Now that you have glimpsed the key topics discussed in the article, let's dive deeper. We'll begin by better understanding what data quality means.
What does data quality mean?
Depending on who you ask, data quality has many definitions. Data quality definition can be applied to three main groups of people: consumers, business professionals, and data scientists.
While the definitions change depending on their intended use, the core meaning of data quality stays relatively the same. I will touch on these core principles later in the article.
For now, let's look at the three most common definitions of data quality.
Data quality for consumers
Consumers understand data quality as data that fits its intended use, is kept secure, and explicitly meets consumer expectations.
Data quality for business professionals
Business experts understand data quality as data that aids in daily operations and decision-making processes. High data quality will enhance workflow, provide insights, and meet a business's practical needs.
Data quality for data scientists
Data scientists understand data quality on a more technical level, as data that fulfills its inherent characteristics of accuracy, usefulness, and satisfies its intended purpose.
Ultimately, these definitions of data quality are all united by their emphasis on purpose and accuracy. While these are important, many other dimensions can be used to measure data quality.
Now, let's examine the most common data quality issues.
What are the most common data quality issues?
Raw data often contains issues that can skew the insights. For instance, the analysis will provide incorrect insights if a dataset has many duplicates.
One of the most important tasks for the data analyst before starting analysis is to clean and organize the dataset. This not only gets rid of any issues, but it's also a chance for the analyst to get to know the dataset, understand its schema, and ensure that the data at hand provides the right type of data.
While we don't need to describe in detail how the analysts clean up the data, we can discuss the most common problems that might harm the data quality.
So, what are the most common issues with raw data?
Duplicates
The dataset might include multiple copies of the same data item. Sometimes, this is not a problem—for instance, you might see numerous entries about a customer who bought the same product multiple times. You are golden if you have unique order numbers connected to each order or a date that helps separate the purchases.
However, if you notice the same customer having multiple customer ID numbers, it could mean problems with data entry or order tracking. Reviewing the duplicates is crucial if you want to see the accurate picture.
Inaccurate data
An incorrect customer phone number might be inconvenient for the sales team, and a faulty product price might lead to financial loss. Making sure that the data is correct must be a priority—even if it might take some manual work to double-check the outliers.
Outdated data
People change emails, phone numbers, and job positions. Products get discounted or go out of stock. If not addressed in time, these changes can lead to real-life issues, so keeping the database up to date with fresh data becomes a priority.
Data entry errors
Typos can happen to the best of us. Entering the data manually invites human error, and catching every data issue is not always easy. While entry errors may not seem like a big deal, they become an issue once you start aggregating.
For instance, if you make too many mistakes entering the word "Biotechnology" into your dataset's industry field, you will get more industry categories than usual. Each typo would count as a unique category. At the same time, aggregating the data helps to catch these errors—it's easier to spot the outliers once they are listed in an aggregated table.
Missing data
Every dataset contains multiple data points; in many cases, not all are properly filled in. For example, you might have a dataset that contains customer name, contact information, and order value. If, for some reason, the order value is missing in 50% of the orders, your entire dataset will provide dubious results. Ensuring the data is complete is the key to better data quality.
Conflicting formats
If you sell your products in the US and Australia, they will be sold in dollars. However, USD and AUD have different values, and putting both of them with a $ sign will ruin your results. A big part of data cleaning is ensuring that every format (including currency, dates, units of measurement, and addresses) is unified before being analyzed.
Since we have already discussed some (not all!) common issues with data quality, let's move towards discussing data integrity.
Data integrity
Data integrity is a list of steps that ensure your data points don't change across the data pipeline, from collecting and cleaning to transforming and analyzing the dataset.
There are many opportunities when this could happen.
For example, a dataset could combine information from multiple sources, increasing the risk of duplicate data or conflicting formats. While analyzing, the analyst could make incorrect calculations or use wrong formulas, misinterpreting the data. Finally, data might get corrupted due to data governance issues if multiple departments can access and update the dataset without following consistent procedures.
Once you know that your data has integrity, it's possible to seek out ways to enrich it.
Data enrichment
A dataset might be clean and high-quality, but it could still lack some qualifications to improve it.
There are plenty of ways to enrich the dataset, including cleaning the dataset, adding new variables derived from data aggregation, text descriptions, calculating the temporal data:
- Clean the data. Cleaning helps to get more value from the same data. For instance, you will get long strings of text that you need to add into separate fields – for instance, adding customer's city into a new field instead of having it as a part of their address.
- Aggregate the data. I've already discussed deriving new values through calculation. For example, using a customer's purchase history, you could calculate their average purchase amount or lifetime value (CLV).
- Extract keywords from text. One of the best ways to utilize large language models is to get them to read the larger text descriptions and extract the keywords mentioned, adding even more important data points to the dataset.
- Add temporal data. Consider seeing additional values, such as running totals, that help to track customer behavior or seasonal trends.
Filling in these new elements is a great way to improve data completeness and accuracy.
Why is data quality important?
Overall, data is used to make informed decisions. Low-quality data provides incorrect insights, which can lead to the wrong action. While everyone makes mistakes occasionally, basing your company's budget, marketing strategy, or next year's goals on incorrect data will have significant consequences. There are many risks involved in low data quality.
Forbes cites that low data quality can negatively affect businesses' revenue, lead generation, consumer sentiment, and internal company health. Maintaining high data quality affects virtually every aspect of a company’s workflow, from business intelligence and product/service management to consumer relations and security.
Now, let’s look at the major use cases for data quality.
Use cases for data quality
Since we already discussed using the data across different departments, we should examine various use cases that show how high-quality data can impact your business.
Data must follow the same standard when it’s collected from both public (e.g., investment or company data) and internal sources (e.g., customer contacts or marketing leads)
These are just a few examples of how high-quality data can improve the daily decision-making process.
Lead enrichment for marketing and sales
Personalizing your communications is vital today. High-quality datasets about customer behavior, preferences, industry, and even changes in company structure can help marketing and sales teams customize their pitches.
Business intelligence for C-level executives
Collecting precise data about business performance helps make better strategic choices. Since business intelligence data usually comes from multiple sources and departments, ensuring that the result is a dashboard containing precise information is extremely important.
Competitive intelligence for product development
Keeping in touch with the latest innovations in the industry, including technologies, practices, and products the competitors create, is crucial for every business. Collecting and utilizing high-quality data here can mean that your business succeeds or becomes just another Blockbuster in the age of streaming.
Investment trends for venture capitalists
Of course, venture investors choose their next investment, combining data from various sources, keeping in mind industry trends, company performance, and even the personalities of the founding team. This is why keeping track of industry changes is crucial—you want to start investing in AI startups before they become the hottest commodity.
Now that we know the possible use cases for high-quality data, we can quickly examine the data management principles that guide the entire process.
Data management principles
The data management process includes many policies and activities that ensure high quality across the entire data pipeline. It all starts with data governance.
Data governance
Data governance creates standards and definitions for data quality, aiding in maintaining high data quality across teams, industries, and countries.
The rules and regulations that establish data governance originate from legislative processes, legal findings, and data governance organizations such as DAMA and DGPO. Generally speaking, data governance ensures data ownership and access across different departments and organizations.
Data standardization
Similar to data governance, data standardization involves organizing and inputting data according to negotiated standards established by local and international agencies.
Unlike data governance, which examines data quality management from a more macro and legal perspective, data standardization considers dataset quality on a micro level, including implementing company-wide data standards. This allows for further specification and accuracy in complex datasets.
Data cleansing
We will not go deep into the technical side of the data cleaning. The most crucial thing to remember is that it is a process in which data wrangling tools correct corrupt data, remove duplicates, and resolve empty data entries.
Ultimately, this process aims to delete data that is considered “dirty” and replace it with sound, clear, and accurate data.
Geocoding
Geocoding is the process of correcting personal data such as names and addresses to conform to international geographic standards.
Personal data that does not conform to geographical standards can create negative consumer interactions and miscommunications.
Data profiling
Data profiling is a process that examines and analyzes data from a particular dataset or database in order to create a larger picture (profile summary) for a particular entry, user, data type, etc.
Data profiling is used for risk management, assuring data quality, and analyzing metadata (that sometimes goes overlooked).
Measuring data quality
As I mentioned previously, measuring data quality can be an incredible feat since many dimensions are used to calculate a dataset's quality. Here are the primary dimensions used for data quality measures.
Data quality dimensions
There are six dimensions of data quality: accuracy, completeness, consistency, timeliness, uniformity, and uniqueness. All of them are listed and discussed in more detail below.
Data accuracy
Data accuracy refers to measuring data authenticity as defined by any restrictions the data collection tool has set.
For instance, inaccurate data can occur when someone reports either false data or an error that occurred during the input process.
Data completeness
Completeness measures the degree of known values for a particular data collection process. Incomplete data contains missing values throughout a particular dataset.
Missing data can skew data analysis results, cause inflated results, or render a particular dataset useless if it is severely incomplete. Each data point in the dataset should have a fill rate (a percentage of how much data is included in the column) to ensure that the insights are derived from complete data.
Data consistency
Data consistency refers to the measure of coherence and uniformity of data across multiple systems. Significantly, inconsistent data will contradict itself throughout your datasets and may need clarification about which data points contain errors.
Additionally, inconsistent data can occur when data is input by different users across different data entry systems.
Timeliness
Timeliness refers to the rate at which data is updated. Timely data is updated often and does not contain outdated entries that may no longer be accurate.
Data uniformity
Data uniformity measures the consistency of the measurement units used to record data. Data that is not uniform will have entries with different measurement units, such as Celsius versus Fahrenheit, centimeters to inches, etc.
Data uniqueness
Data uniqueness measures originality within a dataset. Specifically, it aims to account for duplicates within a dataset.
Uniqueness is typically measured on a percentage-based scale, where 100% data uniqueness signifies no duplicates in a dataset.
6 methods to help you improve your data quality
As previously mentioned, data is an essential component of business intelligence. Likewise, compiling enough data for your datasets is just as crucial as collecting quality data.
Evaluating datasets is a difficult but necessary task that sets you apart from competitors. Data quality issues can occur often during a data point's life cycle.
Nevertheless, creating clear guidelines and setting thoughtful intentions when analyzing your data will increase your data quality, allowing a more precise understanding of what your data is telling you.
With that, let's look at methods for improving data quality.
Collect unique data
Uniqueness refers to the specific type of data you are collecting. It is important to utilize data that is specific to your business objectives and matches the intentions behind your data usage.
For example, maybe your company wants to monitor its competitors. Simply put, you should ask yourself: "Is this data relevant to my company's established business goals?" If not, consider reassessing what specific data you are collecting.
Collect it frequently
Frequency surrounding data collection, also known as timeliness (discussed above), indicates how fresh your dataset's information is. Evaluating the frequency at which you collect new data or update the current information in your dataset will directly affect its quality.
The goal here is to establish recurring data collection cycles that support your business objectives.
Collect accurate data
While sometimes accurate data is not possible, due to the data collection process's human component, creating parameters on data intake reduces inaccuracies. For instance, analyzing datasets manually at random will give you insights into how accurately consumers are inputting data.
At first glance, some datasets may look complete, but that doesn't necessarily equate to accuracy.
Reduce noise in data
Noisy data can create unnecessary complexities and convoluted datasets.
As we already discussed, duplicates in your data or misspellings in entries cause errors in the data analysis process. Data matching, which compares individual data points with one another to find duplicates, misspellings, and excessive data (data that is not necessarily duplicated but implied in other data entry points), can reduce noise in data.
Identify empty values in your data
Incomplete or missing data can negatively affect the quality of your datasets and, in some cases, create more extensive data reading errors depending on which data profiling tools you are using. The larger the number of empty values, the more inaccurate your data becomes.
Therefore, ensuring completeness throughout the data collection process is essential to guaranteeing data quality.
Invest in data quality best practices
Similar to implementing best practices in the workplace, it is also critical to have best practices for your dataset collection process.
Communicating and interpreting your datasets consistently throughout your company will increase the quality in which your business utilizes said data. Establishing company-wide best practices for your dataset process will ensure consistency and quality.
Data quality checks: 3 data quality rules to keep in mind
Data quality is a broad topic that cannot be covered in one article, but I hope this text helps readers grasp the key concepts.
Still with me? Let's go through a quick data quality checklist that summarizes the article.
So, what should you keep in mind when discussing data quality?
Wrapping up
Data quality affects all facets of business operations. Poor-quality data results in inefficient operations and inaccurate insights, which could hurt your business instead of helping it.
Data quality management tools must be in place to sustain high-quality data. Customer data is critical. It changes constantly, and you should be the first to know about any changes in customer data to deliver relevant and appropriate pitches. Data quality tools will help you keep the data valid and valuable. Consistent data management is critical to successful data-driven business strategies.
Once you master data management, you will be able to reap the benefits of high-quality data. Obtaining and maintaining quality data is a priority in a successful business.
