There’s a saying in machine learning circles: “garbage in, garbage out.” It sounds obvious, almost dismissive. But after watching organizations pour resources into AI initiatives only to see them fail, we’ve come to believe this simple phrase captures the most important lesson in applied AI.
The model you choose matters less than you think. GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3, Mixtral—they’re all remarkably capable. The differences between them, while real, are often smaller than the difference between clean data and messy data. Yet teams consistently obsess over model selection while treating data quality as an afterthought. Before you even evaluate whether AI is the right approach, you need to assess whether your data can support it.
This is backwards. Let’s fix it.
Why Data Quality Trumps Model Choice
Consider a real scenario we encountered recently. A company wanted to build a customer support classifier—something to automatically route incoming tickets to the right department. They had access to GPT-4o and were excited about its capabilities.
Their training data consisted of two years of support tickets, each labeled with the department that ultimately handled it. Sounds reasonable, right?
Here’s what they discovered upon closer inspection:
- 15% of tickets had been mislabeled due to human error during the original routing
- Department definitions had changed nine months ago, but old labels weren’t updated
- Many tickets were labeled based on who was available, not who should have handled them
- Duplicate tickets existed with conflicting labels
They spent weeks trying different prompting strategies, fine-tuning approaches, and model configurations. Accuracy stubbornly plateaued around 71%. Then they cleaned their data—removed duplicates, corrected obvious mislabels, filtered out the period before the department restructure. With the same model and the same prompts, accuracy jumped to 89%.
The model was never the problem. The data was.
The Data Quality Checklist
Before you evaluate any AI model or technique, run your data through this checklist. It will save you months of frustration.
1. Accuracy: Is the Data Correct?
This seems obvious, but it’s routinely overlooked. Sample your data manually. Pick 100 random records and verify them against ground truth.
For labeled data, this means checking whether the labels are actually correct. If you’re training a sentiment classifier and 20% of your “positive” examples are actually negative, your model’s ceiling is already compromised.
For input data, check whether the values make sense. Are there impossible dates? Negative values where only positives are valid? Entries that are clearly data entry errors?
Practical step: Randomly sample 100-200 records. Manually review them. Calculate your data accuracy rate. If it’s below 95%, cleaning should be your top priority.
2. Completeness: What’s Missing?
Missing data isn’t just about empty fields. It’s about whether you have the signals you need to solve the problem.
A fraud detection model needs data about what fraud looks like—confirmed fraud cases with the patterns that preceded them. If your organization catches fraud but doesn’t document the indicators, you have incomplete data even if every field is filled in. This is one of the hidden costs of AI projects that organizations often discover too late.
Common completeness issues:
- Survivorship bias: You only have data on customers who stayed, not those who left
- Selection bias: Your data represents one segment but you want to apply the model broadly
- Temporal gaps: Missing data from certain time periods that may matter
- Feature gaps: The most predictive variables were never captured
Practical step: List the ideal features for your model. For each one, check: Do you have it? Is it reliably captured? Going back how far? Across which segments?
3. Consistency: Does the Same Thing Mean the Same Thing?
Data collected from multiple sources, over time, or by different people often has consistency problems. The same concept gets recorded different ways.
We’ve seen “California,” “CA,” “Calif.,” and “california” in the same state field. Dates stored as MM/DD/YYYY in one system and YYYY-MM-DD in another. Product categories that mean different things in different regions.
These inconsistencies create noise that models have to learn to ignore—or worse, that models learn as spurious patterns.
Practical step: For categorical fields, list all unique values. Look for near-duplicates, misspellings, and semantic overlaps. For numerical fields, check units and scales. A revenue field mixing dollars and cents will wreak havoc.
4. Currency: Is the Data Fresh Enough?
AI models learn patterns from historical data and apply them to current situations. If the patterns have changed but your training data hasn’t, the model will learn outdated relationships.
This is especially critical in fast-moving domains: consumer behavior, financial markets, technology trends. A model trained on 2023 customer data may be learning behavior patterns that no longer apply as AI tools have shifted how people search, shop, and communicate.
Practical step: Analyze whether the patterns in your oldest data match your most recent data. Look for distribution shifts in key variables. Consider whether major external events (economic changes, competitor actions, global events) might have altered the relationships you’re trying to model.
5. Relevance: Does the Data Actually Predict What You Care About?
Having lots of data doesn’t help if it’s not relevant to your prediction task. A classic mistake is training on data that’s correlated with the outcome for incidental reasons rather than causal ones.
If you’re predicting employee attrition and include “badge swipe frequency” as a feature, you might find it’s highly predictive. But if it’s predictive because people stop swiping in their final weeks (when they’ve already decided to leave), it’s not useful for early intervention.
Practical step: For each feature, ask: Could this plausibly cause or indicate the outcome? Or is the correlation likely spurious? Does it help with early prediction or only contemporaneous detection?
The Data Quality Improvement Process
Once you’ve assessed your data quality, here’s how to systematically improve it.
Step 1: Prioritize by Impact
You won’t fix everything at once. Focus on issues that most affect model performance.
Generally, the priority order is:
- Label accuracy (for supervised learning, wrong labels are the most damaging)
- Missing critical features (you can’t predict what you don’t have signals for)
- Systematic biases (these create models that fail silently in production)
- Inconsistencies and noise (these reduce accuracy but rarely break models entirely)
Step 2: Fix at the Source When Possible
Data quality problems usually originate somewhere in your data pipeline. A manual entry process with no validation. An integration that drops records. A definition that nobody documented.
Fixing these source problems prevents future data quality issues. Cleaning data retroactively is necessary but insufficient—you’ll just accumulate more dirty data.
Practical step: For each major quality issue, trace it back to its origin. Is there a process change, validation rule, or documentation update that would prevent recurrence?
Step 3: Build Validation Into Your Pipeline
Don’t assume data quality is maintained. Verify it continuously.
This means automated checks that run when data is ingested:
- Schema validation (correct data types, expected fields present)
- Range checks (values within expected bounds)
- Distribution monitoring (alert if statistical properties shift unexpectedly)
- Referential integrity (foreign keys match, relationships are valid)
Practical step: Implement at least basic validation checks before any data enters your AI training pipeline. Flag or quarantine records that fail validation rather than silently including them.
Step 4: Document What “Clean” Means
Data cleaning involves judgment calls. Is this edge case valid or an error? Should this outlier be included or excluded? How do you handle missing values?
These decisions should be documented, consistent, and reviewable. Different team members making different judgment calls introduces inconsistency.
Practical step: Create a data dictionary that defines valid values, expected ranges, handling of missing data, and rules for resolving ambiguities. Treat this as living documentation that evolves.
Step 5: Invest in Labeling Infrastructure
If your AI approach requires labeled data, the labeling process often determines data quality more than anything else. This is especially critical when building your first AI proof of concept, where early data decisions set the foundation for everything that follows.
Good labeling requires:
- Clear, unambiguous instructions with examples
- Multiple labelers for ambiguous cases (to measure agreement and catch errors)
- Quality audits on labeled data
- Feedback loops so labelers learn from mistakes
Practical step: If you’re labeling training data, start with a pilot batch of 100-200 examples. Have multiple people label them independently. Measure agreement. The cases where labelers disagree often reveal ambiguities in your task definition that need resolution.
When to Stop Cleaning
Data quality improvement has diminishing returns. At some point, further cleaning yields minimal model improvement while consuming significant resources.
How do you know when you’ve reached that point?
Track the relationship between data quality and model performance. Clean a batch of data, retrain the model, measure improvement. As improvements shrink, you’re approaching the point of diminishing returns.
Consider the cost of errors. In high-stakes domains (medical diagnosis, financial fraud), higher data quality standards are justified. For recommendation systems where errors are annoying but not catastrophic, “good enough” might be sufficient.
Leave room for iteration. You don’t need perfect data to start. You need sufficient data to get meaningful signal. You can improve data quality iteratively as you learn what matters.
The Culture Shift
Data quality isn’t just a technical problem—it’s an organizational one. The companies that excel at AI have cultures that value data quality at every level.
This means:
- Data quality metrics that are tracked and visible
- Clear ownership of data assets
- Incentives aligned with data quality, not just data volume
- Investment in data infrastructure, even when it’s not glamorous
It also means resisting the temptation to jump to the exciting model-building phase before the unglamorous data-preparation phase is complete.
The organizations we see succeeding with AI in 2025 aren’t necessarily using the most sophisticated models. They’re the ones who treated data quality as a first-class concern, invested in their data foundations, and only then turned their attention to models and algorithms.
The model can wait. Get your data right first.
For guidance on the mistakes that derail AI projects (including underestimating data work), see our article on Seven AI Implementation Mistakes That Sink Projects.