Data pipelines & cleanup
Getting your data into one reliable place, in one consistent shape.
This is the unglamorous work that everything else on this page quietly depends on, and it is the item most often skipped.
A pipeline collects your data on a schedule, fixes the predictable problems — the same customer spelled three ways, four date formats, prices stored as text — and puts the result somewhere stable that everything else reads from.
It is boring and it is usually the highest-value project on the list. Automation built on messy data automates the mess. A prediction model trained on it learns the mess. A dashboard makes the mess look authoritative, which is arguably the worst outcome of the three.
"Ali Trading", "Ali Trading Sdn Bhd" and "ali trading s/b" are three customers as far as your system is concerned. Every report starts with half an hour of tidying, and two departments quote different revenue for the same month.
One customer, one record, one number. The tidying stops happening because there is nothing left to tidy, and the arguments about whose figure is right stop with it.
What actually goes wrong with business data
The problems are unglamorous and almost universal, and naming them makes it obvious why this work comes first.
The same customer exists several times. "Ali Trading", "Ali Trading Sdn Bhd" and "ali trading s/b" are three separate customers to your system, which means their true combined value is invisible, and any model or report treats them as three small accounts rather than one significant one.
Dates arrive in incompatible formats. One system writes 03/04/2026, another 2026-04-03, a third "3 Apr". Sorted together without being normalised, they interleave nonsensically and every time-based report is quietly wrong.
Numbers are stored as text, so they sort alphabetically — 1, 10, 100, 2 — and totals silently exclude anything with a stray space or currency symbol. Product codes drift, with the same item appearing under three SKUs after a supplier change.
None of this is anyone's fault. It is what happens when systems are added one at a time over a decade by busy people. But every one of these issues gets inherited by whatever you build next, which is why cleaning up first is usually the highest-return decision available.
Why this is the first project more often than not
Clients are frequently disappointed when we suggest starting here, because it is not the exciting part. The reason is straightforward: everything else on this site fails on messy input, and each fails in a different and expensive way.
Automation built on inconsistent data automates the inconsistency, faster and at larger scale. A prediction model trained on it learns the mess as if it were signal. A dashboard built on it presents contradictory numbers with total visual confidence, which is arguably the worst of the three, because people act on it.
The compounding benefit is that this work is not specific to any one project. Once orders, stock and customers are consistent in one place, the chatbot, the churn model and the dashboard all become materially cheaper and faster to build — and they finally agree with each other.
Where this goes wrong
If bad data is being created because two branches enter orders differently, a pipeline cleaning it up afterwards is a permanent tax. Fix the entry point where you can; clean up only what you genuinely cannot.
Every decision — which duplicate wins, how a missing date is treated — must be visible and reversible. Silent cleaning is how people stop trusting the numbers entirely.
How we actually build it
-
01
We find where the truth lives
Usually several places that disagree. Deciding which one wins is a business decision, not a technical one, and you make it.
-
02
We fix the predictable problems
Deduplicate, standardise, validate. The rules are written down, so you can see exactly what was changed and why.
-
03
We schedule it
Nightly is right for most SMEs. It runs, it logs, and it tells someone if a source stops arriving.
-
04
We leave one place to read from
Everything downstream — reports, models, automations — points at the same table. Reports stop disagreeing with each other.
- The same customer exists three times, spelled differently
- Your data lives in five places and none of them agree
- Every report starts with half an hour of manual tidying
- One small spreadsheet that one person maintains. Not worth the machinery
- The mess is a process problem — fix how data is entered before automating around it
- Read access to each source system
- Someone who can settle which source wins when they disagree
2–4 weeks, depending on how many sources and how bad the mess is.
What people ask about this
Can we skip this and go straight to the AI part?
Sometimes. Often it is why the AI part underperforms. We will tell you honestly which situation you are in — it is not in our interest to sell you a model that will disappoint.
Where does the data live afterwards?
Wherever you want it — your own server, your existing database, or a managed one. It stays yours either way.
Often paired with
Would this work for you?
The first call is free, and we will say plainly if this is the wrong tool for your problem.
Book a free call