ETL or ELT for AI workloads?
Where to transform data when its consumers are models rather than dashboards — and why AI shifts the answer.
The ETL versus ELT argument was settled for analytics by cheap warehouse compute. AI workloads reopen it, because models want different things from data than dashboards do.
Why ELT won for analytics
Load raw, transform in the warehouse. You keep the original, you can rebuild any derived table when requirements change, and lineage is inspectable. For reporting this is straightforwardly better.
What AI adds to the old argument
Priority
- For dashboards
- Rebuildable derived tables
- For models
- Consistency between training and serving
Worst bug
- For dashboards
- A wrong number in a report
- For models
- Training-serving skew, invisible in testing
Transformation
- For dashboards
- Anywhere, if it is documented
- For models
- One definition, shared by both paths
Raw data
- For dashboards
- Useful to keep
- For models
- Essential — you will reprocess repeatedly
| For dashboards | For models | |
|---|---|---|
| Priority | Rebuildable derived tables | Consistency between training and serving |
| Worst bug | A wrong number in a report | Training-serving skew, invisible in testing |
| Transformation | Anywhere, if it is documented | One definition, shared by both paths |
| Raw data | Useful to keep | Essential — you will reprocess repeatedly |
What AI adds
Model inputs need consistency between training and serving above all else. A transformation that runs one way in a nightly batch and another way in a live request produces training-serving skew — a bug that is invisible in testing and quietly degrades production.
The property that actually matters
Not where transformation happens — that there is exactly one definition of each feature.
- 1
Land raw, immutable
Keep the untransformed source. Embedding models and chunking change; you will reprocess.
- 2
One definition per feature
Shared by training and serving. Written twice means training-serving skew.
- 3
Version the outputs
So a model can be tied to the exact data that produced it.
- 4
Make reprocessing routine
It should be a scheduled job, not an incident.
Transform once, use everywhere
The property that matters is not where transformation happens but that there is exactly one definition of each feature, used by both training and inference. Whether that lives in the warehouse or in a shared library matters less than that it is not written twice.
Keep the raw data
Whatever the pipeline shape, retain the untransformed source. Embedding models change, chunking strategies change, feature definitions change — and every one of those means reprocessing from source. Teams that only kept the processed form pay for that repeatedly.
- Raw landing zone, immutable and dated
- One transformation definition per feature, shared across paths
- Versioned outputs, so a model can be tied to the data that produced it
- Reprocessing that is routine rather than an incident