Data Engineering in the Agent Era
TL;DR: Large language models keep getting more capable, but a model that knows everything public still does not know your business. Every enterprise runs on proprietary knowledge, definitions, metrics, and judgments that exist in no training set. The people closest to that data, analytics and data engineering professionals, have a distinct job: turning that knowledge into structured, versioned context that models can use, and verifying that the answers are correct. It comes in three layers: build the data foundation, encode the meaning, and evaluate the behavior.
Why this matters now
Somewhere in your company, an agent is already answering questions about your business. A salesperson asks it which accounts are at risk. A finance analyst asks it for last quarter's margin by segment. It may already be answering customers directly.
The failure mode that deserves attention is not a broken dashboard. It is a fluent, confident, well-formatted answer that is wrong, and no one notices, because the model sounded certain and the number was plausible. The definition of "active customer" changed last quarter and the agent never received the update. The revenue figure it cited uses the old product hierarchy. The board deck goes out, and the error is discovered after the fact, by someone outside the company.
This is not a model problem. It is a grounding problem. The models are performing as designed; the enterprise has not yet done its part, which is to make what it knows legible and verifiable to machines.
The observation
AI research labs own general capability. Every year, models improve at language, reasoning, and synthesis across the public record. But a medium-to-large enterprise generates its own proprietary data and insight: definitions, metrics, and judgments that exist in no public training set and never will.
The enterprise task, then, is grounding: turning proprietary knowledge into structured context the model can use, and evaluated answers you can trust. That responsibility belongs to whoever is closest to the data's truth, because meaning that drifts from its artifacts will not stay true.
Three layers: from foundation to verifiable behavior
The framework is a three-layer model. Each layer has a different failure mode, and the layers only work as a system.
| Layer | What it is | Failure mode |
|---|---|---|
| 1 — Build the data foundation | Logging, architecture, pipelines, data quality, lineage — the artifacts that make enterprise data computable at all | Everything downstream guesses |
| 2 — Encode the meaning | The semantic layer: approved definitions, metric logic, grain rules, versioned as artifacts | Agents improvise meaning |
| 3 — Evaluate the behavior | Golden datasets, judges, and regression runs | Enrichment degrades unnoticed |
Taken together: data engineering is not disappearing. It is moving up the stack. As agents become consumers of enterprise data, the scarce skill is making data machines can understand, trust, and act on. The pipelines may increasingly be written by AI. Accountability for the data cannot be outsourced.
Who does this work, and why the title is the wrong question
It is tempting to assign this responsibility by job title. But "analytics professionals" is too broad, and "data engineers" is too narrow. The better criterion is proximity to artifact truth: the people who can say where a number came from, how it was transformed, and where it can mislead.
An analyst who authored the definitive churn definition owns its meaning. The data engineer who owns the pipeline owns its provenance. If you can answer those questions in your organization, this work is already yours, whether or not anyone has named it yet. The framework holds when the criterion is artifact truth, not headcount.
Beyond RAG
Retrieval-augmented generation (RAG) is one implementation of enrichment, not the framework itself. The RAG flow is familiar: a user asks a question, the system retrieves relevant text chunks from a corpus, those chunks are added to the prompt, and the model answers from the injected context. It is useful, and it is insufficient for enterprise grounding in three specific ways.
| RAG alone | Structured grounding (this framework) | |
|---|---|---|
| What is retrieved | Document chunks chosen by similarity search | Curated definitions, metric logic, grain rules, and lineage from a semantic layer |
| What it knows | "What did our docs say?" | "What is true about our business, and which governed computation proves it?" |
| Failure it misses | Wrong joins, stale definitions, double-counting — none visible to similarity search | Explicitly guarded by definitions and by evaluation against ground truth |
| Proof | Fluent answer with citations to retrieved chunks | Answer scored against golden answers; regressions block release |
Enrichment is a choice among mechanisms, and the tradeoffs are worth laying out plainly. The framework does not prescribe one mechanism; it sets the criterion: whichever keeps meaning tied to the artifacts it describes, and remains verifiable, is the right one for the job.
| Mechanism | How it supplies knowledge | Tradeoff |
|---|---|---|
| Fine-tuning | Baked into model weights | Opaque, and stale the day after training |
| Prompt injection | Context supplied at query time | Fragile, and limited by context window |
| Semantic layer | Governed computation — the metric is defined, not merely described | Requires maintained definitions |
| RAG | Retrieval over documents at query time | Misses wrong joins, stale definitions, double-counting |
The gap RAG does not close is proof. A RAG system can show that the model sounded right. Only evaluation — golden questions with known-correct answers, run continuously — can show that it was right. That is the bridge to the feedback loop.
In short: context is what you give the model; evaluation is how you know it still holds.
The feedback loop that keeps meaning honest
Enrichment decays. Insights are updated, definitions drift, new data lands. A one-time enrichment is a project; the loop below is what turns it into a maintained system. It exists because without step 3, step 1 decays silently: no one notices the agent answering with last quarter's revenue definition until a board deck is wrong.
| Step | What happens | "Active customer" example |
|---|---|---|
| 1 — Define | Write the approved definition — SQL, filters, grain rules — in the semantic layer | The approved definition (v3): its SQL, exclusions, and grain |
| 2 — Serve | Agents answer questions using those definitions | Agents answer "how many active customers?" from that definition, not from improvisation |
| 3 — Evaluate | Run the golden dataset continuously and on every change | 50–100 golden questions run on every definition or model change |
| 4 — Detect drift | Classify each failure: agent, semantic, or data drift | Failures classified as described in the drift section below |
| 5 — Feed back | Route to the owner; version the fix back into the layer | Model errors to the agent owner; stale semantics to the definition owner; data issues to the pipeline owner |
What "done" looks like: a dashboard the definition owners actually consult, with pass rates per metric, trend lines, and a changelog showing which definition version answered which question. And one standing rule: no definition ships without its golden questions, and no golden question passes without a versioned definition behind it. That rule is what makes the system maintainable rather than a one-off project.
Evaluation in practice: graders and runs
The golden dataset is built once and maintained indefinitely: 50 to 100 questions with known-correct answers, reviewed by the definition owners. It covers the straightforward questions and the difficult ones — "active customers in EMEA last quarter, excluding trial accounts," "month-over-month change under the Q2 definition" — along with deliberately ambiguous prompts ("how many customers do we have?") that test whether the agent asks a clarifying question or guesses. Each entry records the expected answer, the approved SQL or metric, and the definition version it was validated against.
| Grader | What it checks | Scale |
|---|---|---|
| Deterministic | Approved metric ID, filters, and grain — exact match | Every query; inexpensive |
| LLM-as-judge | Whether open-ended answers match the golden answer; calibrated against human reviewers so the judge itself does not drift | Sampled; moderate cost |
| Human spot-checks | A sample of production answers reviewed by the definition owners | Weekly; catches what automation misses |
| Run | Trigger | What it catches |
|---|---|---|
| Pre-deploy | Any change: model version, agent prompt, edited definition | Regressions — what used to pass and now fails |
| Scheduled | Nightly or weekly, sampling real production questions | Slow degradation and drift in live traffic |
| Shadow | New definitions run alongside old ones for a period | Divergence beyond threshold before the new definition is trusted |
Identifying drift in practice
Drift is how a correct system becomes a confidently wrong one. There are three distinct failure modes, each with its own signals and owner. Conflating them misdirects the response to the wrong owner.
| Drift | What changed | How you spot it | Owner |
|---|---|---|---|
| Agent drift | The model answers differently; nothing else changed | Golden-set regression with definitions and data unchanged; judge-score trendlines sliding on one metric | Agent / model owner |
| Semantic drift | The business changed what it means; the definitions did not follow | The golden answer is challenged rather than the agent's; cross-system discrepancies (finance vs. agent); product or org changes with a quiet definition changelog | Definition owner |
| Data drift | The ground shifted under correct definitions | Freshness SLAs, row-count and null-rate anomalies, distribution shifts; answer-level jumps traced to a late partition or broken job; lineage names the affected definitions | Pipeline owner |
Operating rule: every detection routes to an owner. Agent drift goes to the agent owner, data drift to the pipeline owner, semantic drift to the definition owner. An alarm that cannot be routed is not a detection system; it is a dashboard no one reads.
Closing
Nothing in this framework is speculative. Every component exists in practice: teams already maintain semantic layers, run golden datasets, and page pipeline owners on data drift. What is new is assembling the components as one system, with each layer owned by the people closest to its truth, and a loop that keeps the whole system honest over time.
If you are one of those people — if you can say where a number came from, how it was transformed, and where it can mislead — this work is already yours. The agents are not going away, and neither is the need for someone to be accountable for what they say about your business.
Further reading
- dbt Semantic Layer FAQs — what a semantic layer is and what it does: centrally defined metrics, dynamic SQL generation, APIs for downstream tools.
- Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (2020) — the original RAG paper; useful for seeing exactly what the retrieval-over-documents pattern does and does not claim.
- Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (2023) — the case for and the limits of using strong LLMs as evaluators, including the biases to calibrate out.