Chapter 4: The Four-Layer Architecture #

"All problems in computer science can be solved by another level of indirection." -- David Wheeler

25 min read | Priya, Marcus, Sarah | Part II: The Architecture

What you'll learn:

• Why a layered architecture outperforms monolithic agent systems
• The four layers: Infrastructure, Platform Intelligence, Analytical Intelligence, and Orchestration
• How all 14 agents map to their respective layers
• How data flows through layers via async pub/sub, an artifact registry, and an event bus
• The communication layer that binds everything together

The Problem #

It is Tuesday morning and Priya's team is debugging a production outage. The churn prediction model is returning stale scores. The root cause? A schema change in the upstream OLTP system broke the ETL pipeline, which silently dropped 12 columns from the feature table. The data scientist's model kept scoring on the old features. The monitoring dashboard showed green because it only checked row counts. The governance team had no idea the PII column date_of_birth was now flowing unmasked into the feature store.

Four different teams. Four different tools. Four different dashboards. Zero coordination.

This is not a people problem. It is an architecture problem. When every component operates in isolation, failures cascade invisibly. When a schema change at the infrastructure layer cannot signal the analytical layer, you get silent model degradation. When governance checks run in a silo, you get compliance violations discovered in audits, not in pipelines.

The solution is not to wire together more point-to-point integrations. That path leads to the n-squared connectivity problem -- with 14 agents, that would be 182 possible direct connections, each one a maintenance burden and a potential failure point.

The solution is layers.

Why Layers, Not a Monolith #

Before we introduce the architecture, let us address the obvious question: why not build one large agent that does everything?

The monolithic agent approach is tempting. A single LLM call with a massive system prompt containing all domain knowledge, all tool bindings, all decision logic. Some teams try this. Here is what happens:

The layered approach solves each of these problems through separation of concerns, bounded context, and well-defined interfaces between layers.

The Four Layers #

The Intelligent Data Organization is organized into four horizontal layers, each building on the one below it. Here is the complete architecture:

Each layer has a clear responsibility:

Layer 1: Data Infrastructure #

Layer 1 is the foundation. It handles the physical reality of data -- where it lives, how it moves, and what shape it takes. Without a solid infrastructure layer, everything above it is building on sand.

The Agents #

Data Agent -- The source and schema manager. It declares typed schema contracts, manages source connections (PostgreSQL, S3, Kafka, APIs), configures sinks, defines quality gates, and routes computation to the appropriate engine. Every data movement starts with a Data Agent declaration.

ETL Agent -- The warehouse builder. It handles dimensional modeling (Kimball star schemas, Inmon, Data Vault 2.0), semantic layer definitions, SQL-first transformations, and multi-dialect SQL transpilation. When the Data Agent says what data exists, the ETL Agent decides how to transform and load it.

Migration Agent -- The platform mover. When organizations change data platforms (Oracle to Snowflake, on-prem to cloud), the Migration Agent handles wave planning, schema translation, data reconciliation, and zero-downtime cutover strategies.

Infrastructure Profiles -- Not an agent, but a critical Layer 1 component. Profiles abstract platform-specific details so the same Neam program runs on PostgreSQL today and Snowflake tomorrow without code changes.

flowchart TD
  PG["PostgreSQL (OLTP)"]
  S3["S3 / GCS (files)"]
  KF["Kafka (streaming)"]
  API["REST APIs (external)"]

  subgraph DA ["Data Agent"]
    DA1["Schema validation"]
    DA2["Quality gate >95%"]
    DA3["PII classification"]
    DA4["Source registration"]
    DA1 --> DA2 --> DA3 --> DA4
  end

  subgraph ETL ["ETL Agent"]
    E1["Staging transforms"]
    E2["Dimensional modeling"]
    E3["Feature engineering"]
    E4["Lineage tracking"]
    E1 --> E2 --> E3 --> E4
  end

  subgraph DW ["Data Warehouse / Feature Store"]
    DW1["Star schema (dims + facts)"]
    DW2["Feature tables"]
    DW3["Aggregate tables"]
  end

  PG --> DA
  S3 --> DA
  KF --> DA
  API --> DA
  DA --> ETL
  ETL --> DW

Layer 2: Platform Intelligence #

Layer 2 keeps the platform running, compliant, and understandable. While Layer 1 handles data movement, Layer 2 handles data governance, observability, and metadata intelligence.

The Agents #

DataOps Agent -- The SRE for data. It monitors pipeline health, detects anomalies in data volumes and latency, performs automated triage when pipelines fail, manages SLA tracking, and can auto-heal certain classes of failures (like restarting a stalled ingestion job). Think of it as the on-call engineer who never sleeps.

Governance Agent -- The compliance officer. It classifies data (PII, PHI, financial), enforces access policies (RBAC/ABAC), tracks column-level lineage, validates regulatory compliance (GDPR, CCPA, DORA, BCBS 239), and maintains audit trails. Every data access, transformation, and model prediction is logged.

Modeling Agent -- The data architect. It reverse-engineers existing schemas, proposes ER models, analyzes normalization levels (1NF through BCNF), designs dimensional models (star, snowflake, vault), and suggests schema amendments when new requirements arrive.

Analyst Agent -- The query engine for humans. It translates natural language questions into SQL across 9 dialects (PostgreSQL, Snowflake, BigQuery, Databricks SQL, Redshift, Oracle, Teradata, Trino, DuckDB), executes queries through governed channels, and delivers results in multiple formats (Excel, PDF, HTML, Slack).

flowchart TD
  PM["Pipeline Metrics"]
  SC["Schema Catalog"]
  DWS["DW Schema"]

  subgraph DO ["DataOps Agent"]
    DO1["Anomaly detection"]
    DO2["SLA tracking 99.5%"]
    DO3["Auto-heal: restart/retry"]
    DO4["Escalation to human"]
    DO1 --> DO2 --> DO3 --> DO4
  end

  subgraph GOV ["Governance Agent"]
    G1["PII/PHI classification"]
    G2["Access policy enforcement"]
    G3["Column-level lineage"]
    G4["Audit trail logging"]
    G1 --> G2 --> G3 --> G4
  end

  subgraph MOD ["Modeling Agent"]
    M1["ER analysis"]
    M2["Normalization check"]
    M3["Amendment proposals"]
  end

  subgraph ANA ["Analyst Agent"]
    A1["NL2SQL"]
    A2["9 dialects"]
    A3["Governed execution"]
  end

  PM --> DO
  SC --> GOV
  DWS --> MOD
  DWS --> ANA

Layer 3: Analytical Intelligence #

Layer 3 is the intellectual powerhouse of the system. It is where business requirements are formalized, hypotheses are tested, models are built, causation is established, quality is validated, and models are deployed to production.

The Agents #

Data-BA Agent -- The requirements analyst. It assists with LLM-powered elicitation, generates Business Requirements Documents (BRDs), produces Given/When/Then acceptance criteria, builds traceability matrices, and performs impact analysis. BABOK v3 alignment ensures industry-standard requirements engineering.

DataScientist Agent -- The modeler. It handles problem framing, hypothesis testing, EDA-driven technique selection, volume-aware compute routing (small data to scikit-learn, medium to XGBoost, large to Spark ML), feature engineering, ML/DL/NLP pipelines, and SHAP-based explainability.

Causal Agent -- The "why" engine. While the DataScientist Agent answers what will happen, the Causal Agent answers why it happens. It implements Pearl's Ladder of Causation -- association (Rung 1), intervention via do-calculus (Rung 2), and counterfactual reasoning (Rung 3). It uses Structural Causal Models, Bayesian inference (PyMC), and causal discovery (DoWhy).

DataTest Agent -- The independent critic. It generates tests from acceptance criteria, validates ETL output, checks data warehouse consistency, validates ML model performance, tests API endpoints, and enforces quality gates. Critically, the DataTest Agent is architecturally separated from the builder agents. The agent that builds the model cannot be the agent that validates it.

MLOps Agent -- The production guardian. It handles 6 types of drift detection (data, concept, prediction, feature, label, upstream), deployment strategies (canary, shadow, blue-green, A/B), champion-challenger model management, automated retraining triggers, and serving infrastructure.

flowchart TD
  BN["Business Need: Predict churn"]

  subgraph BA ["Data-BA Agent"]
    BA1["Requirements elicitation"]
    BA2["BRD + acceptance criteria"]
    BA3["Traceability matrix"]
    BA1 --> BA2 --> BA3
  end

  subgraph DS ["DataScientist Agent"]
    DS1["EDA on feature tables"]
    DS2["XGBoost model AUC=0.847"]
    DS3["SHAP: top 5 drivers"]
    DS4["Model registered in MLflow"]
    DS1 --> DS2 --> DS3 --> DS4
  end

  subgraph CA ["Causal Agent"]
    CA1["SCM: churn DAG"]
    CA2["ATE: support quality"]
    CA3["Counterfactual analysis"]
    CA1 --> CA2 --> CA3
  end

  subgraph DT ["DataTest Agent"]
    DT1["47 test cases generated"]
    DT2["45/47 passed (94%)"]
    DT3["Quality gate: PASSED"]
    DT1 --> DT2 --> DT3
  end

  subgraph ML ["MLOps Agent"]
    ML1["Canary deploy 10% traffic"]
    ML2["Drift monitoring configured"]
    ML3["Champion-challenger active"]
    ML1 --> ML2 --> ML3
  end

  BN --> BA --> DS --> CA --> DT --> ML

Layer 4: Orchestration #

Layer 4 is a single agent -- the Data Intelligent Orchestrator (DIO) -- but it is the most complex component in the system. It receives a high-level task ("predict customer churn and identify retention drivers"), decomposes it into sub-tasks, selects the right agents, forms a crew, delegates with RACI accountability, manages execution state, handles errors, and synthesizes the final result.

We dedicate an entire chapter to the DIO (Chapter 6), but here is how it fits architecturally:

flowchart TD
  UT["User Task: Predict churn + identify drivers"]
  S1["1. Task Understanding"]
  S2["2. Pattern Selection"]
  S3["3. Crew Formation"]
  S4["4. RACI Delegation"]
  S5["5. Execution Management"]
  S6["6. Error Recovery"]
  S7["7. Result Synthesis"]

  UT --> S1
  S1 -->|"Intent: prediction + causation"| S2
  S2 -->|"Pattern: churn_prediction"| S3
  S3 -->|"Score by capability, cost, history"| S4
  S4 -->|"BA=Responsible, DS=Accountable"| S5
  S5 -->|"Sequential with checkpoints"| S6
  S6 -->|"Retry → fallback → escalate"| S7

The key architectural insight is that Layer 4 does not do any data work itself. It does not write SQL, train models, or run tests. It coordinates the agents in Layers 1-3 that do. This separation means the DIO's LLM context is focused entirely on coordination logic, not domain-specific work. It stays small, fast, and accurate.

The Communication Layer #

Layers do not communicate through direct method calls. That would create tight coupling and make the system brittle. Instead, three communication mechanisms bind the layers together:

flowchart TD
  subgraph EB ["Event Bus"]
    EV1["schema.changed"]
    EV2["pipeline.failed"]
    EV3["model.trained"]
    EV4["quality.gate.failed"]
  end

  subgraph AR ["Artifact Registry"]
    AR1["BRD v1.2 (Data-BA)"]
    AR2["Feature Table (ETL Agent)"]
    AR3["Trained Model (DataScientist)"]
    AR4["Test Report (DataTest)"]
  end

  subgraph PS ["Async Pub/Sub Channels"]
    PS1["ETL Agent → DataTest Agent"]
    PS2["DataScientist → MLOps Agent"]
    PS3["DataOps Agent → DIO"]
    PS4["Governance → All agents"]
    PS5["DataTest → DIO"]
  end

  EB --> AR
  AR --> PS

1. Async Pub/Sub Channels #

Agents publish messages to typed channels. Other agents subscribe to channels they care about. The ETL Agent publishes feature_table.ready when a new feature table is loaded. The DataScientist Agent subscribes to that channel and begins model training when it receives the message. No direct coupling between the two agents.

2. Artifact Registry #

Every agent produces versioned artifacts: BRDs, feature tables, trained models, test reports, deployment configs. These artifacts are registered with metadata (producer, timestamp, version, lineage). Downstream agents consume artifacts by reference, not by direct data passing. This means an artifact produced on Tuesday is still available for audit on Friday.

3. Event Bus #

System-level events flow through the event bus: schema changes, pipeline failures, quality gate violations, model drift alerts. Any agent can subscribe to any event type. When the DataOps Agent detects a pipeline failure, it publishes pipeline.failed to the event bus. The DIO subscribes, initiates error recovery, and potentially triggers the ETL Agent to retry.

Data Flow Through the Layers: An End-to-End Example #

To make the architecture concrete, let us trace a churn prediction task from business question to production deployment:

flowchart TD
  subgraph L4 ["Layer 4: Orchestration"]
    DIO["DIO — forms crew, assigns RACI"]
  end

  subgraph L3a ["Layer 3: Requirements"]
    BA["Data-BA Agent — BRD + 12 criteria"]
  end

  subgraph L2a ["Layer 2: Governance"]
    GOV["Governance Agent — PII identified"]
  end

  subgraph L1 ["Layer 1: Infrastructure"]
    ETL["ETL Agent — 47-col feature table (PII masked)"]
    DOPS["DataOps Agent — pipeline health (all green)"]
  end

  subgraph L3b ["Layer 3: Analytics"]
    DS["DataScientist — XGBoost AUC=0.847"]
    CA["Causal Agent — DAG + ATE"]
    DT["DataTest — 45/47 passed, PASSED"]
    ML["MLOps — canary deploy 10%"]
  end

  subgraph L4b ["Layer 4: Synthesis"]
    SYN["DIO — executive summary + artifacts"]
  end

  DIO --> BA
  DIO --> GOV
  BA --> ETL
  GOV --> ETL
  ETL --> DOPS
  ETL --> DS
  DS --> CA
  CA --> DT
  DT --> ML
  ML --> SYN

Notice how the flow crosses layers. The DIO (Layer 4) delegates to agents in Layers 1, 2, and 3. The Governance Agent (Layer 2) provides constraints that the ETL Agent (Layer 1) enforces. The DataTest Agent (Layer 3) validates artifacts produced by both Layer 1 (feature tables) and Layer 3 (models). Layers are not strict hierarchies -- they are responsibility boundaries with well-defined interfaces.

The 14 Agents at a Glance #

Here is the complete agent-to-layer mapping:

Why This Layering Matters in Practice #

The four-layer architecture is not an academic exercise. It solves three concrete problems that plague real data organizations:

1. Independent Evolution #

When your organization migrates from PostgreSQL to Snowflake, only Layer 1 changes. The Migration Agent handles schema translation, the Infrastructure Profile switches from platform: "postgres" to platform: "snowflake", and the ETL Agent adjusts its SQL dialect. Layers 2, 3, and 4 remain untouched. The DataScientist Agent does not care where the feature table lives -- it consumes an artifact from the registry.

2. Blast Radius Containment #

When a Layer 1 pipeline fails, the DataOps Agent (Layer 2) detects it, the DIO (Layer 4) is notified via the event bus, and the error is contained. The DataScientist Agent does not receive a corrupted feature table -- the quality gate blocks it. Failures propagate up through well-defined channels, not through silent data corruption.

3. Composable Complexity #

Simple tasks use fewer layers. A data quality audit uses only Layers 1 and 2 (Data Agent + DataOps + Governance). A full churn prediction uses all four layers. The architecture scales with the complexity of the task, not with the number of agents deployed.

Industry Perspective #

The four-layer architecture aligns with established enterprise patterns:

DAMA-DMBOK 2.0 defines 11 knowledge areas for data management. Our four layers map naturally: Layer 1 covers Data Architecture and Data Integration, Layer 2 covers Data Governance, Data Quality, and Metadata Management, Layer 3 covers Data Science and Business Intelligence, and Layer 4 covers the orchestration that DAMA-DMBOK acknowledges but does not prescribe.

Microsoft's Modern Data Platform architecture uses a similar layering: Ingest -> Store -> Prep/Train -> Model/Serve. Our architecture adds governance and orchestration as first-class layers rather than cross-cutting concerns handled ad hoc.

Databricks Lakehouse separates storage, compute, and governance. Our architecture extends this with an analytical layer and an orchestration layer that coordinates across all components autonomously.

The key differentiator is that in traditional architectures, orchestration is a human responsibility -- project managers, scrum masters, and team leads coordinate the work. In the Intelligent Data Organization, orchestration is an agent's responsibility, guided by human-authored specifications.

The Evidence #

In the DataSims experiments, the layered architecture was tested against two alternatives:

Ablation A1 (No DIO -- flat agent system): Without the orchestration layer, agents must self-coordinate. Result: task completion rate dropped from 100% to 45%. Agents duplicated work, missed dependencies, and produced inconsistent artifacts.

Ablation A8 (No RACI): With the DIO but without RACI delegation, traceability dropped 80%. Agents completed tasks, but nobody could determine which agent was responsible for which output. Audit trail integrity collapsed.

Coordination Mode C1 (Swarm): Removing strict layer boundaries in favor of emergent coordination produced faster initial results but 40% lower consistency across repeated runs.

The evidence supports a clear conclusion: layers with explicit orchestration outperform both flat agent systems and emergent coordination for enterprise data workloads where auditability and consistency matter.

Key Takeaways #

• The four-layer architecture (Infrastructure, Platform Intelligence, Analytical Intelligence, Orchestration) provides separation of concerns that enables independent evolution, blast radius containment, and composable complexity.
• All 14 agents map to specific layers based on their responsibility, not their implementation technology.
• Three communication mechanisms (async pub/sub, artifact registry, event bus) connect layers without tight coupling.
• The DIO in Layer 4 coordinates but never performs data work itself -- keeping its LLM context focused on orchestration logic.
• Layering is not overhead. DataSims experiments show that removing layers (flat agent system) drops task completion from 100% to 45%.

For Further Exploration #

• Neam Architecture Reference -- Deep dive into the compilation pipeline and VM architecture
• DataSims Repository -- Run the experiments yourself and see the layered architecture in action
• DAMA-DMBOK 2.0, Chapter 1: Data Management -- Industry framework for data management knowledge areas