Pipeline architecture

core-data-updater is one native Dagster code location with two connected data layers:

providers -> nc-feeds -> Python flow operations -> raw
                                              \-> monitoring + repair debt
raw -> dbt sources -> facts -> analytics

nc-feeds owns provider calls, shaping, completeness evidence, and gaps. nc-data owns physical raw/config/monitoring models, repositories, landing policies, provisioning, and the Dagster/dbt compiler. This repository owns the five business flows, dbt semantics, workload selections, sensors, and repair routing. No raw table is rebuilt or duplicated by dbt.

Python-owned raw flows

Flow	Operational workloads	Full-state and repair workloads	Canonical responsibility
Metadata	`metadata_daily`, `metadata_weekly`, `metadata_reconcile`	`metadata_build`, `metadata_backfill`	exchanges, calendars, equity/ETF/index/crypto catalogs, CIK continuity, constituent windows, entity/ETF metadata, ETF holdings
Events	`events_update`	`events_build`, `events_backfill`	splits, dividends, symbol changes
Prices	market/crypto daily, intraday-core, and crypto-alignment jobs	`prices_build`, `prices_backfill`; full-intraday sweep jobs remain disabled pending object-storage release	daily bars, core intraday bars, audited crypto market-session alignment, coverage
Metrics	daily short data, current snapshots, crypto, short interest, FTD, estimates	`metrics_build`, `metrics_backfill`	valuation inputs, floats, estimates, short/borrow/FTD/threshold/rates, USD crypto metrics
Fundamentals	statement/filing updates plus document follow-up	`fundamentals_build`, `fundamentals_backfill`	CIK-preserving filings and documents, income statements, balance sheets, cash flows

Every flow subclasses CoreDataFlow, which enforces the shared runtime law: fail-closed live configuration, fresh step-local clients, fatal-before-write, exact-write completeness, post-gate empty refusal, scratch isolation, honest row receipts, durable gap recording, and terminal monitoring closure.

dbt-owned consumer layer

The root dbt/ project contains 43 models and three statement snapshots. Each model owns one relation and appears as one namespaced Dagster asset under core_data_updater/dbt/*.

Schema	Families	Contract
`facts`	completed universes and identity histories; calendars and daily constituents; native daily/core-intraday prices and actions; aligned crypto prices; valuation, short, borrow, rate, and USD crypto metrics; filings, PIT statements, and ETF holdings	canonical algorithm-facing relations; contracts, tests, and deterministic update/build/repair semantics
`analytics`	cross-family daily price alignment, daily universe view, financials-as-of	reusable cross-table views; feature engineering remains a downstream container responsibility
snapshots schema	income statements, balance sheets, cash flows	preserves knowledge-time revisions used by the PIT facts

The bridge is declared by concept classes in src/core_data_updater/dbt/jobs.py. One @dbt_assets graph is cropped into 25 named update, build, and repair jobs; readiness sensors bind raw or upstream dbt materializations to stable semantic targets. The Docker build copies the root project to /app/dbt, parses the manifest, and sets DBT_PROJECT_DIR=/app/dbt. Orchestration receives only the immutable image pointer.

Monitoring and self-healing

Full/fatal failures write no trusted partial state and use ordinary Dagster retry or reexecution. A successful partial additive landing writes structured gap debt with table, key, range, evidence, severity, and attempt state. The born-stopped core_data_repair_queue reads that durable debt and routes a bounded attempt to the existing owner job; it never creates a second business implementation. Only later successful, non-truncated call evidence closes the gap.

dbt updates similarly write transform-state receipts keyed by selection, semantic target, input marker, and manifest digest. Identical successful input is deduplicated; a failed receipt earns a stable retry.

Activation state

The shipped graph contains 200 assets, 56 jobs, 32 sensors, and 11 schedules. All business schedules, readiness sensors, and the repair queue are born STOPPED. The raw, dbt, and repair terminal success/failure sensors are born RUNNING so run ledgers close even for framework-level failures.

Production activation is deliberately staged: provision schemas/config rows, load and inspect the code location, run one manual full-state workload at a time, verify row/check/monitoring receipts, then enable only the intended operational sensors and schedules. Full-intraday Parquet publication stays off until its Garage/PQ storage path has its own release proof.

Evidence gates

The blocking evidence is layered:

Ruff, formatting, mypy, manifest parsing, and Dagster definition loading.
Adversarial in-memory simulations for all five flows, repair routing, dbt workload targeting, and cross-system parity.
Isolated PostgreSQL seams for COPY/transactions/scratch/reexecution and a non-empty dbt build proving late append stability, past corrections, idempotent replay, full-refresh convergence, statement snapshots, and typed fact reads.
Docker build plus gRPC health smoke before an immutable image may register with orchestration.