core_data_updater.jobs.metrics

jobs/metrics.py — MetricsFlow: the measurement streams.

Eleven census tables off distinct publication clocks, so the flow is split by whether work is historically addressable or only observable now — a single nightly job would either fire uselessly (short interest publishes twice a month) or land stale (the daily CDN files):

UPDATE  metrics_daily        short_volume · threshold_flags — exact
                             session partitions with table-date catch-up
UPDATE  metrics_snapshots    market_caps · floats · benchmark_rates ·
                             ib_shortable — current-only observations,
                             retried until the whole job succeeds
UPDATE  metrics_short_interest  short_interest — FINRA posts a new
                             settlement twice a month on no promised day, so
                             the sensor fires on a NEW PARTITION appearing
UPDATE  metrics_ftd          fails_to_deliver — SEC posts a half-month file
                             ~2 weeks lagged on its OWN schedule, so this is
                             a daily walk-back, not FINRA's signal
UPDATE  metrics_estimates    analyst_estimates — the quarterly consensus
BUILD   metrics_build        full history over the stored roster — manual,
                             max_concurrent=1, exact scoped/full replacement
BACKFILL metrics_backfill    nightly catch-up on the named gaps — 10:30 UTC,
                             born STOPPED

Cadences are read off the SOURCE, and every number here is a live receipt taken 2026-08-17, not a guess: market_caps FMP market-capitalization-batch, 1000 symbols/call (1500 → HTTP 414). 23,834 roster → 20,356 rows in 2.8s floats FMP bulk page-walk, 45,646 rows short_volume FINRA CDN, one file per SESSION — 12,195 rows (Fri 08-14); weekend/holiday answers 403 and is a CORRECT empty threshold_flags FINRA daily list, sparse — 6 rows short_interest FINRA partitions — 3,904,106 rows on a cold full pull fails_to_deliver SEC half-month zip — 73,337 rows benchmark_rates FMP treasury + fed funds — 48 rows on a 5d window analyst_estimates FMP per-symbol — 117 rows (AAPL)

Laws carried here, identical to prices/events/metadata: configs read LIVE per step (per-step rows; enabled comes back VISIBLE so a disabled row is a NAMED skip); every landing through CoreDataRepo.land on the METRICS policies; state-as-params (the roster is READ from raw.equities ∪ raw.etfs and PASSED; the landed settlement/half dates are READ via repo.landed_dates and PASSED — the feed stays stateless); the run ledger opens idempotently at the first landing; fulfill-or-name; and ZERO SQL, wire-parsing, or conversion in this file — every one of those lives in nc-feeds or nc-data.

The FATAL law (2026-08-17): _land checks result.has_fatal and kills the step. A feed goes fatal on exactly three things — the vendor hit never worked, the response could not be shaped, or our own pipeline raised. Routine attrition never stops a run.

Classes

`MetricsFlow`

Bases: CoreDataFlow

The metrics flow — METRICS_FLOW.md §5, method for method.

`backfill_ftd`

backfill_ftd(cfg) -> None

Walk the configured number of recent SEC half-month partitions.

`backfill_market_caps`

backfill_market_caps(cfg, fetch_roster: dict) -> None

Re-run the cheap market-cap batch over the active roster.

`backfill_rates`

backfill_rates(cfg) -> None

Re-run the benchmark-rate trailing correction window.

`backfill_short_interest`

backfill_short_interest(cfg) -> None

Re-run the FINRA partition diff over the full landed-date state.

`build_estimates`

build_estimates(cfg, fetch_roster: dict) -> None

Consensus back to 1995 through future periods, per symbol.

`build_floats`

build_floats(cfg, fetch_roster: dict) -> None

Float HISTORY — the ONE sanctioned legacy (v4) call, hard floor 2021-05-18. Updates ride the stable bulk walk, never this.

`build_ftd`

build_ftd(cfg) -> None

FTD half-month files across the window (halves before 2009-07 live on unmapped legacy paths — the policy floor says so).

`build_market_caps`

build_market_caps(cfg, fetch_roster: dict) -> None

Deep per-symbol mcap history (1990s depth; ≤5000 rows/call so the feed walks backward window by window). The go-forward leg is the batch endpoint — this is the one-time floor.

`build_rates`

build_rates(cfg) -> None

The full benchmark-rate series.

`build_short_interest`

build_short_interest(cfg) -> None

Full consolidated SI from the proven 2017-12-29 floor — quarterly window chunks, each offset-paged to completion. 3.9M rows live.

`build_short_volume`

build_short_volume(cfg) -> None

One CDN file per weekday across the window (floor ~2018-08). Holiday 404s are counted, never gapped.

`build_threshold`

build_threshold(cfg) -> None

Threshold-list membership days from 2016-01-04 — sparse by nature.

`fetch_roster`

fetch_roster(cfg) -> dict

The stored universe (raw.equities ∪ raw.etfs), read ONCE and passed — {“all”: active ∪ delisted, “active”: active only}. market_caps rides a ROSTER endpoint, so this is the step that decides what gets measured.

A FAILED roster read RAISES: a metadata outage must block loudly, never masquerade as an empty universe and land a green run measuring nothing. Opens the run ledger (the prices fetch_universe precedent).

`metrics_daily_ready`

metrics_daily_ready(ctx, upstream_done)

Fires once per session, after the daily publishers have posted.

The clock is the SESSION, read from raw.market_sessions (never weekday math): the target is the latest session whose close+90m has passed — later than events’ +45m because FINRA’s CDN file and FMP’s float bulk settle after the tape, and firing early lands a green run that measures nothing. ONE threshold-list hit is the vendor-up canary. run_key dedupes per session.

The requires gate covers equities/etfs and market_sessions so this sensor cannot launch before those products have materialized. It then reads current roster/calendar rows; the gate itself does not claim freshness.

`metrics_snapshots_ready`

metrics_snapshots_ready(ctx, upstream_done)

Collect current-only snapshots once per completed session.

This sensor never backdates a missed IB/float/market-cap observation. It retries the latest session until the whole Dagster job succeeds, then stops on that success watermark.

`setup`

setup(cfg: Any) -> None

Build the runtime and stamp a tagged session before any receipt.

`short_interest_ready`

short_interest_ready(ctx)

THE PARTITION SENSOR — fires when the vendor has something we do not.

Short interest publishes twice a month on no promised day. This sensor does not guess a date: it asks FINRA for its partitions, diffs against what raw.short_interest already holds, and fires only on a genuinely NEW settlement. It deliberately has no roster gate because the global FINRA partition verb consumes no roster. Hourly probe, run_key on the settlement itself makes each partition run once.

Cheap by construction: the probe is the partition LIST, not the data. Auth, transport, and malformed partition responses raise the sensor tick; only a valid list with no unseen date is a named quiet skip.

`update_crypto_caps`

update_crypto_caps(cfg) -> None

Crypto caps off the CoinMetrics COMMUNITY tier. The asset roster is CM’s OWN catalog filtered to codes that actually serve a 1d community mcap (135 of 6,075 today) — deriving CM codes from our stored BTC-USD pairs would be identity-map work the census reserves for facts.

`update_crypto_metrics`

update_crypto_metrics(cfg) -> None

The wider CM community surface in LONG form. Flows metrics are Pro-gated beyond btc/eth and CM 400s the pair — the feed names those as gaps and carries on, which is routine attrition, not a failure.

`update_estimates`

update_estimates(cfg, fetch_roster: dict) -> None

Consensus per fiscal period over the roster. Quarterly by cadence — the estimates for a period drift until the report lands, so the repull

upsert IS the correction mechanism.

`update_floats`

update_floats(cfg) -> None

The stable bulk page-walk — the whole universe’s float% + float shares + outstanding shares in a handful of calls. No roster needed: the endpoint IS the universe.

`update_ftd`

update_ftd(cfg) -> None

SEC fails-to-deliver — the feed walks back from the current half until a zip answers (SEC publishes ~2 weeks lagged) and reports which half landed. Upsert on (cusip, settlement_date) makes the inevitable re-landing of an already-held half a no-op.

`update_ib_shortable`

update_ib_shortable(cfg) -> None

THE BORROW SNAPSHOT — and the one step where a missed run is lost forever: IB publishes the CURRENT list and keeps NO history, so the series only exists because we append a snapshot every day. The file’s own #EOF row count is verified by the client, so a truncated transfer raises rather than landing short (live 2026-08-17: 19,725 rows).

Anonymous FTP — no session, no authenticator, nothing to expire.

`update_market_caps`

update_market_caps(cfg, fetch_roster: dict) -> None

FMP’s market-capitalization-batch over the roster — 1000 symbols per call (live-probed: 1500 answers HTTP 414). Each row carries ITS OWN observation date, so the upsert on (symbol, date) lands every one at its true date: today’s names at today, a name that last priced on Friday at Friday, a halted name at its last real day. Nothing is stamped.

`update_rates`

update_rates(cfg) -> None

Treasury curve + fed funds — the carry model’s input. A trailing window (not a single day) because the publisher revises and back-fills; upsert on (series, date) makes the overlap free.

`update_short_interest`

update_short_interest(cfg) -> None

FINRA consolidated short interest, STATE-AS-PARAMS: the settlement dates already landed are READ here and PASSED, and the feed pulls only the partitions we do not have. A new partition appearing IS the update signal — no calendar guess about FINRA’s publication day.

Lands zero rows on every day between publications; that is the normal resting state, hence no nonempty.

`update_short_volume`

update_short_volume(cfg) -> None

The sensor-selected FINRA session file (manual runs may omit it).

A weekend or holiday answers 403 and is a correct empty only when the canonical session set agrees. Sensor runs always carry an exact target, so Monday asks for Friday and a delayed retry cannot drift to Sunday or the current day. A real missing canonical file remains fatal.

`update_threshold`

update_threshold(cfg) -> None

The Reg SHO threshold list for the latest partition. DELIBERATELY no nonempty: the list is genuinely sparse — 6 names on 2026-08-17 — and a day with no threshold securities is a true zero, not a failure.