Skip to content
core-data-updater

Getting Started

Canonical raw ingestion plus dbt facts and analytics.

Overview

Container type: dagster-location

core-data-updater is a native Dagster code location: this repo owns its jobs, its Definitions (in src/core_data_updater/main.py), any dbt project, and its container image. There is no Helm chart here — CI builds the image and registers an immutable pointer with the orchestration repo, whose Dagster deployment serves this location over gRPC (see the Deployment guide).

Prerequisites

  • uv — manages Python, the venv, and all dependencies: curl -LsSf https://astral.sh/uv/install.sh | sh
  • Docker — for image builds

If this repo consumes private packages, configure your registry read token first — see CONTRIBUTING.md in the repo root for the one-time setup.

Set up a clone

Terminal window
git clone https://github.com/null-capital/core-data-updater.git
cd core-data-updater


doppler run --project global --config prd -- ./run sync

./run sync creates the venv, installs all dependencies (uv sync --all-extras), and installs the git hooks. Prefer the raw tool? uv sync does the dependency half on its own. For private packages, the standard doppler run command routes GEMFURY_READ_TOKEN to uv without persisting it.

Run locally

Terminal window
./run local

which starts a local Dagster UI serving this location’s Definitions (jobs, sensors, dbt):

Terminal window
uv run dagster dev -m core_data_updater.main

Materialize and launch runs from the UI exactly as production Dagster will.

Run with Docker

Terminal window
./run build    # docker build
docker run --env-file .env core-data-updater:dev

Published images live at ghcr.io/null-capital/core-data-updater — every merge builds and pushes one (see the Deployment guide).

Verify

Terminal window
./run check

./run check runs exactly what CI runs (lint + format + types + test), so green locally means green in CI.

Next steps