ADR 027 — DAG Config Filename and Pipeline Name Convention¶

Status¶

Accepted

Amended by ADR 030: triggered dbt DAGs drop the {grain} slot; only scheduled dbt DAGs keep it; and GDrive-intermediated sources use a {source}-gdrive compound source slot (e.g., wbp-aws-gdrive-extract-daily) rather than a plain gdrive source slot. The rest of this ADR stands.

Context¶

27 DAG configs in apps/airflow/xo-pipelines/dags/configs/ had accumulated naming inconsistencies that made the Airflow UI hard to scan:

Client prefix inconsistency: condenast- vs cnd-, warbyparker- vs wbp- for the same client
Slot order inconsistency: {client}-dbt-{source} vs {client}-{source}-{frequency} vs {client}-{source} — no agreed-upon structure
Pipeline name disambiguation: no clear convention when a client has multiple pipelines for the same source (e.g., sprout_messages_bronze vs sprout_messages_incremental vs sprout_messages_updated_at)

Because Airflow sorts DAGs alphabetically in the UI, inconsistent naming means related pipelines (e.g., the Gladly extract and its downstream dbt transform) do not sort together.

Decision¶

Adopt a canonical 4-slot naming convention for all DAG config filenames and the pipeline_name field:

{client}-{source}-{type}-{grain}

The DAG ID is derived from domain + "_" + pipeline_name (existing factory behavior, unchanged). The config filename uses hyphens; the DAG ID uses underscores.

Slot definitions¶

Slot	Values	Notes
`{client}`	`wbp`, `cnd`, `core`, `jcr`	Short client code; `legacy` for OPERATIONS-targeting pipelines
`{source}`	`gladly`, `gsheets`, `gdrive`, `maestro`, `medallia`, `sprout`, `aspect`, `aspen`, `medallion`	External system involved. For `extract`/`dbt`: the source system. For `feed`: the destination system. `medallion` is used for feed DAGs (data comes from Gold `feed_` views).
`{type}`	`extract`, `dbt`, `feed`	Pipeline role in the stack
`{grain}`	`daily`, `hourly`, `weekly`	Schedule frequency or data grain

Type-specific patterns¶

Pipeline type	Pattern	Example
ELT extract	`{client}-{source}-extract-{grain}`	`wbp-gladly-extract-daily`
dbt transform	`{client}-{source}-dbt-daily`	`wbp-gladly-dbt-daily`
Feed export	`{client}-medallion-feed-{grain}`	`wbp-medallion-feed-daily`
Legacy (OPERATIONS)	`legacy-{client}-{source}-{descriptor}`	`legacy-wbp-gladly-reports-hourly`

Disambiguation¶

When two pipelines share the same {client}-{source}-{type}-{grain}, add a qualifier before {type}:

cnd-gladly-agents-extract-weekly      ← agents endpoint, not contacts
cnd-gladly-exports-extract-daily      ← export API endpoints, not reports
core-gsheets-bi-roster-extract-daily  ← bi-roster source, not ph-roster

Notes on specific conventions¶

dbt grain: dbt DAGs have no schedule — they are triggered by extract DAGs via TriggerDagRunOperator. The daily grain in their name refers to the data grain they process, not the schedule mechanism.

medallion as feed source: Feed DAGs read from Gold feed_ views in Snowflake. medallion is the source slot value — it is accurate (data comes from the medallion Gold layer), team-familiar, and distinct from all external system names. When a client has multiple feed pipelines, add a content qualifier: wbp-medallion-feed-ops-daily.

Legacy prefix: pipeline_type: legacy_operations configs target the OPERATIONS database, not the medallion architecture. These use legacy as both the {client} prefix in the filename and the domain field, so they sort as a separate group at the bottom of the Airflow UI.

Airflow sort order (post-migration)¶

cnd_gladly_agents_extract_weekly
cnd_gladly_dbt_daily
cnd_gladly_exports_extract_daily
cnd_gladly_extract_daily
cnd_gladly_extract_hourly
cnd_maestro_dbt_daily
cnd_medallia_extract_daily
cnd_sprout_extract_daily
─
core_gsheets_bi_roster_dbt_daily
core_gsheets_bi_roster_extract_daily
core_gsheets_ph_roster_dbt_daily
core_gsheets_ph_roster_extract_daily
core_medallion_feed_weekly
─
jcr_gdrive_extract_daily
─
legacy_cnd_sprout_daily
legacy_cnd_sprout_updated_daily
legacy_wbp_gladly_reports_backfill
legacy_wbp_gladly_reports_hourly
─
wbp_aspect_conformance_extract_weekly
wbp_aspen_conformance_extract_daily
wbp_gladly_agents_extract_weekly
wbp_gladly_dbt_daily
wbp_gladly_extract_daily
wbp_gladly_extract_hourly
wbp_gsheets_dbt_daily
wbp_gsheets_extract_daily
wbp_medallion_feed_daily

Consequences¶

Gets easier: - Scanning the Airflow UI: DAGs from the same source sort together (e.g., all wbp_gladly_* DAGs — extract daily, extract hourly, dbt — appear as a contiguous block) - Understanding a DAG's role at a glance from its name - Onboarding: new engineers can infer the convention without a lookup table - Adding new clients: new configs naturally fit the pattern

Gets harder / trade-offs: - One-time migration cost: all 27 configs renamed; generated DAG .py files regenerated; golden test files regenerated; docs updated - Airflow history: existing Airflow run history is associated with old DAG IDs; new DAG IDs start with a clean slate in the Astronomer UI - Legacy DAGs use legacy as the domain field value — this is semantically valid but slightly unusual (domain is not a client code for these)

Options Considered¶

Derive DAG ID from filename (not domain + pipeline_name): Would eliminate the need to keep domain and pipeline_name in sync with the filename. Rejected for this migration because it requires factory logic changes (out of scope for this chore) and the current approach achieves the same result by updating both fields together.

Type-first slot order ({client}-extract-{source}-{grain}): Would group all extracts together and all dbt together, but separate related pipelines for the same source (gladly extract and gladly dbt would not sort adjacently). Rejected — source-first grouping is more operationally useful.

Feed DAGs as {client}-gsheets-feed-{grain}: Using the destination system in the source slot groups feed DAGs with other GSheets pipelines, but gsheets in the source slot implies GSheets is the data source (it is not). Using medallion in the source slot is accurate and avoids the semantic confusion. The feed type slot communicates direction.