ADR 030 — dbt DAG Grain Slot and GDrive Source Naming Corrections¶

Status: Accepted Date: 2026-06-08 Amends: ADR 027 (dbt grain note; GDrive-intermediated source naming)

Context¶

ADR 027 established the 4-slot DAG naming convention {client}-{source}-{type}-{grain} and, in its "Notes on specific conventions" section, ruled:

dbt grain: dbt DAGs have no schedule — they are triggered by extract DAGs via TriggerDagRunOperator. The daily grain in their name refers to the data grain they process, not the schedule mechanism.

In practice this convention is misleading:

The {grain} slot otherwise means schedule frequency. For extract DAGs, daily/hourly/weekly is the cron cadence. Using the same slot to mean "data grain" for dbt DAGs overloads it with two incompatible meanings — a reader cannot tell from the name whether a DAG is scheduled or not.
A triggered dbt DAG has no single cadence. wbp_gladly_dbt is triggered by both wbp-gladly-extract-daily and wbp-gladly-extract-hourly. Calling it *_dbt_daily is simply false — it runs whenever any upstream extract completes, on whatever cadence that extract has.
Schedule lives on extracts, not transforms. The triggered dbt DAGs do not own a schedule at all; the extract DAGs do. The name should reflect that the transform follows its trigger, not a clock.

This surfaced while scoping per-source dbt selection for the Google Drive pipelines (AWS daily vs Aspect weekly, both transformed by triggered dbt runs): the _daily suffix on the triggered dbt DAGs implied a cadence they do not have.

Decision¶

The {grain} slot on a dbt (type: dbt) DAG reflects the DAG's own schedule reality:

dbt DAG kind	Has own `schedule:`?	Pattern	Example
Triggered (fired by an extract via `TriggerDagRunOperator`)	No	`{client}-{source}-dbt`	`wbp-gladly-dbt`, `wbp-gsheets-dbt`, `core-bi-roster-dbt`
Scheduled (owns a cron `schedule:`)	Yes	`{client}-{source}-dbt-{grain}`	`cnd-maestro-dbt-daily`

Rule of thumb: the {grain} slot describes a DAG's own schedule. A DAG with no schedule carries no grain slot.

This amends only the dbt-grain note and the dbt-transform row of the "Type-specific patterns" table in ADR 027. The 4-slot convention itself, slot order, client/source/type definitions, disambiguation rule, and all non-dbt patterns are unchanged and remain authoritative.

Decision 2 — GDrive-intermediated source naming¶

ADR 027's {source} slot uses the source system name. For two WBP pipelines, the source system (AWS Connect, Aspect WFM) is distinct from the transport mechanism (Google Drive folder). Using a plain {source} name (aws, aspect) reserves that slot for future direct API integrations.

Convention: append -gdrive to the source name when the data is not directly API-accessible and lands in a Google Drive folder first.

Pattern	When to use	Example
`{source}-gdrive`	Data lands in a GDrive folder; no direct API access	`wbp-aws-gdrive-extract-daily`
`{source}`	Direct API integration (current or future)	`wbp-aws-extract-daily` (future direct AWS API)
`gdrive` (no prefix)	All datasets in the DAG share GDrive transport AND run on the same cadence in one DAG	`jcr-gdrive-extract-daily`

dbt tag counterpart: the Silver model tag mirrors the source slot with underscores — aws_gdrive, aspect_gdrive. This keeps the DAG selector (tag:wbp,tag:aws_gdrive+) aligned with the config naming.

Migration applied (2026-06-08)¶

dbt DAG grain slot renames:

Renamed (config file, pipeline_name, generated DAG .py, golden fixture, and all trigger_dag references updated together):

Old DAG ID	New DAG ID
`cnd_gladly_dbt_daily`	`cnd_gladly_dbt`
`core_bi_roster_dbt_daily`	`core_bi_roster_dbt`
`core_ph_roster_dbt_daily`	`core_ph_roster_dbt`
`wbp_gladly_dbt_daily`	`wbp_gladly_dbt`
`wbp_gsheets_dbt_daily`	`wbp_gsheets_dbt`

cnd_maestro_dbt_daily is retained — it owns a cron schedule (0 7 * * *) and is therefore genuinely daily.

GDrive source naming renames:

Old DAG ID	New DAG ID	Reason
`wbp_gdrive_extract_daily`	`wbp_aws_gdrive_extract_daily`	Source = AWS Connect; `-gdrive` = GDrive delivery
`wbp_gdrive_extract_weekly`	`wbp_aspect_gdrive_extract_weekly`	Source = Aspect WFM; `-gdrive` = GDrive delivery

dbt Silver model tags updated in the same migration: aws → aws_gdrive, aspect → aspect_gdrive.

Consequences¶

Gets easier: - A DAG's name now honestly signals whether it is scheduled or trigger-driven — no false cadence claims. - The {grain} slot has a single consistent meaning across all DAG types: schedule frequency. - Reinforces the platform principle that schedule lives on extracts; transforms follow their triggers. - aws and aspect source slots are reserved for future direct API integrations, which can be added without naming conflicts. - The -gdrive suffix is a self-documenting signal that data arrives indirectly via a GDrive folder handoff.

Gets harder / trade-offs: - One more rule to internalize (though it removes the special-case exception ADR 027 had to carve out). - DAG run history resets in the Astronomer UI for the 7 renamed DAGs — the same accepted consequence as the ADR 027 migration.

Options Considered¶

Keep ADR 027 as-is (grain = data grain for dbt): Rejected — overloads the {grain} slot with two meanings and produces names that lie about cadence for multi-trigger dbt DAGs.

Drop the grain slot from all dbt DAGs, including scheduled ones: Rejected — cnd-maestro-dbt-daily genuinely owns a daily schedule; hiding that is as misleading as the original problem. The slot should be present exactly when a schedule exists.

Add a separate triggered token instead of omitting grain ({client}-{source}-dbt-triggered): Rejected — verbose, and absence of a grain slot already communicates "no schedule" once the rule is known.

Use aws-drive or aws-gdrive in the source slot: Chose aws-gdrive — drive alone is ambiguous (OneDrive?), and gdrive is the established term in this codebase (JCR's DAG is already jcr-gdrive-extract-daily). Hyphenated compound source names are consistent with the disambiguation pattern in ADR 027.

Keep plain gdrive for all GDrive-delivered sources: Rejected — a single gdrive source tag over-selects when sources run on different cadences. tag:wbp,tag:gdrive+ rebuilds both the daily AWS chain and the weekly Aspect chain on every run.