Skip to content

Architecture Decision Records

ADRs document significant technical decisions, the context that forced them, and the trade-offs accepted. They are permanent — if a decision changes, a new ADR supersedes the old one. Nothing is deleted or rewritten.

Index

# Decision Status
001 Load strategy terminology: full_refresh, incremental, historical, batch_replace Accepted
002 Intraday refresh patterns for dynamic tables Accepted
003 Build-time DAG generation (YAML → Python) over runtime dag-factory Accepted
004 Metadata columns: RECORD_KEY, RECORD_HASH, BATCH_ID, SOURCE_FILE, PIPELINE_RUN_ID, DATE_TO_WAREHOUSE Accepted
005 Three-tier stage configuration priority Accepted
006 Bronze truncate + FORCE = TRUE for legacy pipelines Accepted
007 Skip logic inside task functions, not in DAG structure Accepted
008 Gold layer: fct_, dim_, agg_, rpt_ Accepted
009 Naming conventions — tables, DAGs, S3 paths, columns Accepted
010 Rosters in CORE_DB, glossaries in client DB Accepted
011 batch_replace for new pipelines, truncate_insert for legacy Accepted
012 Documentation structure: docs/ as single source of truth Accepted
013 Bronze all-VARCHAR + Silver owns type casting via TRY_TO_* Accepted
014 XOOS analytics data delivery: direct Snowflake + CloudFlare edge cache Accepted
015 SANDBOX schema for ad-hoc / supplementary analyst datasets Accepted
016 GitHub issue quality standards for XO Data Ops project (#15) Accepted
017 Partner-managed Bronze: accepting external Snowflake loading (Rippit/MAESTRO_DB) Accepted
018 Separate ELT and dbt DAGs via TriggerDagRunOperator to prevent Cosmos import timeouts Accepted
019 Gold layer standardization: naming, structure, KPI conventions, client/layer project layout Accepted
020 Silver incremental strategy: merge + RECORD_HASH is canonical; append is legacy Accepted
021 Gold exploration layer (exp_): pilot/ad-hoc views, no SLA, promote to rpt_ when stable Accepted
022 WBP topic-level metric aggregates: fan-out grain, CSAT fuzzy join, QA URL parsing Accepted
023 Config storage: env vars for credentials/static infra, Airflow Connections for host+auth, Airflow Variables for runtime tunables Accepted
024 Bronze merge strategy for rolling-window state-snapshot sources (Gladly exports, mutable 30-day window) Accepted
025 Snowflake-to-Google Sheets sync: full_replace as default; row-count management via exp_ view rollups Accepted
026 Gold feed layer (feed_): production-grade external export views with conditional daily/weekly grain rollup Accepted
027 DAG config filename and pipeline_name convention: {client}-{source}-{type}-{grain} Accepted
028 dbt test policy: test once at the layer where data enters, drop Bronze source tests, not_null on TRY_TO_* structural columns, reclassify summary fct_ as agg_ Accepted
029 S3 lifecycle: stage/ transient (expire 14d), ingest/ archive (Glacier 90d, expire 2yr); failed/ quarantine on COPY failure Proposed
030 dbt DAG grain slot and GDrive source naming: triggered dbt DAGs drop {grain}; scheduled keep it; GDrive-intermediated sources use {source}-gdrive compound slot. Amends ADR 027 Accepted
031 WBP contact volume: use INBOX_WHEN_QUEUED as inbox dimension (not INBOX_WHEN_ENDED); won't match Gladly contact summary by design Accepted
— Branching strategy Accepted
— Single account strategy Accepted
— Snowflake environments Accepted
— Naming conventions (legacy) Superseded by 009

How to write a new ADR

  1. Open a Decision issue to work through options with the team
  2. Pick the next number and create NNN-kebab-case-topic.md in this directory
  3. Use this structure:
# ADR NNN — Title

## Status
Accepted  |  Proposed  |  Superseded by ADR NNN

## Context
What situation or constraint forced this decision? What would happen if we did nothing?

## Decision
What did we decide to do?

## Consequences
What gets easier? What gets harder? What do we give up?

## Options Considered
What else did we evaluate and why did we reject it?
  1. Link the ADR from the Decision issue and close it
  2. Update this index table

ADRs are append-only. Never rewrite history — future readers need to understand why decisions made sense at the time they were made.