ADR 017 — Partner-Managed Bronze: Accepting External Snowflake Loading¶

Status¶

Accepted

Context¶

When onboarding Rippit QA evaluation data for Condé Nast (#540), two options were evaluated:

Option 1 — S3 Pipeline (standard xo-foundry pattern) Build an xo-foundry extractor that hits the Rippit API, stages files to S3, and loads into CND_DB.BRONZE via COPY INTO. This is the standard ELT pattern used for Gladly, Sprout, and Google Sheets.

Option 2 — Partner-Managed Bronze (selected) The Rippit team already maintains a native Snowflake integration that loads QA evaluation data directly into MAESTRO_DB.MAESTRO within the XtendOps Snowflake account (FBRITWN-XOS6437). XO-Data accepts this database as an externally-managed Bronze source and references it from dbt Silver models.

The Rippit team manages their own load schedule, warehouse (RIPPIT_LOAD_WH), and role (MAESTRO_ROLE). The data is available for XO-Data to read via cross-database references.

Decision¶

When a data partner or tool vendor already loads data into a table in the XtendOps Snowflake account, we accept the partner-managed database as the Bronze source rather than building a redundant extraction pipeline.

For Rippit specifically: - MAESTRO_DB.MAESTRO is the Bronze source for CND QA evaluation data - The Rippit team (MAESTRO_USER / MAESTRO_ROLE) owns all loading into this database - XO-Data dbt Silver models reference MAESTRO_DB.MAESTRO as a cross-database source - No xo-foundry extractor, S3 staging, or schemachange migration is needed for the Bronze layer

Partner database connection details (read reference)¶

Field	Value
Account	`FBRITWN-XOS6437` (XtendOps account — same as XO-Data)
Database	`MAESTRO_DB`
Schema	`MAESTRO`
Warehouse (Rippit-owned)	`RIPPIT_LOAD_WH`
Role (Rippit-owned)	`MAESTRO_ROLE`

XO-Data queries this database using its own read-capable role; no credentials sharing is required.

Consequences¶

Gets easier: - Zero ingestion infrastructure to build or maintain for this data source - Data is always current — Rippit controls freshness and load cadence - Schema exploration and Silver model development can start immediately against live data

Gets harder / risks to manage: - XO-Data does not own Bronze ingestion — we cannot control load timing, retries, or backfill - Rippit schema changes (column renames, type changes, table drops) can silently break downstream Silver/Gold models; we must monitor for schema drift - No BATCH_ID / LOADED_AT metadata columns are guaranteed — Silver models must adapt to whatever columns Rippit provides - Pipeline observability is split: Rippit owns the load, XO-Data owns the transformation; failure attribution requires coordination

Mitigations: - Add a dbt source freshness check on MAESTRO_DB.MAESTRO tables so Airflow alerts if the Rippit load stops - Document the Rippit schema in the Silver model's schema.yml so column changes surface as dbt compilation errors - Establish a communication channel with the Rippit team for schema change notifications

Options Considered¶

Option 1 — S3-Based xo-foundry Extractor (not selected)¶

Pros: XO-Data owns the full pipeline; consistent with all other CND/WBP pipelines; Bronze metadata columns (BATCH_ID, LOADED_AT, RECORD_KEY) guaranteed; easier to debug load failures.

Rejected because: Rippit already has a working Snowflake integration loading live data. Building a parallel S3 pipeline would duplicate the ingestion work, require Rippit API credentials and maintenance, and add latency. The marginal benefit of owning the Bronze layer does not outweigh the cost when the partner database is already in the same Snowflake account.

When to Apply This Pattern¶

Accept partner-managed Bronze when all three conditions are true:

The data partner already loads into a table in the XtendOps Snowflake account (FBRITWN-XOS6437)
The partner database is readable by XO-Data roles (confirmed access)
The data partner commits to providing advance notice of breaking schema changes

If any condition is not met, fall back to the standard xo-foundry S3 pipeline.