Airflow Pool Inventory¶

Airflow pools control task concurrency against external APIs and shared resources. This is the authoritative inventory of all pools used in xo-pipelines.

Naming Convention¶

Pools are named after the resource they protect, not the construct itself:

{client}_{api}_api — client-scoped API pool (e.g., wbp_gladly_api)
xo_{service}_api — shared cross-client service pool (e.g., xo_google_api)

The _api suffix signals that the pool serializes calls to an external API. Avoid the _pool suffix — it's redundant in the Pools context.

Pool Registry¶

Pool Name	Size	Protects	DAGs
`xo_google_api`	10	Google Sheets API, Google Drive API, Gmail API — all share the same per-project quota in our Data Engineering GCP project	`core_bi_roster_extract_daily`, `core_ph_roster_extract_daily`, `wbp_gsheets_extract_daily`, `wbp_aws_gdrive_extract_daily`, `wbp_aspect_gdrive_extract_weekly`, `jcr_gdrive_extract_daily`
`wbp_gladly_api`	1	Warby Parker Gladly API	`wbp_gladly_extract_daily`, `wbp_gladly_extract_hourly`, `wbp_gladly_extract_weekly`, `wbp_gladly_exports_extract_daily`
`cnd_gladly_api`	1	Conde Nast Gladly API	`cnd_gladly_extract_daily`, `cnd_gladly_extract_hourly`, `cnd_gladly_extract_weekly`, `cnd_gladly_exports_extract_daily`

How to Configure a Pool in a DAG Config¶

Set runtime.default_pool in the pipeline YAML config:

runtime:
  default_pool: xo_google_api   # or wbp_gladly_api, etc.

This applies the pool to all extract tasks in the DAG. After changing the YAML, regenerate the DAG:

uv run xo-foundry generate-dag \
  --config apps/airflow/xo-pipelines/dags/configs/<config-name>.yaml \
  --output apps/airflow/xo-pipelines/dags/

Notes¶

xo_google_api is shared across all gsheets, gdrive, and gmail DAGs. All data engineering Google API calls use the same GCP project, so they share a single quota bucket. One pool keeps them from colliding.
Gladly pools are per-client. Each client has its own Gladly account with independent rate limits, so WBP and CND pipelines are isolated.
Snowflake-to-GSheets feed DAGs (e.g. core_xosec_roster_feed_weekly, wbp_feed_looker_daily) write to Google Sheets but use the snowflake_to_gsheet pipeline type, which does not currently support pool configuration. They are low-frequency and unlikely to contend with extract DAGs.
Pool size = 1 is the conservative default. Increase it only after confirming the external API can handle parallel calls without throttling.
Pools must be created manually in Airflow (Admin → Pools) or via the Airflow API — they are not managed by code in this repo.