Airflow Collected Data

This page lists what Sifflet imports from Airflow. It applies to all deployments: self-hosted Airflow, Amazon MWAA, and Cloud Composer.

Imported Assets

DAGs: every DAG returned by the Airflow REST API to the identity Sifflet uses, imported into the Data Catalog as an Airflow DAG asset.

Collected Metadata

For each DAG:

Name: the DAG ID.
Description: the description of the DAG.
Tags: the Airflow tags of the DAG.
Schedule: the schedule of the DAG (cron expression, time delta, or relative delta).
Link to Airflow: a link to the DAG page in the Airflow web server.
Latest run: the most recent DAG run, with its state (queued, running, success, or failed), its run type (scheduled, manual, backfill, or dataset-triggered), its logical date, start date, and end date, whether it was triggered externally, and its note. Sifflet shows this state next to the DAG in the Data Catalog, on asset pages, and in the lineage.

Lineage

The Airflow API doesn't tell which tables a DAG updates. Sifflet links DAGs to data assets from the query history of your data platform sources: when a query that updates a table carries the airflow_dag_id:<DAG_ID> tag (a Snowflake query tag or a BigQuery job label), Sifflet links that table to the DAG. The DAG then appears above the table in the lineage, and in the Orchestrator field of the asset page.

This works with BigQuery and Snowflake (Enterprise Edition) sources. To tag your queries, see Linking Airflow DAGs to Data Assets.

Data Freshness

Sifflet updates DAGs and their latest run at each refresh of the Airflow source. For refresh frequencies and scheduling, see Integrations Management. Between two refreshes, the run state shown in Sifflet can lag behind Airflow.

Limitations

Airflow 3 isn't supported, on any deployment: Sifflet uses the Airflow REST API v1, which Airflow 3 removed.
Sifflet only reads recent DAG runs: a DAG that hasn't run recently has no latest run in Sifflet.

Data Exposure

The Airflow integration reads only metadata about DAGs and DAG runs. It doesn't read the data your DAGs process, so no Sifflet feature stores or displays row-level data from Airflow. See Security.