Data Pipelines

We refer by Data Pipelines to all the operations, jobs, and assets that participate in the data flow within the data stack. These may include:

• A transformation operation that generates a table (example: a dbt model)
• An orchestration operation that triggers a transformation
• A data movement operation that takes data from one system to another

Integrating Sifflet with your data pipelines allows you to enrich your lineage graph with pipeline metadata, access up-to-date pipeline status within Sifflet, and have additional context when debugging data incidents.

Sifflet currently integrates with the following data pipelining tools:

• Apache Airflow (and its cloud variations: MWAA and Cloud Composer)
• dbt (both dbt Core and dbt Cloud)
• Fivetran

Don't see all of your tools on the list? Reach out to us and we'd be happy to discuss your use case!