Data Pipelines

What Is a Data Pipeline?

Data pipelines are all the operations, jobs, and assets that participate in the data flow within the data stack. These may include:

• A transformation operation that generates a table (example: a dbt model)
• An orchestration operation that triggers a transformation
• A data movement operation that takes data from one system to another

Why Connect Sifflet to Your Data Pipelines?

Integrating Sifflet with your data pipelines allows you to benefit from the following features:

• Get alerted in case of a pipeline failure
• Enrich your lineage graph with pipeline metadata
• Access up-to-date pipeline status within Sifflet
• Leverage additional context when debugging data incidents

Sifflet's Data Pipeline Integrations

Sifflet currently integrates with the following data pipelining tools:

• Apache Airflow: self-hosted, Amazon Managed Workflows for Apache Airflow (MWAA), and Cloud Composer
• Azure Data Factory
• Databricks Workflows
• dbt (both dbt Core and dbt Cloud)
• Fivetran

If one of your tools is not on the list, contact Sifflet support to discuss your use case.

Refresh Frequency

See Integrations Management for the default refresh frequency and how to change it.