Data Pipelines
What's a data pipeline?
We refer by Data Pipelines to all the operations, jobs, and assets that participate in the data flow within the data stack. These may include:
- A transformation operation that generates a table (example: a dbt model)
- An orchestration operation that triggers a transformation
- A data movement operation that takes data from one system to another
Why connect Sifflet to your data pipelines?
Integrating Sifflet with your data pipelines allows you to benefit from the following features:
- Get alerted in case of a pipeline failure
- Enrich your lineage graph with pipeline metadata
- Access up-to-date pipeline status within Sifflet
- Leverage additional context when debugging data incidents
Sifflet's data pipeline integrations
Sifflet currently integrates with the following data pipelining tools:
- Apache Airflow (and its cloud variations: MWAA and Cloud Composer)
- dbt (both dbt Core and dbt Cloud)
- Fivetran
Don't see all of your tools on the list? Reach out to us and we'd be happy to discuss your use case!
Data pipelines default refresh frequency
By default, Sifflet performs an hourly metadata refresh for data pipeline sources to keep statuses current in Sifflet and enable prompt detection of pipeline failures.
Updated 25 days ago