Data Pipelines

What's a data pipeline?

We refer by Data Pipelines to all the operations, jobs, and assets that participate in the data flow within the data stack. These may include:

A transformation operation that generates a table (example: a dbt model)
An orchestration operation that triggers a transformation
A data movement operation that takes data from one system to another

Why connect Sifflet to your data pipelines?

Integrating Sifflet with your data pipelines allows you to enrich your lineage graph with pipeline metadata, access up-to-date pipeline status within Sifflet, and have additional context when debugging data incidents.

Sifflet's data pipeline integrations

Sifflet currently integrates with the following data pipelining tools:

Apache Airflow (and its cloud variations: MWAA and Cloud Composer)
dbt (both dbt Core and dbt Cloud)
Fivetran

Don't see all of your tools on the list? Reach out to us and we'd be happy to discuss your use case!

Data pipelines default refresh frequency

By default, Sifflet performs an hourly metadata refresh for data pipeline sources to keep statuses current in Sifflet and enable prompt detection of pipeline failures.