Azure Data Factory Collected Data

This page describes what Sifflet imports from Azure Data Factory (ADF). To connect ADF, see Azure Data Factory.

Imported Assets

Sifflet imports the pipelines of each selected data factory. Each pipeline appears in the Data Catalog as an ADF pipeline asset, with its own asset page.

Collected Metadata

For each pipeline, from the Azure Resource Manager API:

• Name and description.
• Annotations, imported as tags.
• Latest run: start and end time, duration, status (succeeded, failed, canceled), and trigger type (manual, schedule trigger, or event trigger).

Sifflet only considers completed runs from the last 45 days, the retention period of pipeline runs in Data Factory. When a run was rerun, Sifflet uses the latest attempt. A pipeline with no completed run in this period has no latest run in Sifflet.

Lineage

The Azure Data Factory integration doesn't add lineage: Sifflet doesn't link pipelines to the datasets they read or write.

Data Freshness

Sifflet refreshes the pipeline metadata on the source's schedule. See Integrations Management for the default frequency and how to change it. The latest run status in Sifflet is the one of the latest ingestion.

Limitations

• Sifflet imports the latest run of each pipeline, not its full run history, and doesn't send notifications on pipeline failures.
• Activities, triggers, datasets, and data flows aren't imported.

Data Exposure

The Azure Data Factory integration only reads pipeline and run metadata through the Azure Resource Manager API. It doesn't read or store any data processed by your pipelines. See Security.