Azure Data Factory

Sifflet connects to Azure Data Factory (ADF) to show the state of your data pipelines next to the rest of your data stack. Each ADF pipeline appears in the Data Catalog with the status, trigger type, and duration of its latest run, so you can spot failed or late pipelines from Sifflet and check them when investigating a data incident. See Azure Data Factory Collected Data for everything Sifflet imports.

Permissions Required

Sifflet uses a Microsoft Entra ID service principal (app registration with a client secret) to call the Azure Resource Manager API with read-only requests. It never runs, edits, or cancels pipelines.

Assign the permissions with a custom role at the scope of the resource group that contains your data factories:

Ingestions don't start any pipeline or consume any Data Factory compute.

Requirements

Required Roles for Setup

• In Microsoft Entra ID: the right to create app registrations and client secrets (for example, the Application Developer role).
• In Azure: the right to create custom roles and role assignments on the resource group (for example, the Owner or User Access Administrator role).
• In Sifflet: the Admin system role to create the credential, and the Admin or System Editor system role to create the source. See Access Control.

Supported Versions and Editions

• Azure Data Factory (V2) in the Azure public cloud.
• Each source covers the data factories of one resource group in one subscription. To monitor data factories in several resource groups, create one source per resource group.

Network Access

Sifflet connects to the public Azure endpoints over the internet: Microsoft Entra ID (https://login.microsoftonline.com) to authenticate, and Azure Resource Manager (https://management.azure.com) to read metadata.

Other Prerequisites

• Each selected data factory must contain at least one pipeline. Before connecting and before each ingestion, Sifflet also checks that a data factory of the resource group has at least one pipeline run in the last 365 days: otherwise, Test connection and the ingestion fail (see Azure Data Factory Troubleshooting).

Connect Sifflet to Azure Data Factory

Step 1: Create a Service Principal in Azure

Create the App Registration

1. In the Azure portal, go to Microsoft Entra ID > App registrations and click New registration.
2. Enter a name, for example sifflet-adf-app, and click Register.
3. On the Overview page, copy the Application (client) ID and the Directory (tenant) ID.
4. Go to Certificates & secrets > Client secrets and click New client secret. Enter a description and an expiry.
5. Copy the secret Value immediately: Azure shows it only once. Before the secret expires, create a new one and update the Sifflet credential.

Create the Custom Role

1. Go to Resource groups and select the resource group that contains your data factories, for example MyAdfResourceGroup.
2. Go to Access control (IAM) and click Add > Add custom role.
3. Enter a name, for example sifflet-adf-role.
4. In the Permissions tab, click Add permissions and add the following permissions of the Microsoft Data Factory provider:
   • Required: Microsoft.DataFactory/factories/read, Microsoft.DataFactory/factories/pipelines/read, and Microsoft.DataFactory/factories/querypipelineruns/read.
   • Optional (support diagnostics): Microsoft.DataFactory/factories/datasets/read and Microsoft.DataFactory/factories/linkedServices/read.
5. Click Review + create, then Create.

Alternatively, create the role with the Azure CLI (az role definition create --role-definition role.json) and the following definition, where role.json contains the required permissions only:

{
  "Name": "sifflet-adf-role",
  "Description": "Read-only access to Azure Data Factory metadata for Sifflet",
  "Actions": [
    "Microsoft.DataFactory/factories/read",
    "Microsoft.DataFactory/factories/pipelines/read",
    "Microsoft.DataFactory/factories/querypipelineruns/read"
  ],
  "NotActions": [],
  "AssignableScopes": [
    "/subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP>"
  ]
}

Assign the Role to the Service Principal

1. In the Access control (IAM) page of the same resource group, click Add > Add role assignment.
2. Select the custom role (sifflet-adf-role) and click Next.
3. Under Assign access to, select User, group, or service principal, click Select members, and select the app registration (sifflet-adf-app).
4. Click Review + assign.

Keep the following values for step 2: the client ID, the client secret value, the tenant ID, the subscription ID (in Subscriptions), and the resource group name.

Step 2: Add the Azure Data Factory Source in Sifflet

Create the Credential

In Integrations > Credentials, create a new credential, for example my-adf-resource-group-sp, with the following JSON value:

{
  "clientId": "<CLIENT_ID>",
  "clientSecret": "<CLIENT_SECRET>"
}

Create the Source

1. Go to Integrations > Sources > New source and choose Azure Data Factory.

2. Fill in the fields:

   Field	Description	Example
   Source name	Name of the source in Sifflet	ADF - production
   Azure AD tenant id	Directory (tenant) ID, in Microsoft Entra ID > Overview	00000000-0000-0000-0000-000000000000
   Subscription id	ID of the subscription that contains the resource group, in Subscriptions	11111111-1111-1111-1111-111111111111
   Resource Group	Name of the resource group that contains your data factories	MyAdfResourceGroup
   Credentials	The credential created above	my-adf-resource-group-sp

3. Under Asset to include in the Data Catalog, click Load assets and select the data factories to monitor. Enable Automatically include new data factories to monitor data factories created later in the resource group without editing the source.

   

4. In Set frequency, keep the default schedule or change it. See Integrations Management.

5. Click Test connection: Sifflet checks that it can list the data factories, and the pipelines and pipeline runs of a data factory. Then save the source.

Sifflet runs a first ingestion, which usually takes a few minutes. You can follow it in the source details, in Integrations > Sources. Once it succeeds, your pipelines appear in the Data Catalog: filter the catalog on the Azure Data Factory source to see them with their latest run status.

Troubleshooting

See Azure Data Factory Troubleshooting.