Databricks Workflows

Sifflet connects to your Databricks workspace to show your Databricks jobs and the status of their latest run next to the data they produce. Combined with the Databricks integration, the lineage shows which job writes each table: when a job fails, you see which tables and dashboards it affects, and when a table has an incident, you see whether the job upstream is the root cause. Sage, Sifflet's root cause analysis agent, uses these links during its investigations.

For the full list of assets and metadata Sifflet imports, see Databricks Workflows Collected Data. To stop a job when a Sifflet monitor fails, see Flow Stopper.

Permissions Required

Sifflet uses read-only access. It reads Databricks system tables through a Databricks SQL warehouse, with the personal access token of a service principal (recommended) or of a user. Use the same service principal as for the Databricks integration.

Sifflet checks these permissions when you test the connection: if one is missing, the test fails.

The links between jobs and tables come from the Databricks source, which reads system.access.column_lineage and, optionally, system.lakeflow.pipeline_update_timeline. See Databricks permissions.

Cost and load: Sifflet runs metadata queries on the SQL warehouse you configure during each refresh, which consumes Databricks compute. You can use the same warehouse as for the Databricks source.

Requirements

Required Roles for Setup

• In Databricks:
  • a workspace admin, to create the service principal and grant it access to the SQL warehouse;
  • an account admin, to enable the system.access system schema if it isn't enabled yet;
  • a metastore admin, to grant privileges on the system catalog.
• In Sifflet: the Admin role to create the credential, and the Admin or System Editor role to create the source. See Access Control.

Supported Versions and Editions

• Databricks workspaces on AWS, Azure, and Google Cloud, with Unity Catalog enabled: Databricks only exposes system tables in Unity Catalog workspaces.
• Databricks jobs (Lakeflow Jobs, formerly Databricks Workflows). Jobs deleted in Databricks aren't imported.
• Authentication with a personal access token only. OAuth machine-to-machine (M2M) authentication is not supported.

Network Access

• Default: Sifflet connects to your Databricks workspace over the internet, on port 443, so the workspace must be reachable from the public internet.
• IP allowlisting: if you restrict access to your workspace with Databricks IP access lists, add the IP addresses of your Sifflet instance, listed in Settings > Integration Preferences.

Other Prerequisites

• The Databricks source for the same workspace, to link jobs to the tables they write. Without it, Sifflet still imports your jobs and their run status, but they aren't linked to any table.
• The system.lakeflow and system.access system schemas are enabled in your Databricks account. system.lakeflow is enabled by default. See Databricks Other Prerequisites.

Connect Sifflet to Databricks Workflows

Step 1: Create a Service Principal in Databricks

1. If you already connected the Databricks source, reuse its service principal, its token, and its SQL warehouse. Otherwise, create the service principal, allow it to use the SQL warehouse, and generate its personal access token, as described in Databricks Step 1. To use a user's token instead, see Alternative: User Personal Access Token.

2. Run the following script in a SQL editor of your workspace, as a metastore admin. Replace <APPLICATION_ID> with the Application ID of the service principal (or the user's email address).

   SQL

   GRANT USE CATALOG ON CATALOG system TO `<APPLICATION_ID>`;
   GRANT USE SCHEMA ON SCHEMA system.lakeflow TO `<APPLICATION_ID>`;
   GRANT USE SCHEMA ON SCHEMA system.access TO `<APPLICATION_ID>`;
   GRANT SELECT ON TABLE system.lakeflow.jobs TO `<APPLICATION_ID>`;
   GRANT SELECT ON TABLE system.lakeflow.job_run_timeline TO `<APPLICATION_ID>`;
   GRANT SELECT ON TABLE system.access.workspaces_latest TO `<APPLICATION_ID>`;

Keep the personal access token for step 2.

Step 2: Add the Databricks Workflows Source in Sifflet

1. Create the credential: if the Databricks source already uses this token, reuse its credential. Otherwise, go to Integrations > Credentials, click New credential, and paste the personal access token as-is:

   Text

   dapi0123456789abcdef0123456789abcdef

2. Create the source: go to Integrations > Sources, click New source, select Databricks Jobs, and fill in the form with the values of the Connection details tab of the SQL warehouse:

   Field	Description	Example
   Source name	The name of the source in Sifflet	Databricks Jobs Production
   Host	The Server hostname of the workspace, without https://. Sifflet imports only the jobs of this workspace.	dbc-a1b2c3d4-e5f6.cloud.databricks.com
   Port	The Port from the connection details	443
   Http Path	The HTTP path from the connection details	/sql/1.0/warehouses/1234567890abcdef
   Credential	The credential that contains the token	databricks-token

3. Scope: there is no scope to select. Sifflet imports all the jobs of the workspace. See Integrations Management for the refresh frequency and scheduling.

4. Test and save: click Test. Sifflet checks the permissions on the system tables, and shows a report with any missing permission. Then save the source. Sifflet then refreshes the source: follow the status and logs of the refresh in the source details, from Integrations > Sources. Your jobs appear in the Data Catalog once the refresh completes.

Troubleshooting

Catalog Permissions Are Missing

Symptom

Missing required catalog permission: SELECT on TABLE system.lakeflow.job_run_timeline

Cause

The service principal can't read one of the system tables listed in Permissions Required, or the system.access schema isn't enabled in your account.

Fix

1. Check that the system.access system schema is enabled. See Other Prerequisites.
2. Run the grants of Step 1 again, then click Test.

Jobs Aren't Linked to Tables

Symptom

Your jobs appear in the Data Catalog, but the lineage doesn't show which tables they write.

Cause

Sifflet builds these links from the lineage that the Databricks source reads in system.access.column_lineage. The links are missing when no Databricks source covers the tables the job writes, or when Unity Catalog didn't capture lineage for the job. For tables written by Lakeflow Declarative Pipelines, Sifflet also needs the optional system.lakeflow.pipeline_update_timeline permission on the Databricks source.

Fix

1. Check that a Databricks source includes the schemas the job writes to.
2. For tables written by pipelines, grant the optional job lineage permissions of the Databricks source.
3. Wait for the next refresh of the Databricks source, or refresh it manually.