BigQuery Lineage with the Data Lineage API

By default, Sifflet builds BigQuery lineage from your BigQuery job history (see BigQuery Collected Data). Sifflet can instead use the lineage that Google Cloud computes with the Data Lineage API, and falls back to the job history when the API returns no lineage.

This option is turned off by default. Contact Sifflet support to turn it on for your instance.

Permissions Required

Permission	Used for	Required
`datalineage.locations.searchLinks` (included in `roles/datalineage.viewer`) on the data project	Read the lineage computed by Google Cloud	Yes
`serviceusage.services.use` (included in `roles/serviceusage.serviceUsageConsumer`) on the billing project, or on the data project if you don't use a billing project	Bill Sifflet's Data Lineage API requests to this project	Yes
`roles/datacatalog.viewer` on the data project	Not used today. Granting it now avoids a new grant when Sifflet uses Data Catalog metadata in the future	Optional
`roles/bigquery.dataViewer` on the data project	Not used today: the custom role from the main setup already covers Sifflet's reads. Granting it now avoids a new grant for future features	Optional

Requirements

The Data Lineage API is enabled in your project.
Sifflet reaches the Data Lineage API over the internet, like the BigQuery API. See Network Access.

Set Up Lineage with the Data Lineage API

Contact Sifflet support to turn on this option for your instance.
Enable the Data Lineage API in your project, as described in the Google Cloud documentation.
Grant the Sifflet service account the Data Lineage Viewer role (roles/datalineage.viewer), or a custom role with datalineage.locations.searchLinks, on the data project.
Grant the Sifflet service account serviceusage.services.use, for example with the Service Usage Consumer role (roles/serviceusage.serviceUsageConsumer), on the billing project, or on the data project if you don't use a billing project.
Optional: grant the Data Catalog Viewer (roles/datacatalog.viewer) and BigQuery Data Viewer (roles/bigquery.dataViewer) roles on the data project. Sifflet doesn't use them today; granting them now avoids new grants for future features.

Lineage from the Data Lineage API appears after the next metadata refresh of the BigQuery source.