Field-level lineage

Having the lineage view at the table level is useful for many tasks. For instance, finding the root cause of a data issue, understanding data dependencies, and so on.
Sifflet now allows you to observe the lineage even further at field-level lineage.
To enable this feature, Sifflet will need a read-access only to your SQL logs so that it can automatically detect the dependencies between fields.

You will find below:

  • Coverage: what cases are currently handled
  • Navigation: the tips to make the most out of your lineage view
  • Set up: the minimum access rights to enable this feature

Coverage

Most common SQL queries are supported today.
In terms of Data Warehouses, we currently support:

  • Snowflake
  • BigQuery
  • Redshift
  • Firebolt
    Many more will be added regularly.

Navigation

To access the lineage view:

  • Go to "Catalog" -> tab "Lineage"

  • By clicking on the "+" you can expand to display the linked tables, either upstream or downstream.

  • By clicking on the table itself, you can see its fields and type:

  • By default, the links are at table level. To display the field-level lineage between two tables, you can display fields for both tables, find an example below:
When clicking on table "sales", it will show the field-level lineage between tables "sales" and "agg_by_event"When clicking on table "sales", it will show the field-level lineage between tables "sales" and "agg_by_event"

When clicking on table "sales", it will show the field-level lineage between tables "sales" and "agg_by_event"

The links at field-level are now displayed !The links at field-level are now displayed !

The links at field-level are now displayed !

  • If you have a view with an overwhelming number of tables, it can be not very clear.
    You can focus on a given table or field by:
    • pressing the discover button on the top left OR holding the Control key on your keyboard
    • then hover the data asset you want to highlight

Set up

Sifflet needs read access on the SQL logs to enable this feature.

  • BigQuery: how to connect to BigQuery on Sifflet is detailed here
    The additional permission required for Sifflet to access the SQL logs is the following:
bigquery.jobs.listAll
  • Snowflake: how to connect to BigQuery on Sifflet is detailed here
    The additional permissions required for Sifflet to access the SQL logs are the following:
-- Grant access to query history
grant imported privileges on database "SNOWFLAKE" to role identifier($role_name);

Did this page help you?