Field-level lineage

Having the lineage view at the table level is useful for many tasks. For instance, finding the root cause of a data issue, understanding data dependencies, and so on.
Sifflet now allows you to observe the lineage even further at field-level.
To enable this feature, Sifflet will need a read-access only to your SQL logs so that it can automatically detect the dependencies between fields.

You will find below:

Coverage: what cases are currently handled
Navigation: the tips to make the most out of your lineage view
Set up: the minimum access rights to enable this feature

Coverage

Most common SQL queries are supported today.
In terms of Data Warehouses, we currently support:

Snowflake
BigQuery
Redshift
Firebolt

Many more will be added regularly.

Navigation

To access the lineage view:

Go to "Catalog", select an asset, then "Lineage"

By clicking on the "+" you can expand to display the linked tables, either upstream or downstream.

📘
Tips - Expand downstream/upstream all at once
By holding the key SHIFT, you can expand all downstream/upstream data assets at once.

By clicking on the arrow on the left of the asset, you can see its fields and types:

By default, the links are at table level. To display the field-level lineage between two tables, you can display fields, find an example below:

Click the arrow at the bottom-left corner of the box to display the fields

The field level lineage will then be displayed

If you have a view with an overwhelming number of tables, it can be not very clear.
You can focus on a given table or field by:
- pressing the discover button on the top left OR holding the Control key on your keyboard
- then hover the data asset you want to highlight

The toolbar enables you to use additional features:

Funnel: To only display the fields that are linked to others
Rounded arrow: To go back to the original view
Up and down arrows: To hide or show all fields
Target: To centre the view
Camera: To export the lineage a png
Document: To export the lineage as csv
Extending arrow: To use full screen view

Set up

Sifflet needs read access on the SQL logs to enable this feature.

BigQuery: how to connect to BigQuery on Sifflet is detailed here
Snowflake: how to connect to Snowflake on Sifflet is detailed here
Redshift: how to connect to Redshift on Sifflet is detailed here