HomeChangelogBlog

Field-level lineage

Suggest Edits

Having the lineage view at the table level is useful for many tasks. For instance, finding the root cause of a data issue, understanding data dependencies, and so on.
Sifflet now allows you to observe the lineage even further at field-level.
To enable this feature, Sifflet will need a read-access only to your SQL logs so that it can automatically detect the dependencies between fields.

You will find below:

  • Coverage: what cases are currently handled
  • Navigation: the tips to make the most out of your lineage view
  • Set up: the minimum access rights to enable this feature

Coverage

Most common SQL queries are supported today.
In terms of Data Warehouses, we currently support:

  • Snowflake
  • BigQuery
  • Redshift
  • Firebolt

Many more will be added regularly.

Navigation

To access the lineage view:

  • Go to "Catalog", select an asset, then "Lineage"
  • By clicking on the "+" you can expand to display the linked tables, either upstream or downstream.

📘

Tips - Expand downstream/upstream all at once

By holding the key SHIFT, you can expand all downstream/upstream data assets at once.

  • By clicking on the arrow on the left of the asset, you can see its fields and types:
  • By default, the links are at table level. To display the field-level lineage between two tables, you can display fields, find an example below:
Click the arrow at the bottom-left corner of the box to display the fields

Click the arrow at the bottom-left corner of the box to display the fields

The field level lineage will then be displayed

The field level lineage will then be displayed

  • If you have a view with an overwhelming number of tables, it can be not very clear.
    You can focus on a given table or field by:
    • pressing the discover button on the top left OR holding the Control key on your keyboard
    • then hover the data asset you want to highlight

The toolbar enables you to use additional features:

  • Funnel: To only display the fields that are linked to others
  • Rounded arrow: To go back to the original view
  • Up and down arrows: To hide or show all fields
  • Target: To centre the view
  • Camera: To export the lineage a png
  • Document: To export the lineage as csv
  • Extending arrow: To use full screen view

Set up

Sifflet needs read access on the SQL logs to enable this feature.

  • BigQuery: how to connect to BigQuery on Sifflet is detailed here
  • Snowflake: how to connect to Snowflake on Sifflet is detailed here
  • Redshift: how to connect to Redshift on Sifflet is detailed here.

Updated 4 months ago