BigQuery Collected Data

This page lists what Sifflet imports from BigQuery once you connect your project.

Imported Assets

Sifflet imports the following objects from the datasets selected in the source:

• Tables
• Views
• Materialized views
• External tables (on Cloud Storage, Google Drive, and Google Sheets)

Collected Metadata

For all assets:

• Name, type, and description
• Column names, types, and descriptions, including nested fields
• Labels, imported as tags
• Data usage: which users recently read each table and how often, from the job history
• Asset update history (optional, disabled by default): the latest jobs that modified each table, from the job history of the data project and of the worker projects

Depending on the asset type:

• Tables: partitioning (time-unit or integer-range partitioning, partitioning field, and whether a partition filter is required)
• Views and materialized views: SQL definition
• External tables: source location (such as a Cloud Storage URI) and source format

Lineage

Sifflet builds table-level and field-level lineage between your BigQuery assets from the job history of the data project and of the worker projects. It parses the successful query jobs that write to a table: CREATE TABLE AS SELECT, INSERT, MERGE, and SELECT statements with a destination table. External tables are linked to their source location, such as a Cloud Storage URI.

If Sifflet support turned on lineage with the Data Lineage API for your instance, Sifflet uses the Google Cloud Data Lineage API first, and falls back to the job history when the API returns no lineage.

Sifflet also links your BigQuery tables to the assets of your other integrations, such as dbt models and BI dashboards. Query jobs labeled with airflow_dag_id and airflow_task_id are linked to the matching Airflow DAGs.

Data Freshness

Sifflet refreshes BigQuery metadata on the source's schedule (see Integrations Management for the default frequency and scheduling).

Limitations

• BigQuery ML models and table snapshots aren't imported, nor are tables without any column.
• External tables on Bigtable aren't supported.

Data Exposure

• Data preview and failing rows display rows of your BigQuery tables. Sifflet fetches them on demand and doesn't store them.
• The asset update history, disabled by default, stores the SQL statements that modified your tables, which can contain sensitive values.

See Security for the data Sifflet accesses and stores.