Release: Simplify Collaboration by Assigning Owners to Your Data Assets

✨ Feature Highlights

Simplify Collaboration by Assigning Owners to Your Data Assets

You can now assign individuals as owners of your data assets, enabling you to:

• Drive accountability
• Simplify collaboration by making it easier to know who to report issues and request changes to
• Assess impact in case of data quality issues
• Keep a data fleet clean by facilitating the identification of unused objects

Read more about ownership

Incidents from DBT Tests

Incidents in SIfflet are a powerful way to collaborate on the resolution of data issues. Incidents are now automatically created when Sifflet receives DBT test failures.

Incidents on DBT Test failures are a powerful way to track the resolution of your DBT model issues !

App version: v210

Release: Notifications on DBT monitors

✨ Feature Highlights

Notifications on DBT monitors

You can now use sifflet to alert when various dbt monitors fail. DBT monitors can now be configured with notification settings such as Slack/MS Teams alerting or Email Alerting.

Simply select your dbt monitors from the monitor page or from the data quality tab of an asset and click the edit button to add notification settings.

App version: v208

Release: Domain-Based Access Control, Automated Field-Level Lineage: Coverage Improvement on Bigquery and Snowflake, Get a More Granular Understanding of Your Data Pipelines Health With Tag-Based Dashboard Filtering, Data Quality As Code Supporting Continuous Scan

✨ Feature Highlights

Domain-Based Access Control

We are introducing domain-based access control to offer more flexibility when defining users permissions associated to multiple domains. For example, a user can now be a viewer (with a viewer role permissions) in Domain A, and editor (with an editor role permissions) in Domain B.
Learn more about it in the Access Control documentation page.

Automated Field-Level Lineage: Coverage Improvement on Bigquery and Snowflake

Sifflet’s automated field-level lineage supports temporary tables and incremental pipelines on Snowflake. We have also extended the support for BigQuery-specific SQL dialect, resulting in a higher coverage for all the transformations happening within BigQuery.

Get a More Granular Understanding of Your Data Pipelines Health With Tag-Based Dashboard Filtering

You can now filter the entire content of your dashboard (data assets, monitors, incidents, etc.) by any of your tags. This makes it easy for you to tailor this bird's eye view of your Sifflet account to specific data products, teams, and more.

Thanks to dashboard filtering, you can now:

• Get a high level understanding of the monitoring coverage and overall health of data pipelines associated with a specific data product
• Check out configuration improvement suggestions for your team's monitors
• Analyse the incidents that occurred on a specific environment over the past months

Read more about dashboard filtering

Data Quality As Code Supporting Continuous Scan

We've added support for continuous scan in Data Quality as code monitors.

continuousScan: Boolean       # (optional - default false) Enable continuous scan

App version: v207

Release: Sifflet Insights Chrome Extension Launch, Smart Regex Feature supported by the Open AI GPT-4 model, Updated Monitor Templates Library

✨ Feature Highlights

Sifflet Insights Chrome Extension Launch

Your BI dashboard is your go-to source for critical insights, and the accuracy and reliability of the data it displays are paramount. Sifflet Insights is here to reinvent your BI dashboard experience. Connected to your Sifflet account, it provides up-to-date information about upstream data quality issues directly within your preferred BI tool. Currently, it supports Looker and Tableau, with plans to expand to additional BI tools in the near future.

Available for download in the Chrome Web Store.

Sifflet AI Assistant support extended to the Open AI GPT-4 model

The AI Assistant combines multiple cutting-edge technologies to offer the best tailor-made support for Sifflet users. It's Smart Regex feature has just received a boost of the Open AI GPT-4 model. The functionality allows for generating complex regex syntax based on natural language input, at the same time ensuring privacy - no customer data or metadata is being passed to Open AI.

Updated Monitor Templates Library

Sifflet Monitor Templates Library is constantly evolving to better fulfil the changing needs of our users. We've introduced a new category, Table-level health, to help you focus on value, not implementation details. The new category includes all former Metadata Monitors, with the Completeness Templates rebranded to Volume, and Duplicates to Row-level Duplicates. Additionally, to clarify their functions, some Metrics and Smart Metrics templates got improved names, including Interlinked Metrics becoming Correlated Metrics.

App version: v204

Release: Smarter Monitor Search Parameters

✨ Feature Highlights

Smarter Monitor Search Parameters

Search parameters like filters and search terms, are from now retained in the Monitor List page, simplifying navigation and monitor maintenance workflow. Moreover, thanks to being included in the URL, they persist during link sharing, facilitating collaboration.

Read more about Monitors

App version: v201

Release: Continuous Scan for all ML-based Monitors

✨ Feature Highlights

Continuous Scan for ML-based Monitor Templates

Sifflet has just release a Continuous Scan feature for ML-based Monitor Templates.

It's a parameter that enables alerting on data quality issues occurring between subsequent Monitor Runs. If turned ON, it automatically includes all anomalies that happen since the last Monitor Run into the next Incident. It prevents existence of unmonitored periods of data for example in case misalignment between Time-based Data Aggregation and Schedule, or changing Time Offset value.

Read more about the Continuous Scan feature.

App version: v197

Release: Simplified Data Discovery Through Data Warehouse Collected Tags

✨ Feature Highlights

Simplified Data Discovery Through Data Warehouse Collected Tags

Sifflet now collects BigQuery table-level labels and Snowflake table and field-level tags and surfaces them in your Data Catalog.

This ensure the time your teams might have spent classifying their data at the data warehouse level also benefits data stakeholders leveraging the Data Catalog to discover and understand your data. Data warehouse collected tags indeed make it simpler for data stakeholders to find the asset they might be interested in by allowing them to filter on specific dimensions (team, environment, etc.). They also give them additional technical and business context about the asset as they browse available data assets.

🛠 Bug Fixes

• dbt integration: Fixed a bug where dbt models would sometimes not link properly with their corresponding tables
• dbt Core integration: Fixed a bug that was preventing dbt artifacts from being uploaded in some specific cases

App version: v195

Release: Differential Static Completeness Template, Dbt Ephemeral Models No Longer Create REF Table in the Lineage, Sifflet Airflow Operators Now Available for Self-Hosted Deployments, SSO Settings Page Revamped User Interface

✨ Feature Highlights

Differential Static Completeness

Sifflet is introducing a new Monitor Template: Differential Static Completeness (available in the Metadata category). It comes with two modes: Absolute and Difference with previous run value and it's especially useful for pipelines in which data gets re-uploaded in its entirety at every update. Use it to make sure that no data gets lost between runs by comparing the number of rows to either an absolute or the previous run value.

Read more about the Completeness (static) Monitor Template.

dbt Ephemeral Models No Longer Create REF Table in the Lineage

As stated in dbt docs, dbt ephemeral models are not directly built in the data warehouse. Therefore, the REF table referring to the DWH table will no longer be created in the lineage view.

Sifflet Airflow Operators Now Available for Self-Hosted Deployments

Self-hosted deployments now support Sifflet Airflow operators. Sifflet Airflow operators allow you to:

• Collect dbt models, tests, and lineage data as part of the dbt Core integration
• Integrate Sifflet monitors within your data pipelines to proactively prevent data quality issues from propagating (Flow Stopper)

Read more about Sifflet Airflow operators

SSO Settings Page Revamped User Interface

The user interface for the Settings > SSO page was revamped, making the SSO configuration experience smoother.

Read more about SSO

App version: v193

Release: Rolling Time Reference for Conditional Monitors

✨ Feature Highlights

Rolling Time Reference available for Conditional Monitors

There's now a new way to define a Time Condition for Conditional Monitors. By utilising the new TIME PERIOD Parameter, it's possible to define a Rolling Time Reference, mimicking the behaviour of a Time Window and/or Time Offset. Use it to create conditions as: monitor only data from the last 7 days with a 2 days offset.

Read more about the TIME PERIOD Parameter.

Failing Rows Columns Display Selection

Debugging unsuccessful Monitor Runs has just become easier. It's now possible to select columns to be displayed in the Failing Rows preview, greatly facilitating the process for complex datasets.

App version: v191

Release: Nested and Repeated Fields support for BigQuery

✨ Feature Highlights

Nested and Repeated Fields support for BigQuery

BigQuery users can now define all Sifflet monitoring types on tables with repeated and nested fields to provide more comprehensive monitoring coverage on this data asset type. Users can define field-level monitors on nested and repeated fields and rely on these fields for monitor parameters, such as using a repeated field as a time parameter or in the Group By parameter for multi-dimensional monitoring.

App version: v190