Release: Continuous Scan for all ML-based Monitors

✨ Feature Highlights

Continuous Scan for ML-based Monitor Templates

Sifflet has just release a Continuous Scan feature for ML-based Monitor Templates.

It's a parameter that enables alerting on data quality issues occurring between subsequent Monitor Runs. If turned ON, it automatically includes all anomalies that happen since the last Monitor Run into the next Incident. It prevents existence of unmonitored periods of data for example in case misalignment between Time-based Data Aggregation and Schedule, or changing Time Offset value.

Read more about the Continuous Scan feature.

App version: v197

Release: Simplified Data Discovery Through Data Warehouse Collected Tags

✨ Feature Highlights

Simplified Data Discovery Through Data Warehouse Collected Tags

Sifflet now collects BigQuery table-level labels and Snowflake table and field-level tags and surfaces them in your Data Catalog.

This ensure the time your teams might have spent classifying their data at the data warehouse level also benefits data stakeholders leveraging the Data Catalog to discover and understand your data. Data warehouse collected tags indeed make it simpler for data stakeholders to find the asset they might be interested in by allowing them to filter on specific dimensions (team, environment, etc.). They also give them additional technical and business context about the asset as they browse available data assets.

🛠 Bug Fixes

• dbt integration: Fixed a bug where dbt models would sometimes not link properly with their corresponding tables
• dbt Core integration: Fixed a bug that was preventing dbt artifacts from being uploaded in some specific cases

App version: v195

Release: Differential Static Completeness Template, Dbt Ephemeral Models No Longer Create REF Table in the Lineage, Sifflet Airflow Operators Now Available for Self-Hosted Deployments, SSO Settings Page Revamped User Interface

✨ Feature Highlights

Differential Static Completeness

Sifflet is introducing a new Monitor Template: Differential Static Completeness (available in the Metadata category). It comes with two modes: Absolute and Difference with previous run value and it's especially useful for pipelines in which data gets re-uploaded in its entirety at every update. Use it to make sure that no data gets lost between runs by comparing the number of rows to either an absolute or the previous run value.

Read more about the Completeness (static) Monitor Template.

dbt Ephemeral Models No Longer Create REF Table in the Lineage

As stated in dbt docs, dbt ephemeral models are not directly built in the data warehouse. Therefore, the REF table referring to the DWH table will no longer be created in the lineage view.

Sifflet Airflow Operators Now Available for Self-Hosted Deployments

Self-hosted deployments now support Sifflet Airflow operators. Sifflet Airflow operators allow you to:

• Collect dbt models, tests, and lineage data as part of the dbt Core integration
• Integrate Sifflet monitors within your data pipelines to proactively prevent data quality issues from propagating (Flow Stopper)

Read more about Sifflet Airflow operators

SSO Settings Page Revamped User Interface

The user interface for the Settings > SSO page was revamped, making the SSO configuration experience smoother.

Read more about SSO

App version: v193

Release: Rolling Time Reference for Conditional Monitors

✨ Feature Highlights

Rolling Time Reference available for Conditional Monitors

There's now a new way to define a Time Condition for Conditional Monitors. By utilising the new TIME PERIOD Parameter, it's possible to define a Rolling Time Reference, mimicking the behaviour of a Time Window and/or Time Offset. Use it to create conditions as: monitor only data from the last 7 days with a 2 days offset.

Read more about the TIME PERIOD Parameter.

Failing Rows Columns Display Selection

Debugging unsuccessful Monitor Runs has just become easier. It's now possible to select columns to be displayed in the Failing Rows preview, greatly facilitating the process for complex datasets.

App version: v191

Release: Nested and Repeated Fields support for BigQuery

✨ Feature Highlights

Nested and Repeated Fields support for BigQuery

BigQuery users can now define all Sifflet monitoring types on tables with repeated and nested fields to provide more comprehensive monitoring coverage on this data asset type. Users can define field-level monitors on nested and repeated fields and rely on these fields for monitor parameters, such as using a repeated field as a time parameter or in the Group By parameter for multi-dimensional monitoring.

App version: v190

Release: Failing Rows CSV for Excel Download, Assess Key Business Terms' Monitoring Coverage

✨ Feature Highlights

Failing Rows Download for Excel

Facilitated debugging of Failing Rows for Excel users. An additional file format (CSV for Excel) is now available for download, ensuring seamless workflow.

Assess Your Key Business Terms' Monitoring Coverage

You can now easily find all monitors attached to your business terms from the Business Glossary tab, allowing you to quickly get an understanding of the monitoring coverage associated with your key business terms and metrics.

Read more about the Business Glossary

App version: v189

Release: Business Glossary for Data Assets Generally Available

✨ Feature Highlights

Business Glossary for Data Assets Is Generally Available

Business Glossary for Data Assets is officially generally available. Defining business terms and associating them to your data assets allows you to build a common reference guide for all data stakeholders and ensure they share a consistent understanding and interpretation of the data.

Read more about Business Glossary

App version: v188

Release: New Time-Based Data Aggregation Frequencies, Get Alerted on Monitor Misconfigurations, Smarter Data Catalog Search Parameters, Business Glossary Bug Fix

✨ Feature Highlights

New Time-Based Data Aggregation Frequencies

Monitors now support weekly and monthly Time-based Data Aggregation. With this new capability, users can aggregate data and metadata weekly and monthly to gain accuracy and better visibility when monitoring the health of pipelines running at this frequency or metrics that should be considered at this level of aggregation.

Read more about Time-based Data Aggregation

Get Alerted on Monitor Misconfigurations

Monitor alert recipients now get alerted on Requires attention and Technical error monitor runs. This ensures no monitor misconfiguration gets overlooked and that your monitoring coverage is complete at all times.

Read more about Monitor notifications

Smarter Data Catalog Search Parameters

Search parameters (filters, search terms, page, etc.) are now only retained when in the context of searching for a specific asset, thus simplifying navigation. These search parameters now also show up in the URL, making it easier to share these with your team and collaborate on specific Data Catalog views.

Read more about the Data Catalog

🛠 Bug Fixes

• Business Glossary: Fixed a bug that was causing an inaccurate Assets & monitors count on business terms

App version: v187

Release: Bulk Editing Monitor Business Terms

✨ New & Improved

• Business Glossary: You can now bulk add business terms to your monitors

App version: v186

Release: MS Teams integration bug fix

🛠 Fixes

• MS Teams Integration: Fixed a bug that was preventing notifications from being sent to MS Teams webhooks

App version: v185