Search parameters like filters and search terms, are from now retained in the Monitor List page, simplifying navigation and monitor maintenance workflow. Moreover, thanks to being included in the URL, they persist during link sharing, facilitating collaboration.
Release: Continuous Scan for all ML-based Monitors
✨ Feature Highlights
Continuous Scan for ML-based Monitor Templates
Sifflet has just release a Continuous Scan feature for ML-based Monitor Templates.
It's a parameter that enables alerting on data quality issues occurring between subsequent Monitor Runs. If turned ON, it automatically includes all anomalies that happen since the last Monitor Run into the next Incident. It prevents existence of unmonitored periods of data for example in case misalignment between Time-based Data Aggregation and Schedule, or changing Time Offset value.
Release: Simplified Data Discovery Through Data Warehouse Collected Tags
✨ Feature Highlights
Simplified Data Discovery Through Data Warehouse Collected Tags
Sifflet now collects BigQuery table-level labels and Snowflake table and field-level tags and surfaces them in your Data Catalog.
This ensure the time your teams might have spent classifying their data at the data warehouse level also benefits data stakeholders leveraging the Data Catalog to discover and understand your data. Data warehouse collected tags indeed make it simpler for data stakeholders to find the asset they might be interested in by allowing them to filter on specific dimensions (team, environment, etc.). They also give them additional technical and business context about the asset as they browse available data assets.
🛠 Bug Fixes
dbt integration: Fixed a bug where dbt models would sometimes not link properly with their corresponding tables
dbt Core integration: Fixed a bug that was preventing dbt artifacts from being uploaded in some specific cases
Release: Differential Static Completeness Template, Dbt Ephemeral Models No Longer Create REF Table in the Lineage, Sifflet Airflow Operators Now Available for Self-Hosted Deployments, SSO Settings Page Revamped User Interface
✨ Feature Highlights
Differential Static Completeness
Sifflet is introducing a new Monitor Template: Differential Static Completeness (available in the Metadata category). It comes with two modes: Absolute and Difference with previous run value and it's especially useful for pipelines in which data gets re-uploaded in its entirety at every update. Use it to make sure that no data gets lost between runs by comparing the number of rows to either an absolute or the previous run value.
dbt Ephemeral Models No Longer Create REF Table in the Lineage
As stated in dbt docs, dbt ephemeral models are not directly built in the data warehouse. Therefore, the REF table referring to the DWH table will no longer be created in the lineage view.
Sifflet Airflow Operators Now Available for Self-Hosted Deployments
Self-hosted deployments now support Sifflet Airflow operators. Sifflet Airflow operators allow you to:
Release: Rolling Time Reference for Conditional Monitors
✨ Feature Highlights
Rolling Time Reference available for Conditional Monitors
There's now a new way to define a Time Condition for Conditional Monitors. By utilising the new TIME PERIOD Parameter, it's possible to define a Rolling Time Reference, mimicking the behaviour of a Time Window and/or Time Offset. Use it to create conditions as: monitor only data from the last 7 days with a 2 days offset.
Debugging unsuccessful Monitor Runs has just become easier. It's now possible to select columns to be displayed in the Failing Rows preview, greatly facilitating the process for complex datasets.
Release: Nested and Repeated Fields support for BigQuery
✨ Feature Highlights
Nested and Repeated Fields support for BigQuery
BigQuery users can now define all Sifflet monitoring types on tables with repeated and nested fields to provide more comprehensive monitoring coverage on this data asset type. Users can define field-level monitors on nested and repeated fields and rely on these fields for monitor parameters, such as using a repeated field as a time parameter or in the Group By parameter for multi-dimensional monitoring.
Release: Failing Rows CSV for Excel Download, Assess Key Business Terms' Monitoring Coverage
✨ Feature Highlights
Failing Rows Download for Excel
Facilitated debugging of Failing Rows for Excel users. An additional file format (CSV for Excel) is now available for download, ensuring seamless workflow.
Assess Your Key Business Terms' Monitoring Coverage
You can now easily find all monitors attached to your business terms from the Business Glossary tab, allowing you to quickly get an understanding of the monitoring coverage associated with your key business terms and metrics.
Release: Business Glossary for Data Assets Generally Available
✨ Feature Highlights
Business Glossary for Data Assets Is Generally Available
Business Glossary for Data Assets is officially generally available. Defining business terms and associating them to your data assets allows you to build a common reference guide for all data stakeholders and ensure they share a consistent understanding and interpretation of the data.
Release: New Time-Based Data Aggregation Frequencies, Get Alerted on Monitor Misconfigurations, Smarter Data Catalog Search Parameters, Business Glossary Bug Fix
✨ Feature Highlights
New Time-Based Data Aggregation Frequencies
Monitors now support weekly and monthly Time-based Data Aggregation. With this new capability, users can aggregate data and metadata weekly and monthly to gain accuracy and better visibility when monitoring the health of pipelines running at this frequency or metrics that should be considered at this level of aggregation.
Monitor alert recipients now get alerted on Requires attention and Technical error monitor runs. This ensures no monitor misconfiguration gets overlooked and that your monitoring coverage is complete at all times.
Search parameters (filters, search terms, page, etc.) are now only retained when in the context of searching for a specific asset, thus simplifying navigation. These search parameters now also show up in the URL, making it easier to share these with your team and collaborate on specific Data Catalog views.
