Release: Model training improvements, lookback period improvements, improved SSO to support BoxyHQ

✨ Feature Highlights

Model Training Improvements

Optimal hyper parameters for machine learning models are now computed and applied on initial run. Ensuring the first run of the monitor runs optimally. Hyper parameters for each model are also periodically improved and recalculated by sifflet.

Lookback Period Improvements

Monitors now alert on the Lookback Period. The lookback period is used to add a past buffer to monitor runs to account for pipelines who update or backfill past data points. This was previously used only to update graphs and improved the model for better predictions.

What Changed: If a data point in the lookback window was within the expected range at the time of the initial run but has since then been changed the Sifflet monitor will treat that change as a new anomaly

Additionally if a data point was an anomaly when it was first generated but a subsequent run identifies that the point is now in the correct range. It will automatically be marked as Fixed

🛠 Fixes

• Improved SSO to support BoxyHQ

App version: v217

Release: Automatic configuration of users' email addresses

✨ Feature Highlights

Automatic configuration of users' email addresses

Sifflet users' email addresses are now automatically provisioned to your mail integration. Your teams consequently no longer have to first go through the step of configuring their personal email addresses at the mail integration level before using them in monitors, making the first step to monitor creation swifter.

Read more about the email integration

App version: v216

Release: Continuous sensitivity, Incident creation toggle, monitor search improvements, programmatically manage Sifflet users access controls

✨ Feature Highlights

ML Monitoring - Continuous Sensitivity

Sensitivity for Anomaly detection has always been a complex issue. To reduce noise, it is ideal to find the best sensitivity value that fits the most valid points and still detects anomalies in the data when they do occur. Sifflet has now switched from having set sensitivity thresholds to allowing for even more granular control of anomaly detection.

Mapped values to previous setup: Low -> 25, Medium -> 50, High -> 85

ML Monitoring - Sensitivity recommendations

Along with the change, Sifflet's AI assistant can now deliver more specific and efficient Sensitivity suggestions, aiming to fit as many valid points as possible (including those marked as false positives !) and still detect anomalies for true errors!

Monitors - Incident Creation Toggle

There are some monitors for which you don't necessarily want to create a Data Incident, so we've added a toggle that let's you turn off SIfflet's automatic creation of an incident when a monitor fails. This is great for monitors you want to review periodically without initiating any resolution processes for!
We use this feature on some of our internal sifflet instance where we detect anomalous patterns in Sifflet usage. If a Customer is creating more monitors than usual we want to proactively reach out to them to help but we don't necessarily need an incident for that! 💡

Monitors as code:

incident:                   # (REQUIRED) 
  severity:                 # (REQUIRED) Severity of the incident
    "Low" | "Moderate" | "High" | "Critical"
  message: String           # (optional - default null) Custom message to add to the incident and notifications
  createOnFailure: Boolean

Monitors - Search Improvements

We've improved the monitor search page to further improve the way you the monitors you want.

• Improved freetext search patterns
• Count previews in filters to identify quickly how many monitors belong to each filter option
• Filter on business terms

Learn more about monitors

Programmatically manage Sifflet users' access controls

User management API endpoints were updated to allow you to programmatically associate roles to the domains your Sifflet users have access to. This simplifies access controls management and ensures only the right people access the appropriate data inside of Sifflet.

Learn more about API-based user management

App version: v213

Release: dbt Cloud integration bug fix, Data Catalog search bug fix

🛠 Fixes

dbt Cloud - Fixes and Future Proofing

A dbt Cloud outage last week temporarily broke our ingestion of dbt test runs for dbt Cloud integrations. Only customers that ran tests during the outage were affected. We have introduced extra error handling to handle the unforeseen "malformed" run data reflecting the impacted test runs.

Data Catalog - Search Failures

Fixed a bug that was causing searches on the Data Catalog to fail when too many filters were selected.

App version: v212

Release: Monitor Improvements - History refetch

✨ Feature Highlights

Monitor Improvements - History refetch

When changing parameters for a monitor with a historical time window, Sifflet will sometimes refetch the historical data to ensure the monitor has an accurate graph representation of the data being monitored.

For example, when adding a WHERE clause to a monitor, Sifflet refetches the history to ensure all past data points reflect the filtered data rather than the data without the WHERE clause. This also ensures the model is trained on accurate data.

Optimisations to the historical refetches have been made to retrigger only when needed. A monitor will now avoid a refresh when sensitivity is changed or when the time window period is reduced and will trigger only for cases such as increasing the time window period, filtering the data on a new condition or grouping the data by a subcategory.

App version: v211

Release: Simplify Collaboration by Assigning Owners to Your Data Assets

✨ Feature Highlights

Simplify Collaboration by Assigning Owners to Your Data Assets

You can now assign individuals as owners of your data assets, enabling you to:

• Drive accountability
• Simplify collaboration by making it easier to know who to report issues and request changes to
• Assess impact in case of data quality issues
• Keep a data fleet clean by facilitating the identification of unused objects

Read more about ownership

Incidents from DBT Tests

Incidents in SIfflet are a powerful way to collaborate on the resolution of data issues. Incidents are now automatically created when Sifflet receives DBT test failures.

Incidents on DBT Test failures are a powerful way to track the resolution of your DBT model issues !

App version: v210

Release: Notifications on DBT monitors

✨ Feature Highlights

Notifications on DBT monitors

You can now use sifflet to alert when various dbt monitors fail. DBT monitors can now be configured with notification settings such as Slack/MS Teams alerting or Email Alerting.

Simply select your dbt monitors from the monitor page or from the data quality tab of an asset and click the edit button to add notification settings.

App version: v208

Release: Domain-Based Access Control, Automated Field-Level Lineage: Coverage Improvement on Bigquery and Snowflake, Get a More Granular Understanding of Your Data Pipelines Health With Tag-Based Dashboard Filtering, Data Quality As Code Supporting Continuous Scan

✨ Feature Highlights

Domain-Based Access Control

We are introducing domain-based access control to offer more flexibility when defining users permissions associated to multiple domains. For example, a user can now be a viewer (with a viewer role permissions) in Domain A, and editor (with an editor role permissions) in Domain B.
Learn more about it in the Access Control documentation page.

Automated Field-Level Lineage: Coverage Improvement on Bigquery and Snowflake

Sifflet’s automated field-level lineage supports temporary tables and incremental pipelines on Snowflake. We have also extended the support for BigQuery-specific SQL dialect, resulting in a higher coverage for all the transformations happening within BigQuery.

Get a More Granular Understanding of Your Data Pipelines Health With Tag-Based Dashboard Filtering

You can now filter the entire content of your dashboard (data assets, monitors, incidents, etc.) by any of your tags. This makes it easy for you to tailor this bird's eye view of your Sifflet account to specific data products, teams, and more.

Thanks to dashboard filtering, you can now:

• Get a high level understanding of the monitoring coverage and overall health of data pipelines associated with a specific data product
• Check out configuration improvement suggestions for your team's monitors
• Analyse the incidents that occurred on a specific environment over the past months

Read more about dashboard filtering

Data Quality As Code Supporting Continuous Scan

We've added support for continuous scan in Data Quality as code monitors.

continuousScan: Boolean       # (optional - default false) Enable continuous scan

App version: v207

Release: Sifflet Insights Chrome Extension Launch, Smart Regex Feature supported by the Open AI GPT-4 model, Updated Monitor Templates Library

✨ Feature Highlights

Sifflet Insights Chrome Extension Launch

Your BI dashboard is your go-to source for critical insights, and the accuracy and reliability of the data it displays are paramount. Sifflet Insights is here to reinvent your BI dashboard experience. Connected to your Sifflet account, it provides up-to-date information about upstream data quality issues directly within your preferred BI tool. Currently, it supports Looker and Tableau, with plans to expand to additional BI tools in the near future.

Available for download in the Chrome Web Store.

Sifflet AI Assistant support extended to the Open AI GPT-4 model

The AI Assistant combines multiple cutting-edge technologies to offer the best tailor-made support for Sifflet users. It's Smart Regex feature has just received a boost of the Open AI GPT-4 model. The functionality allows for generating complex regex syntax based on natural language input, at the same time ensuring privacy - no customer data or metadata is being passed to Open AI.

Updated Monitor Templates Library

Sifflet Monitor Templates Library is constantly evolving to better fulfil the changing needs of our users. We've introduced a new category, Table-level health, to help you focus on value, not implementation details. The new category includes all former Metadata Monitors, with the Completeness Templates rebranded to Volume, and Duplicates to Row-level Duplicates. Additionally, to clarify their functions, some Metrics and Smart Metrics templates got improved names, including Interlinked Metrics becoming Correlated Metrics.

App version: v204

Release: Smarter Monitor Search Parameters

✨ Feature Highlights

Smarter Monitor Search Parameters

Search parameters like filters and search terms, are from now retained in the Monitor List page, simplifying navigation and monitor maintenance workflow. Moreover, thanks to being included in the URL, they persist during link sharing, facilitating collaboration.

Read more about Monitors

App version: v201