Like in the catalog assets you are now able to add rich text descriptions to Monitors
Asset URIs from the Catalog Search page
Asset URIs are a new way of identifying assets which can be used in a variety of ways, whether that is in data quality as code or through APIs. You can now easily retrieve the URIs from the monitor search Page.
Track Data Quality Issues in Jira With the New Integration
You can now manually create Jira issues from Sifflet incidents using the new built-in Jira integration. Generating Jira issues ensures data quality problems detected by Sifflet fit into your existing incident and task management workflow and that no important issue gets accidentally overlooked.
If you're looking for a more automated workflow, note that you can still have Jira issues automatically created in case of Sifflet monitor failure.
Lineage: Extend Power BI native queries support to Snowflake connections
Sifflet now generates lineage between Power BI and Snowflake even in scenarios where you use native queries to interact with Snowflake data sources from Power BI.
🛠 Fixes
Row-level Duplicates monitor - Include date type columns for Databricks and Hive
Improve Your Data Documentation With Description Formatting
Take your data documentation to the next level by adding extensive formatting to your data assets' and business terms' descriptions. Data product owners and other members of the data team can now add all the relevant information to assets' description and structure it in a way that makes it easy to understand by other catalog users.
Markdown-formatted descriptions collected from providers such as dbt are now also properly interpreted, making it simpler for catalog users to leverage those for data self-service.
We're currently in the process of improving the Monitor Creation Experience. Today's release impacts Threshold settings and makes them simpler and more intuitive to use!
Sometimes incidents are found to be related, the new merge incident feature in the Incident List page allows you to combine incidents that are related into a single incident. This is very useful when using the automatic incident creation feature!
Note: Another recent feature released in a previous changelog also allows you to link multiple monitors to an existing incident!
Improved: Qualifications
Monitors with dynamic(ML) thresholds can sometimes be too sensitive or the data may change pattern completely! In those cases you want the model to adjust as quickly as possible to avoid unwanted anomalies.
We've improved the false positive and false negative qualifications to adapt faster and more effectively to the qualifications.
Just one false positive qualification adapts my model to not alert on similar datapoints !
🛠 Fixes
Snowflake: Fixed a bug that caused lineage discrepancies for Snowflake Enterprise.
BigQuery: Fixed a bug that prevented Sifflet from generating accurate lineage when queries contain certain BigQuery-specific keywords.
Incidents: Fixed a bug that was preventing edits on large number of incidents
Snowflake: Configurable look-back period for lineage generation
The look-back period used by Sifflet when querying the Access History view to generate Snowflake lineage is now configurable and no longer limited to 7 days.
Power BI: Support for Power Query M queries with multiple upstreams
Sifflet can now parse table source expressions (built using Power Query M) that reference multiple upstream tables and connect them to the upstream datasets in the data platform.
Incidents: Improved Look and Feel
We've improved the Incident Page to make it more aligned with the Sifflet Experience. Many more improvements to the timeline and more messages !
Microsoft Teams Integration: Power Automate Workflows Support
Note that previously configured webhooks leveraging Connectors Incoming Webhooks mechanism are still supported. More details about the migration from Connectors Incoming Webhooks to Power Automate workflows will be shared with impacted users in the upcoming weeks.
Data Catalog & Monitors: Assets' Fully Qualified Names
Tables & Views and declared assets' fully qualified names are now surfaced on asset, lineage, and monitor wizard pages. This removes the need to manually tweak your data source name to be able to understand which asset you're looking at and ensure the monitor you are creating targets the right dataset.
The Sifflet App for Slack is now officially available on Slack App Directory, making it easier for your teams to discover and set up the Sifflet integration for Slack in order to get alerted on data quality issues.
Quickly access your asset on its original technology platform thanks to the View in button that is now available on three additional technologies: Snowflake, BigQuery, and Databricks.
A common requirement of Sifflet Customers is to ensure data consistency by ensuring relationships between tables are correct. This was previously possible in a limited way using conditional monitors to ensure for example that the customers purchasing items in my orders tables were real customers that existed in the Customer table.
The new Template can be found in the field profiling section when two tables have been selected in the monitor wizard.
Additional Features available in the template.
Incremental Time window: Only check new values in the table and save costs.
Reverse integrity: Ensure that all items in the right table have a value on the left side. i.e. Check that every active store has a line item in the orders table.
Parity checking: Ensure that both tables have exactly the same list of values.
Create Incidents Manually from monitors
Monitors in sifflet have been historically automatically created by having the "Create Incident on Failure" Setting turned on. We are now introducing the first of many upcoming incident improvements: The ability to create incidents manually and from multiple monitors.
From the monitor list page, Sifflet Users are now able to select multiple monitors to create a new incident from. This also enable a brand new capability: Multiple Monitors per incident, allowing the creation of incidents as an escalation and response to multiple monitors failing.
Monitors who are already linked to an active incident will be swapped and linked to the newly created one!
What's next:
Very Soon! Merge incidents together, Link Monitors to existing incidents
Later: Automatically group/suggest grouping of monitors into the same incident if related!
There are many scenarios where certain days are going to be anomalies no matter what, and we want to avoid sifting through a sea of alerts to classify them as normal ! We've recently introduced a new functionality to exclude specific days from alerting and anomaly detection.
When creating monitors with a time window setting you can now specify calendars for which you want to exclude dates from anomaly detection. The Calendar list comes preloaded with public holidays calendars for multiple countries as well as other specific calendars to ignore Sundays or Weekends, while still ensuring the graph stays complete for these dates !
When a calendar is active on the monitor points that fall out of the range will be displayed as grey and won't trigger an alert, they will also indicate that the date is being excluded because of a specific calendar !
Custom Calendars
Sometimes standard calendars don't suit everyone's needs! You can also create your own custom calendars via the API
Programmatically Create and Maintain Sifflet Sources
You can now programmatically create and maintain your Sifflet sources thanks to our new sets of API endpoints for credentials and sources. These new endpoints make it easier for your teams to ensure the appropriate data observability coverage by simplifying the roll out of Sifflet on your data stack.
Support for empty and whitespace strings in Null monitors
For textual fields, you can now select what values to check in Null monitors. This capability is provided both for monitors set up in the UI or in Monitors as Code (via the new nullValues parameter).
Sifflet now supports the usage of dynamic parameters in Power BI source expressions, allowing the generation of accurate lineage even for dynamic data sources.
Amazon QuickSight and Fivetran URIs
Sifflet now supports URIs for Amazon QuickSight and Fivetran. You can now consequently tie declared assets to existing Amazon QuickSight and Fivetran assets, allowing you to go one step further in comprehensive data observability with end-to-end lineage and data cataloging.
