✨ Feature Highlights

Improve Your Data Documentation With Description Formatting

Take your data documentation to the next level by adding extensive formatting to your data assets' and business terms' descriptions. Data product owners and other members of the data team can now add all the relevant information to assets' description and structure it in a way that makes it easy to understand by other catalog users.

Markdown-formatted descriptions collected from providers such as dbt are now also properly interpreted, making it simpler for catalog users to leverage those for data self-service.

Read more about asset and business terms descriptions

Improvement - Monitoring Threshold Settings

We're currently in the process of improving the Monitor Creation Experience. Today's release impacts Threshold settings and makes them simpler and more intuitive to use!

App version: v314-317

✨ Feature Highlights

Merge Incidents

Sometimes incidents are found to be related, the new merge incident feature in the Incident List page allows you to combine incidents that are related into a single incident. This is very useful when using the automatic incident creation feature!

Note: Another recent feature released in a previous changelog also allows you to link multiple monitors to an existing incident!

Improved: Qualifications

Monitors with dynamic(ML) thresholds can sometimes be too sensitive or the data may change pattern completely! In those cases you want the model to adjust as quickly as possible to avoid unwanted anomalies.

We've improved the false positive and false negative qualifications to adapt faster and more effectively to the qualifications.

🛠 Fixes

• Snowflake: Fixed a bug that caused lineage discrepancies for Snowflake Enterprise.
• BigQuery: Fixed a bug that prevented Sifflet from generating accurate lineage when queries contain certain BigQuery-specific keywords.
• Incidents: Fixed a bug that was preventing edits on large number of incidents

App version: v308-313

✨ Feature Highlights

Snowflake: Configurable look-back period for lineage generation

The look-back period used by Sifflet when querying the Access History view to generate Snowflake lineage is now configurable and no longer limited to 7 days.

Power BI: Support for Power Query M queries with multiple upstreams

Sifflet can now parse table source expressions (built using Power Query M) that reference multiple upstream tables and connect them to the upstream datasets in the data platform.

Incidents: Improved Look and Feel

We've improved the Incident Page to make it more aligned with the Sifflet Experience. Many more improvements to the timeline and more messages !

Microsoft Teams Integration: Power Automate Workflows Support

Following up on Microsoft's Office 365 connectors deprecation notice, Sifflet modified the Microsoft Teams integration to support Power Automate workflows.

Read more about Microsoft Teams integration

Note that previously configured webhooks leveraging Connectors Incoming Webhooks mechanism are still supported. More details about the migration from Connectors Incoming Webhooks to Power Automate workflows will be shared with impacted users in the upcoming weeks.

Data Catalog & Monitors: Assets' Fully Qualified Names

Tables & Views and declared assets' fully qualified names are now surfaced on asset, lineage, and monitor wizard pages. This removes the need to manually tweak your data source name to be able to understand which asset you're looking at and ensure the monitor you are creating targets the right dataset.

Read more about assets' fully qualified names on assets, lineage, and monitor pages

Sifflet App for Slack on Slack App Directory

The Sifflet App for Slack is now officially available on Slack App Directory, making it easier for your teams to discover and set up the Sifflet integration for Slack in order to get alerted on data quality issues.

Read more about Sifflet App for Slack

Quick Access to Assets With "View in" Buttons

Quickly access your asset on its original technology platform thanks to the View in button that is now available on three additional technologies: Snowflake, BigQuery, and Databricks.

Read more about asset pages

App version: v300-307

✨ Feature Highlights

New Monitor Template: Referential Integrity

A common requirement of Sifflet Customers is to ensure data consistency by ensuring relationships between tables are correct. This was previously possible in a limited way using conditional monitors to ensure for example that the customers purchasing items in my orders tables were real customers that existed in the Customer table.

The new Template can be found in the field profiling section when two tables have been selected in the monitor wizard.

Additional Features available in the template.

• Incremental Time window: Only check new values in the table and save costs.
• Reverse integrity: Ensure that all items in the right table have a value on the left side. i.e. Check that every active store has a line item in the orders table.
• Parity checking: Ensure that both tables have exactly the same list of values.

Create Incidents Manually from monitors

Monitors in sifflet have been historically automatically created by having the "Create Incident on Failure" Setting turned on. We are now introducing the first of many upcoming incident improvements: The ability to create incidents manually and from multiple monitors.

From the monitor list page, Sifflet Users are now able to select multiple monitors to create a new incident from. This also enable a brand new capability: Multiple Monitors per incident, allowing the creation of incidents as an escalation and response to multiple monitors failing.

Monitors who are already linked to an active incident will be swapped and linked to the newly created one!

What's next:

• Very Soon! Merge incidents together, Link Monitors to existing incidents
• Later: Automatically group/suggest grouping of monitors into the same incident if related!

App version: v294-299

🛠 Fixes

• Sifflet CLI (0.3.14) - Fixes a bug causing exception on sifflet rules list.

App version: v291-293

✨ Feature Highlights

Ignore Alerts on Special Dates

There are many scenarios where certain days are going to be anomalies no matter what, and we want to avoid sifting through a sea of alerts to classify them as normal ! We've recently introduced a new functionality to exclude specific days from alerting and anomaly detection.

When creating monitors with a time window setting you can now specify calendars for which you want to exclude dates from anomaly detection. The Calendar list comes preloaded with public holidays calendars for multiple countries as well as other specific calendars to ignore Sundays or Weekends, while still ensuring the graph stays complete for these dates !

When a calendar is active on the monitor points that fall out of the range will be displayed as grey and won't trigger an alert, they will also indicate that the date is being excluded because of a specific calendar !

Custom Calendars

Sometimes standard calendars don't suit everyone's needs! You can also create your own custom calendars via the API

Programmatically Create and Maintain Sifflet Sources

You can now programmatically create and maintain your Sifflet sources thanks to our new sets of API endpoints for credentials and sources. These new endpoints make it easier for your teams to ensure the appropriate data observability coverage by simplifying the roll out of Sifflet on your data stack.

Learn more about the credentials and sources API endpoints

🛠 Fixes

• Fixed a bug that was preventing the + button from disappearing on the lineage after the deletion of declared assets

App version: v289-290

✨ Feature Highlights

Support for empty and whitespace strings in Null monitors

For textual fields, you can now select what values to check in Null monitors. This capability is provided both for monitors set up in the UI or in Monitors as Code (via the new nullValues parameter).

See Parameters list and example for every monitor type for more details.

🛠 Fixes

• The Fivetran integration will now skip connectors for which metadata can't be retrieved instead of generating an error.

App version: v285-286

✨ Feature Highlights

Power BI: Support for Dynamic Data Sources

Sifflet now supports the usage of dynamic parameters in Power BI source expressions, allowing the generation of accurate lineage even for dynamic data sources.

Amazon QuickSight and Fivetran URIs

Sifflet now supports URIs for Amazon QuickSight and Fivetran. You can now consequently tie declared assets to existing Amazon QuickSight and Fivetran assets, allowing you to go one step further in comprehensive data observability with end-to-end lineage and data cataloging.

Read more about Amazon QuickSight and Fivetran URIs.

Edit Incidents' Names

You can now edit your incidents' names to have them reflect the data quality issue at stake even more accurately than before.

Read more about incidents

App version: v282-284

✨ Feature Highlights

Batch Actions on Incidents

Managing Incidents is a big focus area for Sifflet at the moment as we know a lot of our customers want to manage incidents at scale! The new and improved Incident Search page allows you to now batch edit the Severity, Status and Assignments of incidents.

🔥 Improved Monitors as Code: For Loops and Templates

Many improvements to Monitors as Code today !

Alternative to Monitor UUIDs when defining monitors: friendlyId

Before:

For each monitor a UUID needed to be specifiedid: 7edf1177-1a3c-4d71-b85f-e38b773735b4 which was often difficult since a UUID had to be generated outside of the yaml.

friendlyId alternative

friendlyId is an alternate ID that only needs to be unique per dataset, this means a dataset cannot have two monitors with the same friendlyId.

kind: Monitor
version: 1
friendlyId: customerEmailUnique
datasets:
- name: sales
  datasource:
    name: mySqlDatabase
...

The above monitor has a friendlyId customerEmailUnique , only one of those monitors can be added to the sales table in our mysqlDatabase.

Alternative method to reference Datasets: URIs

Before: Referencing Datasets required knowledge of information specific to Sifflet.

Sifflet ID of dataset:

datasets:
- id: 70217023-1a89-4c0b-9b6a-c85192c918b3

Sifflet dataset name with either the id or the name of the datasource

datasets:
- name: Prices
  datasource: 
    name: BigQuery Data warehouse ## OR
    id: ce3e9dd9-b007-42b0-b884-8c419f7f6daa

Now with URIs

URIs are a sifflet agnostic way to define dataset. Find out more about URIs

datasets:
- uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.TABLE

📘

URIs

You can retrieve URIs from the Catalog's Asset Page.

Multiple Monitors Per File

Before : One file per Monitor

Now : You can now separate monitors with the --- yaml separator and have multiple monitors on the same page

kind: Monitor
version: 1
friendlyId: customer_id_format_check
name: Matches Regex Monitor on CUSTOMER_ID
...

---

kind: Monitor
version: 1
friendlyId: customer_id_duplicate_check
name: Unicity Monitor on CUSTOMER_ID

Templates

Before Each monitor definition needed to contain all the information, this meant a lot of information had to be duplicated, such as tags or alerting.

Now Templates enable the definition of parameters that can be imported into new monitor definitions

Simply define a partial (or complete ) monitor definition with a templateName

templateName: missionCriticalTags
tags:
- name: Mission Critical
notifications:
- kind: Email
  name: [email protected]
- kind: Slack
  name: Alerts
  id: dd6f06ec-fab1-4a87-9544-b113f496d61d

---

templateName: Unimportant
tags:
  - name: Unimportant

This templateName can then be extended

kind: Monitor
version: 1
friendlyId: myMonitor
name: My Monitor
extends:
- missionCriticalTags
...

---

kind: Monitor
version: 1
friendlyId: myOtherMonitor
name: My Other Monitor
extends:
- Unimportant
...

For Loops

Before Once again before each monitor had to be defined individually! However there are scenarios where I want to deploy the same monitor on multiple tables !

Now For loops allow you to define monitors on multiple tables easily !

for each dataset T:
  datasets:
  - uri: snowflake://sifflet-enterprise/DEMO.SE_ENV.ORDERS
  - uri: snowflake://sifflet-enterprise/DEMO.SE_ENV.CUSTOMERS
  - uri: snowflake://sifflet-enterprise/DEMO.SE_ENV.STG_CUSTOMERS
  monitors:
  - kind: Monitor
    version: 1
    friendlyId: customer_id_format_check
    name: "[${T.name}]Format monitor on CUSTOMER_ID" 
    description: ""
    incident:
      severity: Low
      message: ""
    parameters:
      kind: FieldFormat
      field: CUSTOMER_ID
      format:
        kind: Regex
        regex: "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$"

  ...

Wildcards and excludes

use include with * wildcards in your uris to apply to many datasets at the same time! then use exclude to exclude specific datasets

for each dataset T:
  datasets:
    include:
    - uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.*
    - uri: bigquery:*.*.*ORDERS
    exclude:
    - uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.MYTABLETOEXCLUDE

Dynamic Names

name: "[${T.name}]Format monitor on CUSTOMER_ID"

Use ${T.name} to reference the name of your table.

🛠 Fixes

• Fixed an issue where Sifflet pages would require a manual refresh to operate as expected after a new release
• Fixed an issue where in some instances, a recently edited user could not be saved
• Fixed an issue where some domains would not show up in the permissions modal.

App version: v279-281

🛠 Fixes

• Fixed an issue in Data Quality as Code and APIs where tag name validation was case sensitive despite tag names being case insensitive.
• Improved the experience around the release of new versions of the Sifflet application to make it less disruptive for users.

App version: v278