✨ New Features

dbt Impact Analysis for GitHub and GitLab

We're excited to announce the release of our first CI workflow for both GitHub and GitLab: an impact analysis workflow that uses Sifflet's lineage API to generate the potential impact of your dbt code changes and posts it as a pull request comment. With this workflow, you can detect and resolve potential issues before code changes are merged.

The potential impact consists of the list of assets downstream of the dbt model(s) you’re modifying. These assets can be other dbt models, BI dashboards, or any other asset cataloged in Sifflet.

The workflow comes packed with two filtering capabilities, using either asset types or tags to minimize noise and focus on assets that matter.

To activate this new feature, refer to the dedicated documentation for GitHub and GitLab.

Throughout the next months, we'll continue to release Sifflet-powered CI workflows that bring all of Sifflet's capabilities to your software development platform.

Manage Sifflet Credentials and Sources as Code With Sifflet Official Terraform Provider

Ensuring your data observability adapts to changes in your data environment can be challenging. With Sifflet official Terraform provider, you can now manage Sifflet credentials and sources as code, making it easy to automate their creation and maintenance. Leveraging Terraform resources also improves governance and security by simplifying versioning and auditing of any changes performed on credentials and sources.

Read more about Sifflet official Terraform provider

Secure and Private Data Observability With Private Link Support for AWS and Azure

We are excited to announce that Sifflet now supports Private Link for secure, private connectivity between Sifflet and your AWS and Azure sources. With Private Link, Sifflet can access sources located within your private network while ensuring that data traffic stays completely off the public internet. This feature allows your teams to benefit from the value of data observability regardless of your organisation's security and compliance requirements.

Read more about private links

👍 Improvements

Extended Airflow Support to Amazon MWAA and GCP Cloud Composer

Sifflet can now integrate with managed Apache Airflow environments on Amazon MWAA and GCP Cloud Composer. For setup details, you can check out the dedicated documentation:

• Amazon MWAA setup guide
• GCP Cloud Composer setup guide

More Accurate Usage Stats

Since Sifflet now inserts a JSON comment at the top of its queries, we excluded all Sifflet queries from usage stats. This ensures that usage information is more accurate and only represents your own usage.

Simplify Incident Management with Jira Status Changes Sync

You can now optionally enable status changes sync from Jira to Sifflet. This will ensure any Sifflet incident gets automatically moved to In progress or Closed • Fixed when its associated Jira issue gets transitioned to an In-progress or Done status. Enabling status changes sync allows you to benefit from Sifflet incident management troubleshooting and collaboration capabilities while ensuring a smooth incident management process thanks to automated closing of your Sifflet incidents.

Read more about Jira integration

Improve Data Trust With Sifflet Insights Browser Extension Support for Power BI Reports

Ensuring teams have access to dashboard health status while browsing their BI tools is key to ensure trust in your data. Sifflet is going one step further in improving data self-service reliability by now supporting Sifflet Insights browser extension on Power BI reports.

Read more about Sifflet Insights Browser Extension

🛠 Fixes

• Improved Model for Full Data Scan Monitors: When monitors query entire tables, the data often does not change much or have much seasonality, we've improved our models to be less noisy in these cases.
• Improvements to Monitor Pages: More coming but you'll have noticed we've started to improve the UX of the monitor page ! Enjoy!
• Improved Jira Integration: We fixed a bug that was preventing the manual creation of Jira issues in case of large number of projects and issue types on the Jira account.
• Improved Support: We've removed the ability to disable the access of Sifflet support teams to SaaS accounts to ensure optimised support experience.
• User creation: We fixed a bug that was preventing the creation of users in some specific cases.

App version: v335-347

✨ Feature Highlights

Sifflet AI Assistant Account-Level Controls

Users with the Admin system role can now control whether or not users should to be able to leverage Sifflet AI Assistant to generate metadata on Data Catalog assets. This makes it simpler to ensure your Sifflet account is in compliance with your organisation's requirements.

Read more about account preferences for Sifflet AI Assistant

SQL Table Tracer (STT): Sifflet's New Adaptive SQL Parser

With this release, we're introducing SQL Table Tracer (STT), our in-house-built adaptive SQL parser. STT, a generic ANTLR-based parser, will extract lineage for a wide range of integrations, starting with table-level lineage for Amazon Redshift and Azure Synapse next in line.

Stay tuned for an upcoming blog post where we'll go over our journey building STT and the significant improvements it brings with it, both in terms of efficiency and query parsing accuracy, over our legacy parser.

Improvements

Improved Monitor Run Schedule and Timezone Selection

Monitors will now default to having a schedule suitable for the monitor. The default will be daily or hourly depending on the type of monitor and the interface will apply the simpler "Standard Options" format.

We've also added a Timezone selector to both Cron Expressions and Standard options to allow worldwide businesses to easily schedule monitors to meet their business hours !

Improved Support of Concurrent dbt Runs

Previously, if the artifacts of multiple dbt runs (within the same project) were submitted in a short timeframe, Sifflet could occasionally miss processing some files. With this release, we've enhanced our support for concurrent dbt runs to ensure all dbt artifacts are correctly processed, even when received simultaneously.

Improved Credentials Security

Once created, your credentials can still be edited but they are no longer accessible from the UI, improving the security in terms of sources' access.

Read more about credentials

Quickly Access Power BI Reports With the “View In” Button

You can now swiftly access your Power BI reports from Sifflet corresponding asset pages thanks to the addition of the View in button on Power BI reports' asset pages.

Read more about asset pages

Fixes

• Improvements to false Positive Qualification to reduce noise

App version: v328-334

Small Improvements

Rich Text Descriptions in Monitors

Like in the catalog assets you are now able to add rich text descriptions to Monitors

Asset URIs from the Catalog Search page

Asset URIs are a new way of identifying assets which can be used in a variety of ways, whether that is in data quality as code or through APIs. You can now easily retrieve the URIs from the monitor search Page.

bigquery:sifflet-demo-project.sifflet_demo_data.dim_customers

App version: v323-327

✨ Feature Highlights

Track Data Quality Issues in Jira With the New Integration

You can now manually create Jira issues from Sifflet incidents using the new built-in Jira integration. Generating Jira issues ensures data quality problems detected by Sifflet fit into your existing incident and task management workflow and that no important issue gets accidentally overlooked.

If you're looking for a more automated workflow, note that you can still have Jira issues automatically created in case of Sifflet monitor failure.

Read more about Jira integration

Lineage: Extend Power BI native queries support to Snowflake connections

Sifflet now generates lineage between Power BI and Snowflake even in scenarios where you use native queries to interact with Snowflake data sources from Power BI.

🛠 Fixes

• Row-level Duplicates monitor - Include date type columns for Databricks and Hive
• Email monitor - Increase valid Email TLD length

App version: v318-322

✨ Feature Highlights

Improve Your Data Documentation With Description Formatting

Take your data documentation to the next level by adding extensive formatting to your data assets' and business terms' descriptions. Data product owners and other members of the data team can now add all the relevant information to assets' description and structure it in a way that makes it easy to understand by other catalog users.

Markdown-formatted descriptions collected from providers such as dbt are now also properly interpreted, making it simpler for catalog users to leverage those for data self-service.

Read more about asset and business terms descriptions

Improvement - Monitoring Threshold Settings

We're currently in the process of improving the Monitor Creation Experience. Today's release impacts Threshold settings and makes them simpler and more intuitive to use!

App version: v314-317

✨ Feature Highlights

Merge Incidents

Sometimes incidents are found to be related, the new merge incident feature in the Incident List page allows you to combine incidents that are related into a single incident. This is very useful when using the automatic incident creation feature!

Note: Another recent feature released in a previous changelog also allows you to link multiple monitors to an existing incident!

Improved: Qualifications

Monitors with dynamic(ML) thresholds can sometimes be too sensitive or the data may change pattern completely! In those cases you want the model to adjust as quickly as possible to avoid unwanted anomalies.

We've improved the false positive and false negative qualifications to adapt faster and more effectively to the qualifications.

🛠 Fixes

• Snowflake: Fixed a bug that caused lineage discrepancies for Snowflake Enterprise.
• BigQuery: Fixed a bug that prevented Sifflet from generating accurate lineage when queries contain certain BigQuery-specific keywords.
• Incidents: Fixed a bug that was preventing edits on large number of incidents

App version: v308-313

✨ Feature Highlights

Snowflake: Configurable look-back period for lineage generation

The look-back period used by Sifflet when querying the Access History view to generate Snowflake lineage is now configurable and no longer limited to 7 days.

Power BI: Support for Power Query M queries with multiple upstreams

Sifflet can now parse table source expressions (built using Power Query M) that reference multiple upstream tables and connect them to the upstream datasets in the data platform.

Incidents: Improved Look and Feel

We've improved the Incident Page to make it more aligned with the Sifflet Experience. Many more improvements to the timeline and more messages !

Microsoft Teams Integration: Power Automate Workflows Support

Following up on Microsoft's Office 365 connectors deprecation notice, Sifflet modified the Microsoft Teams integration to support Power Automate workflows.

Read more about Microsoft Teams integration

Note that previously configured webhooks leveraging Connectors Incoming Webhooks mechanism are still supported. More details about the migration from Connectors Incoming Webhooks to Power Automate workflows will be shared with impacted users in the upcoming weeks.

Data Catalog & Monitors: Assets' Fully Qualified Names

Tables & Views and declared assets' fully qualified names are now surfaced on asset, lineage, and monitor wizard pages. This removes the need to manually tweak your data source name to be able to understand which asset you're looking at and ensure the monitor you are creating targets the right dataset.

Read more about assets' fully qualified names on assets, lineage, and monitor pages

Sifflet App for Slack on Slack App Directory

The Sifflet App for Slack is now officially available on Slack App Directory, making it easier for your teams to discover and set up the Sifflet integration for Slack in order to get alerted on data quality issues.

Read more about Sifflet App for Slack

Quick Access to Assets With "View in" Buttons

Quickly access your asset on its original technology platform thanks to the View in button that is now available on three additional technologies: Snowflake, BigQuery, and Databricks.

Read more about asset pages

App version: v300-307

✨ Feature Highlights

New Monitor Template: Referential Integrity

A common requirement of Sifflet Customers is to ensure data consistency by ensuring relationships between tables are correct. This was previously possible in a limited way using conditional monitors to ensure for example that the customers purchasing items in my orders tables were real customers that existed in the Customer table.

The new Template can be found in the field profiling section when two tables have been selected in the monitor wizard.

Additional Features available in the template.

• Incremental Time window: Only check new values in the table and save costs.
• Reverse integrity: Ensure that all items in the right table have a value on the left side. i.e. Check that every active store has a line item in the orders table.
• Parity checking: Ensure that both tables have exactly the same list of values.

Create Incidents Manually from monitors

Monitors in sifflet have been historically automatically created by having the "Create Incident on Failure" Setting turned on. We are now introducing the first of many upcoming incident improvements: The ability to create incidents manually and from multiple monitors.

From the monitor list page, Sifflet Users are now able to select multiple monitors to create a new incident from. This also enable a brand new capability: Multiple Monitors per incident, allowing the creation of incidents as an escalation and response to multiple monitors failing.

Monitors who are already linked to an active incident will be swapped and linked to the newly created one!

What's next:

• Very Soon! Merge incidents together, Link Monitors to existing incidents
• Later: Automatically group/suggest grouping of monitors into the same incident if related!

App version: v294-299

🛠 Fixes

• Sifflet CLI (0.3.14) - Fixes a bug causing exception on sifflet rules list.

App version: v291-293

✨ Feature Highlights

Ignore Alerts on Special Dates

There are many scenarios where certain days are going to be anomalies no matter what, and we want to avoid sifting through a sea of alerts to classify them as normal ! We've recently introduced a new functionality to exclude specific days from alerting and anomaly detection.

When creating monitors with a time window setting you can now specify calendars for which you want to exclude dates from anomaly detection. The Calendar list comes preloaded with public holidays calendars for multiple countries as well as other specific calendars to ignore Sundays or Weekends, while still ensuring the graph stays complete for these dates !

When a calendar is active on the monitor points that fall out of the range will be displayed as grey and won't trigger an alert, they will also indicate that the date is being excluded because of a specific calendar !

Custom Calendars

Sometimes standard calendars don't suit everyone's needs! You can also create your own custom calendars via the API

Programmatically Create and Maintain Sifflet Sources

You can now programmatically create and maintain your Sifflet sources thanks to our new sets of API endpoints for credentials and sources. These new endpoints make it easier for your teams to ensure the appropriate data observability coverage by simplifying the roll out of Sifflet on your data stack.

Learn more about the credentials and sources API endpoints

🛠 Fixes

• Fixed a bug that was preventing the + button from disappearing on the lineage after the deletion of declared assets

App version: v289-290