We’ve enhanced our platform’s authentication experience to allow users to log in to Sifflet using either Single Sign-On (SSO) or a traditional username/password combination.
This optional feature provides greater flexibility for organizations with diverse security and user management needs. Teams can leverage SSO for a centralized authentication through their preferred identity provider (IdP) while still maintaining the option for direct login when needed. This ensures accessibility for all users, including contractors or partners who might need to access Sifflet data observability insights without being able to sign in through the organization's IdP. It also ensures you can access Sifflet at all times, including in the event of an IdP outage.
We’ve completely reimagined how Sifflet integrates with dbt to make your workflows smoother and more powerful. This revamp introduces exciting new features that seamlessly combine dbt models with datasets and enrich your lineage and catalog experience with dbt metadata.
dbt Models + Datasets: A Unified Asset Experience
Previously, dbt models and the datasets they generated existed as separate entities in Sifflet, with distinct catalog entries and asset pages. With this release, we’ve combined them into a single asset and brought new dbt metadata to Sifflet. Here’s what this means for you:
In the Catalog: Enhanced Metadata for dbt-Generated Datasets
Datasets created by dbt now include key dbt metadata:
Last Execution Timestamp: See exactly when the corresponding dbt model was last run.
Last Execution Status: Quickly identify if the latest execution was successful, failed, or skipped.
Dataset catalog entry with dbt metadata
On the Asset Page: Introducing the New dbt Tab
The dataset’s asset page now features a dedicated dbt tab, consolidating previously scattered dbt information in one place. This is just the beginning—soon, this tab will include even more insights like the model's group, its access modifier, and custom dbt metadata defined using the meta field.
The brand-new dbt tab
Streamlined Lineage Graph: Fewer Nodes, More Insights
The Sifflet lineage graph is now cleaner and more intuitive. dbt models are no longer displayed as separate nodes. Instead, their metadata is integrated into the dataset node, reducing clutter and redundancies. The result? A more streamlined view with richer, consolidated information.
The new lineage graph with dbt metadata
Looking Ahead: More dbt Features on the Horizon
This is just the first phase of our dbt integration revamp. Here’s a sneak peek at what’s coming next:
Cost & Performance Monitoring: Gain insights into the resource usage and efficiency of dbt runs.
Leveraging Custom Metadata: Use dbt’s custom metadata directly in Sifflet for advanced configurations.
dbt-based Monitor Setup: Define and configure Sifflet monitors directly within your dbt YAML files.
We’re excited about this leap forward and hope you are too. Want to see it in action? Reach out to our team to learn more!
You can now customize the system and domain permissions you want to grant users created through Just-In-Time (JIT) user provisioning. This addition allows for operational efficiency while minimizing risks of over-privileged accounts.
Monitors in Sifflet can be created in multiple ways, whether they come from DBT test runs, are created via the interface, or are deployed within a code-based workspace. New filters in the Monitor page allow for filtering based on these creation methods!
Custom Thresholds for SQL and Conditional Monitors
We recently added the ability to apply custom threshold settings to many monitor templates in SIfflet, we're bringing this functionality to SQL and conditional Monitors.
Rather than alert on any matching result you can now alert:
Static: If more than X results match the condition
Dynamic: If the number of results diverges from the usual trend
Relative: If the number of results increases or drops by a certain value/percent
You can now use the Webhooks integration to trigger events to your endpoints in case Sifflet detects a data quality issue. This makes it possible for your teams to integrate Sifflet with virtually any collaboration tools they might be using (e.g. ServiceNow, PagerDuty, Google Chat, etc.). Incorporating data quality issues into your teams' existing tools and workflows is key to ensure streamlined and efficient operations.
Webhooks can also be leveraged for issue remediation purposes. By triggering custom scripts upon data quality issues detection, webhooks indeed allow you to decrease time to resolution by automating predefined actions (e.g. restart processes, reprocess data, or reroute workflows based on the nature of the issue).
Sifflet prides itself on it's user friendliness, however in some scenarios you may not be sure which template to use or how to configure certain parameters. Introducing AI Monitor Suggestions!
Simply describe the type of monitor you want to create and Sifflet will automatically suggest the template for you and preconfigure it based on your needs, leveraging all the context of your catalog and column descriptions!
When creating a new source in Sifflet, a default refresh frequency is now set based on the source type:
Data platforms and transactional databases: A daily refresh to ensure that new assets are visible in Sifflet within 24 hours of creation.
Data pipelines: An hourly refresh to keep statuses current in Sifflet, enabling prompt detection of pipeline failures.
BI tools: A weekly refresh to accommodate the slower rate of asset changes compared to other sources, reducing the likelihood of exceeding API quotas.
These default frequencies will apply to new sources, but you can still select a custom refresh frequency as needed. We also recommend updating existing sources to align with these frequencies, ensuring that metadata in Sifflet remains up-to-date.
You can now automatically create Jira issues in case of monitor failure: whenever a Sifflet incident gets created, you can decide to have a Jira issue created in the appropriate project and with the appropriate issue type. This issue will then automatically be linked to the failing monitor and corresponding incident in Sifflet.
This streamlines incident management by embedding data quality alerts directly into your existing ticketing workflows, saving valuable time and ensuring issues are never overlooked. By empowering your team with immediate visibility and prioritization of data anomalies, we’re helping you maintain data trust and operational efficiency at scale.
