We're excited to announce significant improvements to our Databricks integration, bringing deeper context and visibility to your data assets. You can now leverage Databricks tags directly within Sifflet and better trace dependencies with our new field-level lineage for Databricks.

What's New?

• Databricks Tags Now in Sifflet: We now automatically retrieve tags associated with your tables and columns in Databricks and display them in the Sifflet catalog. This enriches your data discovery and governance workflows, allowing you to use your existing Databricks tagging strategy to organize and find assets within Sifflet.

  

• Field-Level Lineage: Our data lineage graphs for Databricks now go deeper. We've added support for field-level (or column-level) lineage, allowing you to see exactly how individual fields are connected across your data pipeline. This granularity is crucial for precise impact analysis and root cause investigation.

  

Coming Soon

This is just the beginning. We're continuing to improve our Databricks integration and will be releasing more enhancements (like Databricks Workflows support) in the near future. Stay tuned!

How to Get Started

To enable these new features, you'll need to ensure your Databricks environment is configured to allow Sifflet access to system tables. For detailed instructions, please see our updated Databricks integration documentation.

App version: v501

We're excited to release the Sifflet MCP (Model Context Protocol) Server, a new way to interact with your Sifflet data observability platform directly from your development environment and other tools!

The Sifflet MCP Server empowers you to perform a variety of data observability operations. For example, you can:

• Understand Downstream Impact: Before changing a dbt model or table, quickly identify all dependent downstream assets and their owners to ensure smooth collaboration.
• Access Up-to-Date Metadata: Get the latest operational details for any table—how it's monitored in Sifflet, any active incidents, frequent users, and more—right when you need it.
• Bootstrap New Asset Monitoring: Easily discover and adapt existing Monitor-as-Code YAML configurations from similar assets to quickly set up robust monitoring for new tables or dbt models.

This server enables more integrated data observability operations, bringing crucial context from Sifflet closer to your workflows.

Explore, contribute, and get started on GitHub. We look forward to your feedback!

We're excited to announce the release of Custom Metadata, a powerful new way to add rich, contextual information to your assets, monitors, and other entities within Sifflet! This feature allows you to define and apply your own key-value pairs, tailoring Sifflet to your specific organizational needs and workflows.

Key New Functionalities:

• Define Custom Metadata Attributes:
  • Create custom metadata fields with a specific Key (e.g., "Data Owner," "Project ID," "Sensitivity Level").
  • Choose from new Value Types for your metadata:
    • User: Assign Sifflet users.
    • String: Add free-text values.
    • Label: Select from a predefined list of options.
  • Specify which Asset Types (Datasets, BI Assets, Pipelines, Declarative Assets, Monitors) each custom metadata attribute can apply to.
  • Configure if an attribute can be used for filtering and if it supports multiple values.
• Apply Metadata to Assets (individually or at scale): Add or update custom metadata values directly in the "Details" section of an asset's "Overview" tab, or at scale by leveraging the Bulk Actions in the Catalog or Monitors page.
• Filter by Custom Metadata: If configured, use your custom metadata attributes to filter assets in the Catalog and Monitors pages for improved discoverability.

Why it Matters:

• Deeper Context: Go beyond simple tags to add specific, meaningful information to your Sifflet entities.
• Improved Organization & Discoverability: Structure and find your assets more effectively using your own defined criteria.
• Enhanced Governance: Track ownership, sensitivity, compliance status, and other critical governance information directly on assets.
• Streamlined Workflows: Easily identify and manage assets related to specific projects, teams, or business units.
• Customized Monitoring at Scale (Coming Soon): Leverage custom metadata to define monitoring strategies at scale based on your own criteria. (e.g., automatically creating Freshness Monitors for all tables that are part of a given project.)

To learn more about setting up and using Custom Metadata, check out our documentation page. We look forward to your feedback as we continue to enhance this feature!

App version: v497

We’re excited to roll out Join support for data monitors!

You can now add joins directly in the monitor setup UI, making it easier than ever to build context-rich data quality checks that span across datasets.

✨ What You Can Do:

• Group by columns in a joined table
  Example: Monitor ORDER volume grouped by CUSTOMER_TYPE after joining with the CUSTOMERS table.

• Filter based on joined table values
  Example: Filter out users where CUSTOMERS.LAST_ORDERED_AT > CURRENT_DATE() - 30.

• Flexible join key selection
  Define join conditions using shared keys (e.g., CUSTOMER_ID) between datasets.

To get started, simply click + Add join in the monitor setup panel, pick your table, set a join key, and build more powerful monitors in minutes!

🛠️ Data Quality as Code: Define Joins Programmatically

Prefer infrastructure as code? You can now define dataset joins in your YAML monitor definitions.

🔧 Example:

datasets:
  - uri: snowflake://sifflet-internal/DEMO.TEST.AGG_CASE_WHEN
joins:
  - dataset:
      uri: snowflake://sifflet-internal/DEMO.TEST.ALLUSERS
    joinCondition:
      kind: Equality
      fieldPairs:
        - leftField:
            dataset:
              uri: snowflake://sifflet-internal/DEMO.TEST.AGG_CASE_WHEN
            name: FIRSTNAME
          rightField:
            dataset:
              uri: snowflake://sifflet-internal/DEMO.TEST.ALLUSERS
            name: FIRSTNAME

You can now define:

• Join conditions
• Join fields
• Datasets and datasources all in YAML — enabling version control and CI/CD for data quality.

🔮 Coming Soon: Custom Aggregations in the Wizard

We're working on something exciting! Soon, you'll be able to define custom aggregations directly in the UI.

Example use cases:

• SUM(conversion_rate * amount_in_original_currency)

Stay tuned for more powerful and flexible monitoring options that don’t require SQL!

For additional details on joining tables in monitors, you can refer to the dedicated documentation page. Want help getting started or converting your current monitors to use joins or YAML? Reach out to your customer engineer — we're here to help!

App version: v494

We’ve evolved the Distribution Change monitor to provide more meaningful insights. Instead of performing a statistical analysis on the entire distribution, it now analyzes the volume percentage for each category within the distribution.

✅ What’s New

Aligned Anomaly Detection: Detection now works just like our other monitors, ensuring consistency across your monitoring setup.

Seamless Migration: We've automatically migrated your existing monitors to the new type. You’ll still be able to view your historical graphs from the previous version.

Mode Mapping:

• Dynamic mode remains unchanged.
• Static mode now corresponds to a relative threshold mode, using your original threshold (in percentage points).

New Control Options:

• Fail if new value appears – alerts you when a previously unseen category is detected.
• Fail if value disappears – alerts you when an existing category is missing from the distribution.

Important note
Group-by monitoring is not supported in this new version. Group-by dimensions previously configured have been migrated to dataset fields analysis. We now analyze the concatenated values of dataset fields and dimensions to maintain coverage.

App version: v493

We've introduced a dedicated Users section within the domain management page, significantly simplifying access permissions management. This enhancement streamlines administration, especially for domains with numerous users.

With this new section, you can now:

• Get a clear overview of all users who have access to the domain, displaying their Username and assigned Role.
• Efficiently add multiple users to the domain in one operation, assigning their specific roles during the addition process.
• Update the domain roles of multiple users simultaneously.
• Quickly remove one or multiple users from the domain when access is no longer needed.

This provides a much-requested, centralized way to manage domain membership and permissions directly where you manage the domain itself. You can find this new section within the settings of any domain. (Learn more)

App version: v480

We've enhanced the "Test Connection" process for BigQuery and Snowflake. It now thoroughly checks permissions and provides precise error feedback, telling you exactly what's missing if a permission check fails. This improves initial setup and debugging of permission issues during source refreshes.

As a company founded and headquartered in France, supporting our users in French is part of our DNA. We're incredibly proud and excited to announce that the Sifflet user interface is now fully available in French!

Now you can navigate, monitor, investigate, and manage your data observability directly en français.

• How to Switch: Changing your language preference is easy:
  1. Click on your User Profile icon located in the top-right corner of the Sifflet application.
  2. In the dropdown menu, locate the "Select Language" / "Choix de la langue" selector.
  3. Choose between "English" and "Français". The interface will update immediately.

We hope this makes the Sifflet experience even more accessible and comfortable for our French-speaking users!

Streamline your workflows and quickly access your most common data views with the new Saved Filters feature! Instead of reapplying the same filters repeatedly, you can now easily save and reuse your filter configurations.

• How it Works:
  • Apply any combination of filters on a supported page (Catalog, Monitors, or Incidents).
  • Click the new "Save" button that appears once filters are applied.
  • Give your filter set a descriptive name.
• Reuse Instantly: Next time you visit the page, you can simply select your saved filter set from a list instead of manually configuring the filters again.
• Availability: This feature is now available on the Catalog, Monitors, and Incidents pages.
• Personalized (For Now): Saved filters are currently specific to your user account. We are exploring options to allow sharing saved filters with your team in a future update.

Save time and effort by creating personalized shortcuts to the data views you use most often!

App version: v473

We've significantly enhanced the Sifflet Data Sharing feature to provide you with more timely access to your monitoring and observability data.

• Faster Refreshes: By optimizing our internal processes using a Change Data Capture (CDC) pattern, shared data is now refreshed much more frequently. You can expect data updates every four hours, a significant improvement from the previous daily refresh cycle.
• snapshotted_at Column Behavior Change: Please note an important update regarding the snapshotted_at column in the shared tables. This column no longer reflects the timestamp of the overall refresh batch. Instead, it now accurately represents the timestamp of the last captured change for that specific row, providing more granular insight into when each record was last updated.

This update ensures you have more frequent access to the latest data available through the Data Sharing feature, reflecting changes closer to when they occur. For more details on our Data Sharing feature, review the dedicated documentation.