Product Release 2024-07-02

by Mahdi Karabiben

✨ Feature Highlights

Batch Actions on Incidents

Managing Incidents is a big focus area for Sifflet at the moment as we know a lot of our customers want to manage incidents at scale! The new and improved Incident Search page allows you to now batch edit the Severity, Status and Assignments of incidents.


🔥 Improved Monitors as Code: For Loops and Templates

Many improvements to Monitors as Code today !

Alternative to Monitor UUIDs when defining monitors: friendlyId

Before:

For each monitor a UUID needed to be specifiedid: 7edf1177-1a3c-4d71-b85f-e38b773735b4 which was often difficult since a UUID had to be generated outside of the yaml.

friendlyId alternative

friendlyId is an alternate ID that only needs to be unique per dataset, this means a dataset cannot have two monitors with the same friendlyId.

kind: Monitor
version: 1
friendlyId: customerEmailUnique
datasets:
- name: sales
  datasource:
    name: mySqlDatabase
...

The above monitor has a friendlyId customerEmailUnique , only one of those monitors can be added to the sales table in our mysqlDatabase.

Alternative method to reference Datasets: URIs

Before: Referencing Datasets required knowledge of information specific to Sifflet.

Sifflet ID of dataset:

datasets:
- id: 70217023-1a89-4c0b-9b6a-c85192c918b3

Sifflet dataset name with either the id or the name of the datasource

datasets:
- name: Prices
  datasource: 
    name: BigQuery Data warehouse ## OR
    id: ce3e9dd9-b007-42b0-b884-8c419f7f6daa

Now with URIs

URIs are a sifflet agnostic way to define dataset. Find out more about URIs

datasets:
- uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.TABLE

📘

URIs

You can retrieve URIs from the Catalog's Asset Page.

Multiple Monitors Per File

Before : One file per Monitor

Now : You can now separate monitors with the --- yaml separator and have multiple monitors on the same page

kind: Monitor
version: 1
friendlyId: customer_id_format_check
name: Matches Regex Monitor on CUSTOMER_ID
...

---

kind: Monitor
version: 1
friendlyId: customer_id_duplicate_check
name: Unicity Monitor on CUSTOMER_ID

Templates

Before Each monitor definition needed to contain all the information, this meant a lot of information had to be duplicated, such as tags or alerting.

Now Templates enable the definition of parameters that can be imported into new monitor definitions

Simply define a partial (or complete ) monitor definition with a templateName

templateName: missionCriticalTags
tags:
- name: Mission Critical
notifications:
- kind: Email
  name: [email protected]
- kind: Slack
  name: Alerts
  id: dd6f06ec-fab1-4a87-9544-b113f496d61d

---

templateName: Unimportant
tags:
  - name: Unimportant

This templateName can then be extended

kind: Monitor
version: 1
friendlyId: myMonitor
name: My Monitor
extends:
- missionCriticalTags
...

---

kind: Monitor
version: 1
friendlyId: myOtherMonitor
name: My Other Monitor
extends:
- Unimportant
...

For Loops

Before Once again before each monitor had to be defined individually! However there are scenarios where I want to deploy the same monitor on multiple tables !

Now For loops allow you to define monitors on multiple tables easily !

for each dataset T:
  datasets:
  - uri: snowflake://sifflet-enterprise/DEMO.SE_ENV.ORDERS
  - uri: snowflake://sifflet-enterprise/DEMO.SE_ENV.CUSTOMERS
  - uri: snowflake://sifflet-enterprise/DEMO.SE_ENV.STG_CUSTOMERS
  monitors:
  - kind: Monitor
    version: 1
    friendlyId: customer_id_format_check
    name: "[${T.name}]Format monitor on CUSTOMER_ID" 
    description: ""
    incident:
      severity: Low
      message: ""
    parameters:
      kind: FieldFormat
      field: CUSTOMER_ID
      format:
        kind: Regex
        regex: "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$"

  ...

Wildcards and excludes

use include with * wildcards in your uris to apply to many datasets at the same time! then use exclude to exclude specific datasets

for each dataset T:
  datasets:
    include:
    - uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.*
    - uri: bigquery:*.*.*ORDERS
    exclude:
    - uri: snowflake://xyz12345.eu-central-1/DATABASE.SCHEMA.MYTABLETOEXCLUDE

Dynamic Names

name: "[${T.name}]Format monitor on CUSTOMER_ID"

Use ${T.name} to reference the name of your table.

🛠 Fixes

  • Fixed an issue where Sifflet pages would require a manual refresh to operate as expected after a new release
  • Fixed an issue where in some instances, a recently edited user could not be saved
  • Fixed an issue where some domains would not show up in the permissions modal.

App version: v279-281

🛠 Fixes

  • Fixed an issue in Data Quality as Code and APIs where tag name validation was case sensitive despite tag names being case insensitive.
  • Improved the experience around the release of new versions of the Sifflet application to make it less disruptive for users.

App version: v278

✨ Feature Highlights

Access Reference Assets Details for Improved Data Observability Coverage

You can now click reference assets on the lineage to view details such as the asset name. This makes it simpler to identify where these assets are located (i.e. in which database, schema, etc.) and consequently create the corresponding Sifflet data sources, enhance your overall data observability coverage.

Read more reference assets

🛠 Fixes

  • Fixed a bug that caused lineage inconsistencies for dbt assets following certain data source runs.

App version: v275-277

✨ Feature Highlights

Describe Monitors with AI

We've introduced the capability to generate Monitor names and descriptions with AI. This will ensure you have highly descriptive descriptions and names for your monitors with low effort!

How? Simply select the AI icon in the text fields to write custom monitors. You will need to regenerate when monitors are now.

Tip: Not satisfied with the description, click it again and it will change!

Note: This generation is currently not aware of other descriptions and monitor names you have set on other monitors. Future versions aim to leverage these and make it even easier to standardise naming practices.


Need Help? Send Us a Message From Within the Sifflet App!

You can now reach out to the Sifflet team directly from within your app by clicking your user gravatar at the top right end corner of your application and hitting the Give us feedback entry in the menu. A form will show up letting you enter your name, email address, message to the Sifflet team, and even take a screenshot. You can use this to report bugs, share feedback about the application or even ask for help. The Sifflet team will follow up with you shortly after you submit your message!

App version: v274

✨ Feature Highlights

Snowflake key-pair authentication

Sifflet now supports key pair authentication for Snowflake, offering enhanced authentication security as an alternative to using a username and password.

To use key pair authentication, create the key pair by following the guide provided by Snowflake and then use the private key when adding the credentials to Sifflet as detailed in our documentation.

dbt build support

You can now use the dbt build command to generate the dbt artifacts to send to Sifflet, ensuring that you have full flexibility in configuring your dbt jobs. Refer to the dedicated documentation page for more details regarding our dbt Core integration.

🛠 Fixes

  • Fixed a behaviour where Monitors run with BigQuery repeated fields did not behave as expected
  • Fixed and Improved regexes generated via the AI suggestion feature. Fixed a case where the regex generated did not work correctly on Snowflake.

App version: v272-273

Product Release 2024-06-17

by Margot Lepizzera

✨ Feature Highlights

End-To-End Data Observability With Declarative Assets & Lineage

Sifflet already offers a large number of built-in integrations spanning your entire data pipelines’ stack. These integrations automatically collect metadata and lineage information and make it available in the Data Catalog.

However, there may be instances where you need to programmatically declare certain data pipeline assets. This is essential for achieving end-to-end data observability, especially for custom scripts, data applications, or assets from technologies not yet directly supported by Sifflet (e.g., Salesforce, SAP, Metabase, etc.).

The declarative framework now allows you to programmatically push any assets and lineage data to Sifflet. This enables you to document, govern, and visualize lineage across your entire data stack, regardless of asset type or technology. This new capability removes any limitations on the assets that can be integrated with Sifflet, paving the way for complete end-to-end data observability.

Read more about declarative assets and lineage

Changes on IDs and URIs for Cleaner Asset Pages

In the context of changes done for the declarative framework, you can now access assets' URI from the context menu located in the top right end corner of your asset pages.

Asset IDs that used to be in the Overview tab are now available in the same context menu, ensuring the asset page remains focused on the content that matters the most to users.

Finally, source IDs that also used to be in the Overview tab were entirely removed from asset pages and are now available on source pages.

Read more about asset pages

App version: v271

Product Release 2024-06-14

by Margot Lepizzera

Small Changes

  • In the process of adding batch actions and to avoid spam, we have opted to stop sending Incident un-assignment emails.

App version: v270

Product Release 2024-06-13

by Margot Lepizzera

✨ Feature Highlights

Fivetran integration

Sifflet now integrates with Fivetran, allowing you to gain unparalleled visibility into your data's end-to-end journey, from source to destination.

With this release, the Sifflet lineage graph no longer starts at the data platform level. Instead, you now have access to two types of nodes that represent your data’s journey before it even reaches the data platform:

  • Fivetran connector nodes: These clickable nodes represent your Fivetran connectors, allowing you to get all the useful information about a given connector with just one click. This information includes the connector’s source and destination, its current status, sync frequency, and the timestamp of the latest sync. Accompanied by the Sifflet metadata that you’re accustomed to (owners, description, and last metadata refresh), this panel gives you, at a glance, all the necessary information about a given connector.
  • Source asset nodes: For every Fivetran connector, Sifflet will create nodes for the source assets from which Fivetran extracts data, giving you complete visibility on your data’s journey. These nodes, which allow you to get a truly end-to-end lineage within Sifflet, represent one of the key differentiators of our Fivetran integration.
End-to-end lineage via the Fivetran integration

End-to-end lineage via the Fivetran integration

By integrating your Fivetran account with Sifflet, you'll also be able to swiftly detect connector issues (in the Sifflet data catalog and via the Fivetran connector nodes in the lineage) and easily get a bird’s-eye view of your Fivetran connectors, no matter where you encounter them in the lineage.

To get started with the integration, refer to the dedicated documentation page. For additional information about the exciting features of this integration, check out the official release announcement.

🛠 Fixes

  • Fixed a bug that occasionally prevented Tableau and Looker assets from being connected to their upstream nodes in the lineage.

App version: v269

Product Release 2024-06-11

by Margot Lepizzera

✨ Feature Highlights

Improved Experience on Incidents List Page

The incidents list page got revamped to improve the overall incident search user experience by making it consistent with the rest of the Sifflet application. By default, only Open and In progress incidents now show on the list, making it easier for you to focus on data quality issues that require immediate attention.

Read more about incidents

App version: v267

Product Release 2024-06-10

by Margot Lepizzera

✨ Feature Highlights

Improved Control of Automatic Incident Creation

There are some monitors for which you don't necessarily want to create a Data Incident. You can now control Sifflet's automatic creation of an incident via the bulk edit from the Monitor list page.

This feature works also with dbt monitors.


In addition to the bulk edition of monitors, we've also made this toggle available at the dbt datasource level.


🛠 Fixes

  • Fixed Technical Error in some instances of Duplicate and Null monitors with Group By
  • Fixed Field Duplicate and Completeness monitors reporting incorrect results for empty datasets

App version: v265