A common requirement of Sifflet Customers is to ensure data consistency by ensuring relationships between tables are correct. This was previously possible in a limited way using conditional monitors to ensure for example that the customers purchasing items in my orders tables were real customers that existed in the Customer table.
The new Template can be found in the field profiling section when two tables have been selected in the monitor wizard.
Additional Features available in the template.
Incremental Time window: Only check new values in the table and save costs.
Reverse integrity: Ensure that all items in the right table have a value on the left side. i.e. Check that every active store has a line item in the orders table.
Parity checking: Ensure that both tables have exactly the same list of values.
Create Incidents Manually from monitors
Monitors in sifflet have been historically automatically created by having the "Create Incident on Failure" Setting turned on. We are now introducing the first of many upcoming incident improvements: The ability to create incidents manually and from multiple monitors.
From the monitor list page, Sifflet Users are now able to select multiple monitors to create a new incident from. This also enable a brand new capability: Multiple Monitors per incident, allowing the creation of incidents as an escalation and response to multiple monitors failing.
Monitors who are already linked to an active incident will be swapped and linked to the newly created one!
What's next:
Very Soon! Merge incidents together, Link Monitors to existing incidents
Later: Automatically group/suggest grouping of monitors into the same incident if related!
There are many scenarios where certain days are going to be anomalies no matter what, and we want to avoid sifting through a sea of alerts to classify them as normal ! We've recently introduced a new functionality to exclude specific days from alerting and anomaly detection.
When creating monitors with a time window setting you can now specify calendars for which you want to exclude dates from anomaly detection. The Calendar list comes preloaded with public holidays calendars for multiple countries as well as other specific calendars to ignore Sundays or Weekends, while still ensuring the graph stays complete for these dates !
When a calendar is active on the monitor points that fall out of the range will be displayed as grey and won't trigger an alert, they will also indicate that the date is being excluded because of a specific calendar !
Custom Calendars
Sometimes standard calendars don't suit everyone's needs! You can also create your own custom calendars via the API
Programmatically Create and Maintain Sifflet Sources
You can now programmatically create and maintain your Sifflet sources thanks to our new sets of API endpoints for credentials and sources. These new endpoints make it easier for your teams to ensure the appropriate data observability coverage by simplifying the roll out of Sifflet on your data stack.
Support for empty and whitespace strings in Null monitors
For textual fields, you can now select what values to check in Null monitors. This capability is provided both for monitors set up in the UI or in Monitors as Code (via the new nullValues parameter).
Sifflet now supports the usage of dynamic parameters in Power BI source expressions, allowing the generation of accurate lineage even for dynamic data sources.
Amazon QuickSight and Fivetran URIs
Sifflet now supports URIs for Amazon QuickSight and Fivetran. You can now consequently tie declared assets to existing Amazon QuickSight and Fivetran assets, allowing you to go one step further in comprehensive data observability with end-to-end lineage and data cataloging.
Managing Incidents is a big focus area for Sifflet at the moment as we know a lot of our customers want to manage incidents at scale! The new and improved Incident Search page allows you to now batch edit the Severity, Status and Assignments of incidents.
🔥 Improved Monitors as Code: For Loops and Templates
Many improvements to Monitors as Code today !
Alternative to Monitor UUIDs when defining monitors: friendlyId
Before:
For each monitor a UUID needed to be specifiedid: 7edf1177-1a3c-4d71-b85f-e38b773735b4 which was often difficult since a UUID had to be generated outside of the yaml.
friendlyId alternative
friendlyId is an alternate ID that only needs to be unique per dataset, this means a dataset cannot have two monitors with the same friendlyId.
kind: Monitor
version: 1
friendlyId: myMonitor
name: My Monitor
extends:
- missionCriticalTags
...
---
kind: Monitor
version: 1
friendlyId: myOtherMonitor
name: My Other Monitor
extends:
- Unimportant
...
For Loops
Before Once again before each monitor had to be defined individually! However there are scenarios where I want to deploy the same monitor on multiple tables !
Now For loops allow you to define monitors on multiple tables easily !
Access Reference Assets Details for Improved Data Observability Coverage
You can now click reference assets on the lineage to view details such as the asset name. This makes it simpler to identify where these assets are located (i.e. in which database, schema, etc.) and consequently create the corresponding Sifflet data sources, enhance your overall data observability coverage.
We've introduced the capability to generate Monitor names and descriptions with AI. This will ensure you have highly descriptive descriptions and names for your monitors with low effort!
How? Simply select the AI icon in the text fields to write custom monitors. You will need to regenerate when monitors are now.
Tip: Not satisfied with the description, click it again and it will change!
Note: This generation is currently not aware of other descriptions and monitor names you have set on other monitors. Future versions aim to leverage these and make it even easier to standardise naming practices.
Need Help? Send Us a Message From Within the Sifflet App!
You can now reach out to the Sifflet team directly from within your app by clicking your user gravatar at the top right end corner of your application and hitting the Give us feedback entry in the menu. A form will show up letting you enter your name, email address, message to the Sifflet team, and even take a screenshot. You can use this to report bugs, share feedback about the application or even ask for help. The Sifflet team will follow up with you shortly after you submit your message!
Sifflet now supports key pair authentication for Snowflake, offering enhanced authentication security as an alternative to using a username and password.
To use key pair authentication, create the key pair by following the guide provided by Snowflake and then use the private key when adding the credentials to Sifflet as detailed in our documentation.
dbt build support
You can now use the dbt build command to generate the dbt artifacts to send to Sifflet, ensuring that you have full flexibility in configuring your dbt jobs. Refer to the dedicated documentation page for more details regarding our dbt Core integration.
🛠 Fixes
Fixed a behaviour where Monitors run with BigQuery repeated fields did not behave as expected
Fixed and Improved regexes generated via the AI suggestion feature. Fixed a case where the regex generated did not work correctly on Snowflake.
