Configuring Monitors and Assets at Scale 🚀

As your data landscape grows, managing and configuring data assets and their associated monitors efficiently becomes crucial. Performing these operations one by one can be time-consuming and error-prone. Sifflet provides several powerful ways to manage your assets and monitors in bulk, ensuring consistency and saving you valuable time.

Why Configure at Scale?

You might need to perform large-scale configurations in various situations. Here are a few common scenarios:

• Organizing Your Data Catalog:
  • Tagging all assets belonging to a specific data domain (e.g., "Finance", "Marketing", "Product") with domain-specific tags.
  • Assigning an owner or a team to all assets within a particular schema or database.
  • Categorizing assets based on sensitivity levels or compliance requirements via tags or custom metadata**.
• Managing Data Quality Monitors:
  • Setting a default notification channel (e.g., Slack, email, PagerDuty) for all critical monitors associated with production assets.
  • Updating the alerting policy (whether to create incidents, for example) for a group of monitors after a change in operational procedures.
  • Configuring the schedule of monitors created in bulk using Automatic Monitoring.
• Enriching Metadata:
  • Adding common descriptions or business context to a large number of related assets.
  • Bulk-updating custom properties for assets or monitors that share specific characteristics.

Performing these updates manually across hundreds or even thousands of items is impractical. Sifflet empowers you to handle these tasks efficiently.

Methods for At-Scale Configuration in Sifflet

Sifflet offers multiple avenues for configuring your assets and monitors at scale, catering to different preferences and technical expertise:

• Sifflet API: For programmatic control and integration into your existing scripts or DataOps workflows, our comprehensive API allows you to automate a wide range of configuration tasks.
• Data Quality as Code (DQaC): Define and manage your data quality rules and monitor configurations as code using Sifflet's powerful YAML syntax (that has features like For Loops and Wildcards), enabling version control, collaboration, and automated deployment, which is inherently scalable.
• Bulk Actions in the Sifflet UI: Directly within the Sifflet platform, the bulk actions feature in both the Data Catalog (for assets) and the Monitors page provides a user-friendly way to select and configure hundreds of items simultaneously.

This page focuses on leveraging the Bulk Actions capability within the Sifflet UI.

Using Bulk Actions in the UI ✨

The bulk actions feature is designed to streamline the management of numerous assets or monitors directly from the Sifflet user interface. It allows you to select items across multiple pages and apply common configurations to all selected items at once.

Key Features of Bulk Actions:

• Cross-Page Selection: Sifflet can display up to 100 assets or monitors per page. With bulk actions, you can select items on one page, navigate to subsequent pages, select more items, and Sifflet will remember your entire selection.
  • For example, you can select 100 assets on the first page, go to the second page and select 100 more assets, and then apply a configuration (like adding a tag or an owner) to all 200 assets in a single operation.
• Streamlined Management: This capability significantly reduces the clicks and manual effort required, making it easy to enrich thousands of assets with metadata or update numerous monitors efficiently.
• Consistency: Applying configurations in bulk helps ensure uniformity across your selected assets or monitors.

Configuring Assets at Scale via the Data Catalog

The Data Catalog in Sifflet allows you to discover, understand, and manage all your data assets. Using bulk actions here, you can easily update metadata for multiple assets at once.

Common bulk actions for assets include:

• Adding/Removing Tags
• Adding/Removing Business Terms
• Assigning/Unassigning Owners
• Updating Custom Properties
• Adding/Modifying Descriptions

Configuring Monitors at Scale via the Monitors Page

Similarly, the Monitors page provides an overview of all your data quality monitors. Bulk actions here simplify the management of your monitoring setup.

Common bulk actions for monitors include:

• Enabling/Disabling Monitors
• Updating Time Settings
• Setting Notification Preferences (Channels, Recipients)
• Assigning Severity Levels
• Adding/Removing Tags for monitor organization

Watch how to use bulk actions

By leveraging these at-scale configuration capabilities, you can ensure your data observability platform remains well-organized, accurately reflects your data landscape, and efficiently supports your data quality initiatives, even as your data ecosystem continues to expand.