Configuring Monitors and Assets at Scale

As your data landscape grows, managing and configuring data assets and their associated monitors efficiently becomes crucial. Performing these operations one by one can be time-consuming and error-prone. Sifflet provides several ways to manage your assets and monitors in bulk, ensuring consistency and saving you time.

Why Configure at Scale?

Large-scale configuration is useful in situations such as:

• Organizing Your Data Catalog:
  • Tagging all assets belonging to a specific data domain (e.g., "Finance", "Marketing", "Product") with domain-specific tags.
  • Assigning an owner or a team to all assets within a particular schema or database.
  • Categorizing assets based on sensitivity levels or compliance requirements via tags or custom metadata.
• Managing Data Quality Monitors:
  • Setting a default notification channel (e.g., Slack, email, PagerDuty) for all critical monitors associated with production assets.
  • Updating the alerting policy (whether to create incidents, for example) for a group of monitors after a change in operational procedures.
  • Configuring the schedule of monitors created in bulk using Automatic Monitoring.
• Enriching Metadata:
  • Adding common descriptions or business context to a large number of related assets.
  • Bulk-updating custom properties for assets or monitors that share specific characteristics.

Performing these updates manually across hundreds or even thousands of items is impractical. Sifflet lets you handle these tasks efficiently.

Methods for At-Scale Configuration in Sifflet

Sifflet offers multiple avenues for configuring your assets and monitors at scale, catering to different preferences and technical expertise:

• Sifflet API: For programmatic control and integration into your existing scripts or DataOps workflows, the Sifflet API allows you to automate a wide range of configuration tasks.
• Data-Quality-as-Code (DQaC): Define and manage your data quality rules and monitor configurations as code using Sifflet's YAML syntax (that has features like For Loops and Wildcards), enabling version control, collaboration, and automated deployment at scale.
• Bulk Actions in the Sifflet UI: Directly within the Sifflet platform, the bulk actions feature in both the Data Catalog (for assets) and the Monitors page provides a way to select and configure hundreds of items simultaneously.

This page focuses on using bulk actions within the Sifflet UI.

Using Bulk Actions in the UI

Use bulk actions to manage many assets or monitors directly from the Sifflet user interface. It allows you to select items across multiple pages and apply common configurations to all selected items at once.

Key Features of Bulk Actions:

• Cross-Page Selection: Sifflet can display up to 100 assets or monitors per page. With bulk actions, you can select items on one page, navigate to subsequent pages, select more items, and Sifflet remembers your entire selection.
  • For example, you can select 100 assets on the first page, go to the second page and select 100 more assets, and then apply a configuration (like adding a tag or an owner) to all 200 assets in a single operation.
• Streamlined Management: This capability reduces the clicks and manual effort required, making it easy to enrich thousands of assets with metadata or update numerous monitors efficiently.
• Consistency: Applying configurations in bulk helps ensure uniformity across your selected assets or monitors.

Configuring Assets at Scale via the Data Catalog

The Data Catalog in Sifflet allows you to discover, understand, and manage all your data assets. Using bulk actions here, you can easily update metadata for multiple assets at once.

Common bulk actions for assets include:

• Adding/Removing Tags
• Adding/Removing Business Terms
• Assigning/Unassigning Owners
• Updating Custom Properties
• Adding/Modifying Descriptions

Configuring Monitors at Scale via the Monitors Page

Similarly, the Monitors page provides an overview of all your data quality monitors. Bulk actions here simplify the management of your monitoring setup.

Common bulk actions for monitors include:

• Enabling/Disabling Monitors
• Updating Time Settings
• Setting Notification Preferences (Channels, Recipients)
• Assigning Severity Levels
• Adding/Removing Tags for monitor organization

Watch How to Use Bulk Actions

By using these at-scale configuration capabilities, you can keep your assets and monitors organized as your data ecosystem grows.