QualityMonitor
laktory.models.resources.databricks.QualityMonitor
¤
Bases: QualityMonitorBase
Databricks Quality Monitor
Examples:
import io
from laktory import models
qm_yaml = '''
assets_dir: /.laktory/qualitymonitors
output_schema_name: dev.monitoring
table_name: dev.finance.slv_stock_prices
snapshot: {}
'''
qm = models.resources.databricks.QualityMonitor.model_validate_yaml(
io.StringIO(qm_yaml)
)
References
| BASE | DESCRIPTION |
|---|---|
assets_dir
|
TYPE:
|
baseline_table_name
|
Name of the baseline table from which drift metrics are computed from.Columns in the monitored table should also be present in the baseline table
TYPE:
|
custom_metrics
|
Custom metrics to compute on the monitored table. These can be aggregate metrics, derived metrics (from already computed aggregate metrics), or drift metrics (comparing metrics across time windows)
TYPE:
|
latest_monitor_failure_msg
|
TYPE:
|
skip_builtin_dashboard
|
Whether to skip creating a default dashboard summarizing data quality metrics. (Can't be updated after creation)
TYPE:
|
slicing_exprs
|
List of column expressions to slice data with for targeted analysis. The data is grouped by each expression independently, resulting in a separate slice for each predicate and its complements. For high-cardinality columns, only the top 100 unique values by frequency will generate slices
TYPE:
|
warehouse_id
|
Optional argument to specify the warehouse for dashboard creation. If not specified, the first running warehouse will be used. (Can't be updated after creation)
TYPE:
|
| LAKTORY | DESCRIPTION |
|---|---|
data_classification_config
|
The data classification config for the monitor
TYPE:
|
inference_log
|
Configuration for the inference log monitor
TYPE:
|
monitor_id
|
ID of this monitor is the same as the full table name of the format {catalog}.{schema_name}.{table_name}
TYPE:
|
notifications
|
The notification settings for the monitor.
TYPE:
|
output_schema_name_
|
Schema where output metric tables are created. Its of the format {catalog}.{schema}.
TYPE:
|
schedule
|
The schedule for automatically updating and refreshing metric tables.
TYPE:
|
snapshot
|
Configuration for monitoring snapshot tables.
TYPE:
|
table_name_
|
The full name of the table to attach the monitor too. Its of the format {catalog}.{schema}.{tableName}
TYPE:
|
time_series
|
Configuration for monitoring timeseries tables.
TYPE:
|
| ATTRIBUTE | DESCRIPTION |
|---|---|
additional_core_resources |
TYPE:
|
table_name |
Remove backticks from table name as they are not accepted by the API
TYPE:
|
laktory.models.resources.databricks.qualitymonitor.QualityMonitorCustomMetrics
¤
Bases: BaseModel
| PARAMETER | DESCRIPTION |
|---|---|
definition
|
TYPE:
|
input_columns
|
Columns on the monitored table to apply the custom metrics to
TYPE:
|
name
|
Name of the custom metric
TYPE:
|
output_data_type
|
The output type of the custom metric
TYPE:
|
type
|
The type of the custom metric
TYPE:
|
laktory.models.resources.databricks.qualitymonitor.QualityMonitorDataClassificationConfig
¤
laktory.models.resources.databricks.qualitymonitor.QualityMonitorInferenceLog
¤
Bases: BaseModel
| PARAMETER | DESCRIPTION |
|---|---|
granularities
|
List of granularities to use when aggregating data into time windows based on their timestamp.
TYPE:
|
label_col
|
Column of the model label
TYPE:
|
model_id_col
|
Column of the model id or version
TYPE:
|
prediction_col
|
Column of the model prediction
TYPE:
|
prediction_proba_col
|
Column of the model prediction probabilities
TYPE:
|
problem_type
|
Problem type the model aims to solve. Either PROBLEM_TYPE_CLASSIFICATION or PROBLEM_TYPE_REGRESSION
TYPE:
|
timestamp_col
|
Column of the timestamp of predictions
TYPE:
|
laktory.models.resources.databricks.qualitymonitor.QualityMonitorNotifications
¤
Bases: BaseModel
| PARAMETER | DESCRIPTION |
|---|---|
on_failure
|
Who to send notifications to on monitor failure.
TYPE:
|
on_new_classification_tag_detected
|
Who to send notifications to when new data classification tags are detected.
TYPE:
|
laktory.models.resources.databricks.qualitymonitor.QualityMonitorNotificationsOnFailure
¤
Bases: BaseModel
| PARAMETER | DESCRIPTION |
|---|---|
email_addresses
|
TYPE:
|
laktory.models.resources.databricks.qualitymonitor.QualityMonitorNotificationsOnNewClassificationTagDetected
¤
Bases: BaseModel
| PARAMETER | DESCRIPTION |
|---|---|
email_addresses
|
TYPE:
|
laktory.models.resources.databricks.qualitymonitor.QualityMonitorSchedule
¤
Bases: BaseModel
| PARAMETER | DESCRIPTION |
|---|---|
quartz_cron_expression
|
string expression that determines when to run the monitor. See Quartz documentation for examples.
TYPE:
|
timezone_id
|
string with timezone id (e.g., PST) in which to evaluate the Quartz expression.
TYPE:
|
laktory.models.resources.databricks.qualitymonitor.QualityMonitorTimeSeries
¤
Bases: BaseModel
| PARAMETER | DESCRIPTION |
|---|---|
granularities
|
List of granularities to use when aggregating data into time windows based on their timestamp.
TYPE:
|
timestamp_col
|
Column of the timestamp of predictions.
TYPE:
|