Added pipeline parameters, referenced as ${params.X} in pipeline nodes and resolved at run time from job parameters (LAKEFLOW_JOB), Airflow DAG run conf, pl.execute(params=...) or laktory run --param, to reuse a single deployed pipeline across clients. Sink checkpoints are derived from the resolved sinks, so each client gets its own. [#694]
Added ServicePrincipalSecret (service principal OAuth secret, whose generated secret can be referenced as ${resources.<key>.secret}), GroupRole, Credential (Unity Catalog service credential), SecretUc, PermissionAssignment, CustomAppIntegration and SystemSchema resources. [#695]
Added nested ServicePrincipal.secrets, Group.roles and Credential.grant/grants (with the new CredentialGrant model) to declare service principal secrets, group roles and service credential grants with their parent resource. [#695]
Added TagPolicy, EntityTagAssignment, WorkspaceEntityTagAssignment, PolicyInfo (ABAC), ArtifactAllowlist, ShareProvider (Delta Sharing), File (Unity Catalog volume file) and ExternalMetadata resources, with nested tags on Catalog, Schema, Volume, Table, table columns, App, Dashboard and Notebook, nested ABAC policies on Catalog, Schema and Table, and nested files on Volume. [#702]
Added DataClassificationCatalogConfig, AccessRequestDestinations, AlertV2, AiSearchEndpoint, AiSearchIndex, AiGatewayModelService, AiGatewayModelProviderService and AiGatewayMcpService resources, with nested data_classification on Catalog, access_request_destinations on Catalog, Schema, Table and Volume, and indexes on AiSearchEndpoint. [#704]
Added Lakebase Autoscaling resources PostgresProject, PostgresBranch, PostgresEndpoint, PostgresDatabase, PostgresRole, PostgresCatalog, PostgresSyncedTable, PostgresCdfConfig, PostgresDataApi and PostgresSnapshotSchedule, with nested branches on projects, endpoints, roles, databases, catalogs, synced_tables and snapshot_schedule on branches, and data_api and cdf_configs on databases. PostgresCatalog also declares its Unity Catalog grant/grants and tags. [#707]
Added workspace and account configuration resources: WorkspaceSettingV2, AccountSettingV2, DefaultNamespaceSetting, RestrictWorkspaceAdminsSetting, ComplianceSecurityProfileWorkspaceSetting, EnhancedSecurityMonitoringWorkspaceSetting, DisableLegacyFeaturesSetting, WorkspaceConf, SqlGlobalConfig, AccountNetworkPolicy, WorkspaceNetworkOption (also nested as AccountNetworkPolicy.workspace_ids), WorkspaceBaseEnvironment and DefaultWorkspaceBaseEnvironment. [#703]
Fixed resources nested in a schema or catalog (e.g. volume grants) being listed twice in the parent's core_resources. [#702]
Fixed Metastore failing to build when grants are set without grants_provider: grants now use the default provider.
Fixed ModelServingtelemetry_config.inference_table_config.sampling_fraction typed as an integer, rejecting fractions between 0 and 1 (e.g. 0.1).
Fixed TableDataSinkMetadata.get_current() failing when a table property value contains a comma or = (e.g. delta.workloadBasedColumns.optimizerStatistics added by Predictive Optimization), by reading properties with SHOW TBLPROPERTIES. [#699]
Updated Databricks resources to Terraform provider 1.137.0: new fields on Job, Pipeline, App, Grant, Grants, Cluster, Connection, ModelServing, Secret and MwsNccPrivateEndpointRule, and some fields no longer required (e.g. Secret.string_value, now optional with string_value_wo). [#695]
The default Databricks Terraform provider version constraint is now >=1.137.0 (was >=1.49), the version the resource models are generated from. Stacks with a .terraform.lock.hcl locking an older provider must upgrade it once with laktory init --options -upgrade.
Added shared sinks, written by multiple nodes of a pipeline and/or by multiple pipelines. Each writer owns its rows and a full refresh of a writer only deletes and reprocesses its own rows, so writers run and refresh independently, from any selection of job tasks. Every sink writing to a shared table declares shared (shared: true for the default options). Rows are identified by a {pipeline}.{node} writer column or, without adding a column, by a SQL predicate (shared.where, e.g. client_id = 23). A node reading a writer (node_name) gets the writer rows only, whether read from memory or from the table. Delta commits of concurrent writers conflicting with each other are retried. With Lakeflow / Spark Declarative Pipelines, nodes writing to the same table use append flows. [#675], [#676], [#677]
Added a refresh run parameter (INCREMENTAL, FULL, RESET) - RESET resets tables without reprocessing data, e.g. from a job run without table grants - and reset_mode (DROP, TRUNCATE - table sinks and DELTA file sinks) to control how sinks are reset. reset_mode is set per sink and can be overridden for a run (pl.execute(refresh=..., reset_mode=...), node.execute(...), refresh / reset_mode job parameters), an override falling back to DROP on sinks that can't be truncated. [#669]
Added --refresh, --reset-mode and --tasks options to laktory run, to run jobs with a full refresh or a reset, and on a selection of tasks.
Fixed intermittent The parent folder ... does not exist errors on the first deploy of workspace files and notebooks (including WorkspaceTree files) into new workspace folders: files of a new folder now wait for a first file to create it, and new folders are created one after the other. [#686]
Fixed full_refresh silently skipping the deletion of sink and expectations checkpoints stored in Unity Catalog Volumes when they are not visible on the local file system (e.g. serverless compute), leaving stale streaming state: Auto Loader ingesting zero rows or DIFFERENT_DELTA_TABLE_READ_BY_STREAMING_SOURCE failures. Volumes paths now fall back to the Files API. [#684]
Fixed the AIRFLOW orchestrator running a full refresh when the full_refresh DAG param was passed as the string "false" (replaced by the validated refresh / reset_mode params).
Fixed DataSinkMergeCDCOptions._parent going stale after Pipeline.inject_vars(inplace=False)'s deep copy (same root cause as #653, different mechanism: it was wired up via a one-off manual assignment instead of the standard children_names recursion), corrupting merge_cdc_options.target_name/.target_path/.sink for any sink using mode: MERGE. [#656]
DataFrame namespaces to support multi-level names (accessor chains deeper than one dot e.g. namespace.sub.method). [#672]
Local Spark Declarative Pipeline runs (execute()) now load Delta from the Laktory Spark session, so sinks default to Delta as on Databricks (previously written as Parquet locally). [#679]
Replaced full_refresh by refresh in Pipeline.execute(), PipelineNode.execute(), LAKEFLOW_JOB job parameters, AIRFLOW DAG params and laktory run (--refresh FULL instead of --full-refresh): use refresh="FULL" instead of full_refresh=True. Runs still passing full_refresh=true fail: redeploy jobs deployed before 0.13.0 and update their triggers. See Refresh and Reset.
Nodes of a pipeline writing to the same table must declare shared on every sink writing to it (e.g. shared: true), otherwise the pipeline fails validation (except with Lakeflow / Spark Declarative Pipelines). Each node then owns its rows (_laktory_writer column): a full refresh of a node only deletes its own rows instead of the whole table. These sinks must be DELTA sinks in APPEND mode, and existing tables must be dropped once after upgrading (refresh=RESET, reset_mode=DROP). See Shared Sinks. [#675]
Pipelines of a Stack or of a Databricks Asset Bundle (laktory.dab.build_resources) writing to the same table are validated together: a table written by a declarative pipeline can't be written by other pipelines, and sinks declaring shared must identify their rows consistently (validation errors). Sinks writing to such a table without declaring shared log a warning: each pipeline should declare shared (e.g. shared: true) to own its rows. See Shared Sinks. [#675]
Added ServicePrincipalFederationPolicy model / databricks_serviceprincipalfederationpolicies stack resource, wrapping databricks_service_principal_federation_policy to declaratively manage OIDC workload-identity federation policies for service principals. Also addable inline via ServicePrincipal.federation_policies, mirroring the existing roles/group_ids/workspace_permission_assignments pattern. [#664]
Added AccountFederationPolicy, GitCredential, GlobalInitScript, InstanceProfile, MwsCredentials, MwsStorageConfigurations, MwsNetworks, MwsCustomerManagedKeys, MwsPrivateAccessSettings, MwsVpcEndpoint, MwsLogDelivery, MwsNccPrivateEndpointRule and MwsWorkspaces models / stack resources, closing the remaining gaps in everyday workspace-admin and AWS account-provisioning (mws_*) coverage. Also addable inline via MwsNetworkConnectivityConfig.private_endpoint_rules, mirroring the existing workspace_bindings pattern. [#665]
Fixed laktory.dab.build_resources computing workspace_root with OS-native (backslash) separators on Windows, breaking the deployed job's file path on remote Unix compute. [#639]
Fixed DataFrameExpr (SQL expr: pipeline nodes) splitting on a bare ;, causing a ; inside a -- comment to silently truncate/corrupt the query - expr: must now be a single SQL statement, enforced with a clear validation error instead of a confusing backend parse error. [#640]
Fixed laktory/AGENTS.md incorrectly claiming a Pipeline YAML doesn't need orchestrator.type when deploying via DABs; a pipeline with no orchestrator is actually skipped entirely (no resource created, in both DABs and Terraform/Pulumi), now logged as a warning instead of an easy-to-miss INFO log. [#643]
Fixed Pipeline.inject_vars(inplace=False) (the default, used by laktory.dab.build_resources) leaving grandchildren (e.g. a sink two levels deep) with a stale _parent pointing at the pre-copy tree, causing DataSink.checkpoint_path to silently use unresolved ${var.x}/${vars.x} values instead of the injected ones. [#653]
Fixed PipelineNode.execute() always writing the output DataFrame to every sink, including is_quarantine sinks, which silently duplicated the output data into quarantine tables instead of writing the failing rows. [#658]
Fixed a DataQualityExpectation inheriting a NATIVEdataframe_api from a parent node/global setting, which crashed with [NOT_ITERABLE] Column is not iterable since expectations are always checked against a narwhals-wrapped batch - the nested expression now always resolves to NARWHALS. [#659]
Fixed laktory.SQLParser raising "expression ... is not currently supported" for IS NULL, IS NOT NULL, IS TRUE, IS FALSE, and NOT expressions used in type: SQL expectations. [#660]
Fixed a streaming is_quarantine: true sink with no explicit mode: crashing with Mode 'None' is not supported for Spark Streaming DataFrame - it now defaults to APPEND, the only mode that is both valid (COMPLETE/UPDATE don't work for a non-aggregated streaming write to Delta) and correct (MERGE would key off columns quarantined rows typically violate) there. A staticis_quarantine sink still requires an explicit mode, since OVERWRITE vs APPEND depends on the node's own recompute/ingestion pattern. [#661]
Documented the ${var.x} / ${vars.x} alias in laktory/AGENTS.md (previously only on the docs site), and recommended ${var.x} for pipeline YAML in DAB-integrated projects to match DAB's own native bundle-variable syntax. [#642]
Documented that DAB does not expose a computed reference for an artifacts: block's deployed wheel path, and how to reference it via a bundle variable using ${dab_workspace_root}, in laktory/AGENTS.md and docs/concepts/dab.md. [#641]
Documented the DAB-context equivalent of settings.runtime_root (root_path on a pipeline, or the LAKTORY_RUNTIME_ROOT environment variable) for workspaces with public DBFS root access disabled, in laktory/AGENTS.md and docs/concepts/dab.md. [#644]
Documented the local-debugging pattern (Databricks Connect + pl.execute(write_sinks=False, selects=[...])) - including the write_sinks write-safety caveat, the as_stream inspection limitation, and the shared-checkpoint-with-production footgun - in docs/concepts/pipeline.md and laktory/AGENTS.md. [#645]
Documented that dataframe_api is ignored for expectations (always checked via Narwhals) in docs/concepts/dataquality.md and laktory/AGENTS.md. [#659]
Documented mode's implicit-default rules for is_quarantine sinks (defaults to APPEND when streaming, still required when static) in docs/concepts/dataquality.md and laktory/AGENTS.md, and fixed docs/concepts/dataquality.md's own example, which omitted mode: on its (non-streaming) sinks entirely. [#661]
${current_user.X} variable namespace (currently user_name) exposing the live Databricks identity resolved via the SDK, mirroring ${settings.X}. See Variables
settings.runtime_root guidance for DBFS-disabled Databricks workspaces (serverless compute, or workspaces with legacy DBFS features disabled): recommends a Unity Catalog Volume path (/Volumes/{catalog}/{schema}/{volume}/{path}) instead.
inject_vars() no longer crashes with a Pydantic frozen_field error when a frozen field (e.g. a data source's type: literal, such as type: CUSTOM on CustomDataSource) is explicitly set in YAML/code [#628]
depends_on on a LAKEFLOW_JOB pipeline's auto-generated config-file Permissions resource pointing at a nonexistent resource when settings.workspace_root: "user_root" is used, causing terraform plan to fail [#629]
A LAKEFLOW_JOB or LAKEFLOW_DECLARATIVE_PIPELINE pipeline's own job/config definition (task filepath, dependency/library paths, config_filepath) baked in the raw, unresolved settings.workspace_root instead of the "user_root"-resolved value, causing the deployed job to fail with FileNotFoundError / library-install errors [#630]
Checkpoint purge (DataSink and PipelineNode expectations) no longer falls through to a DBFS API probe after already deleting the checkpoint via the plain filesystem - previously this happened unconditionally, which could error out on a DBFS-disabled workspace even though the checkpoint was already correctly removed (e.g. via a Unity Catalog Volume runtime_root)
custom_writer.parent is now correctly assigned on table-based sinks (UnityCatalogDataSink, HiveMetastoreDataSink) - TableDataSink.children_names previously omitted "custom_writer", so laktory_context.sink was always None inside custom writer functions targeting those sinks [#636]
${settings.x} / ${setting.x} variable syntax to reuse a laktory.settings value (e.g. settings.workspace_root) anywhere else in a stack, in both plain substitution and ${{ expr }} contexts - avoids duplicating the same literal in the settings: block and again elsewhere [#618]
settings.workspace_root: "user_root" sentinel value to auto-compute a Databricks user/stack/environment-scoped deployment root (/Users/{you}/.laktory/{stack}/{env}/). Independent of, but designed to nest cleanly with, terraform.backend.databricks_workspace: true - using both resolves the username via a single shared Databricks SDK lookup. See the new Workspace Root docs page [#618]
${{ <expression> }} evaluation now recursively resolves variables that are themselves ${vars.x} indirections, matching the resolution already applied to plain ${vars.x} substitutions - previously the expression evaluator silently saw the raw, unresolved template string instead of its final value [#615]
WorkspaceTree with an explicit path no longer silently drops that path prefix on Windows for files nested in a subdirectory of source. The dirpath used to build each file's target path is now computed via relative_to(...).as_posix() instead of stringifying filepath.parent (which used the OS-native separator and was then wrongly treated as an absolute anchor when joined with path). Notebook.dirpath / WorkspaceFile.dirpath also now normalize a backslash-prefixed value defensively [#616]
settings: block values (workspace_root, build_root, etc.) using ${vars.x} now resolve correctly and deterministically once Stack.inject_vars() runs (build/preview/deploy/destroy/validate), including per-environment variable overrides - previously this relied on an incidental Pydantic side effect that wasn't guaranteed to fire in the right order relative to other fields. A settings: field containing an unresolved ${resources...} reference (which can never resolve here - it's Terraform-native syntax) now raises a clear error instead of silently deploying a broken literal [#617]
The Terraform state path auto-configured by backend.databricks_workspace: true now uses the resolved stack name (after variable injection) instead of the raw, potentially ${vars.x}-templated Stack.name - a stack combining a templated name: with this backend previously had all its environments collide on one unresolved, broken state path. Note: for that narrow combination only, this changes the computed state file location on upgrade; existing state may need a one-time move in the Databricks workspace
TableDataSinkMetadata.set_tags() now uses ALTER TABLE ... SET TAGS (...) / UNSET TAGS (...) (supported since DBR 13.3 LTS) instead of SET TAG ON ... / UNSET TAG ON ... (only recognized on DBR 16.1+), fixing INVALID_PROPERTY_KEY / INVALID_SET_SYNTAX errors when setting table/column tags on Unity Catalog sinks on older Databricks Runtime versions, where the latter DDL form isn't parsed as tag syntax and instead falls through to Spark's generic SET configuration-statement grammar [#621]
CurrentUser (databricks_current_user) now always deploys as a Terraform data block - it only exists as a data source in the databricks provider, so lookup_existing no longer needs to be set explicitly; omitting it previously generated an invalid resource block that only failed later at terraform plan [#619]
Opt-in variable (${vars.x} / ${{ expr }}) rendering of local file content, for environment-specific deployments (e.g. Databricks Apps, Lakeview dashboards): render_vars flag on Dashboard, WorkspaceFile, Notebook, and DbfsFile; render_paths (glob patterns) on WorkspaceTree for bulk opt-in. Resolved content is staged under settings.build_root, original files are never modified
exclude_paths (gitignore-syntax patterns, supports negation) and use_gitignore (auto-honor a source/.gitignore file) on WorkspaceTree, to exclude files/directories from deployment.
PipelineNodeDataSource batch->batch chains now read the upstream dataset (spark.read.table) inside the Lakeflow (LDP) runtime so Lakeflow can infer the dependency edge
Virtual resources (WorkspaceTree, Pipeline) now participate in depends_on in both directions: their own depends_on propagates to the child resources they generate, and a depends_on reference to them expands to all those children
WorkspaceTree now excludes files inside a dot-directory (e.g. .venv/lib/site.py), not just files whose own name starts with . - a negated pattern in exclude_paths (e.g. !.streamlit/) can re-include one when needed
mcp extra now requires mcp>=2 (updated MCP server to the MCPServer API)
File reads/writes now specify an explicit encoding instead of relying on the platform default, fixing UnicodeDecodeError on non-UTF-8 locales (e.g. cp1252 on Windows) when config/source files contain UTF-8 characters. Reads use utf-8-sig to tolerate a BOM; writes use utf-8
PipelineNode.sources - a named dict of data sources replacing the single source field. All existing source: YAML is automatically migrated to sources: {df: ...}.
null_equals_null option for cdc_merge_options [#586]
All laktory-injected Spark conf / pipeline configuration keys now use laktory. namespace prefix (laktory.executor, laktory.requirements, laktory.config_filepath, laktory.pipeline_name)
PipelineNode.execute() now reads all sources entries before invoking the transformer. Upstream nodes referenced via {nodes.X} SQL placeholders are pre-loaded here rather than inside DataFrameExpr
Dtype class now supports time_unit and time_zone properties.
Renamed Databricks Pipelines orchestrator to Lakehouse Declarative Pipeline
Refactored Lakehouse Declarative Pipeline script to use latest API (apply_changes -> create_auto_cdc_flow)
CLI --dbks-job / --dbks-pipeline flags replaced by --databricks-job / --databricks-pipeline
PipelineNode.source field removed - use PipelineNode.sources (dict) instead. YAML using source: is automatically migrated; Python code accessing .source directly must be updated.
Renamed data sink databricks_quality_monitor to databricks_data_profiling_config
Removed pipeline databricks_quality_monitors_enabled flag (now auto-detected)
Pipeline Lakeflow Job orchestrator data profiling configuration task is no longer active by default. Needs to set data_profiling_config_task to True
QualityMonitor resource renamed to DataQualityMonitor to be aligned with Databricks latest naming convention
Lakeflow Job data profiling task key renamed from post-execute to data-profiling-configs
TerraformStack.model_dump(): resource reference substitution (${resources.xxx}) now uses structured dict traversal with str.replace for simple refs and re.escape + lambda for property refs, fixing incorrect matches when resource names contain regex metacharacters (e.g. a dot)
Improved documentation review as per 2026/05/08 audit.
Documentation website
Improved exceptions reporting during validation of data sources and sinks
VariableType to only accept specific format instead of generic string.
All Databricks resource models are now derived from auto-generated files built directly from the Terraform provider schema, ensuring fields stay in sync with the provider and reducing manual maintenance
Support for TableDataSourcereader_kwargs attribute [#513]
CustomWriter class and custom_writer field on BaseDataSink - fully replaces Laktory's built-in write logic with a user-supplied function; Laktory manages the streaming query lifecycle (foreachBatch, trigger, checkpoint) [#514]
CustomDataSource source type and CustomReader class - fully user-supplied read function as a first-class source alongside FileDataSource, UnityCatalogDataSource, etc.
LaktoryContext dataclass - runtime context object (node, pipeline, sink, source) optionally injected into user-supplied functions in CustomWriter, CustomReader, and DataFrameMethod by declaring laktory_context=None in the function signature [#515]
More robust detection of file existence on DBFS and prevention of "com.databricks.sql.io.CloudFileNotFoundException" exception when purging checkpoints.