Added ServicePrincipalFederationPolicy model / databricks_serviceprincipalfederationpolicies stack resource, wrapping databricks_service_principal_federation_policy to declaratively manage OIDC workload-identity federation policies for service principals. Also addable inline via ServicePrincipal.federation_policies, mirroring the existing roles/group_ids/workspace_permission_assignments pattern. [#664]
Added AccountFederationPolicy, GitCredential, GlobalInitScript, InstanceProfile, MwsCredentials, MwsStorageConfigurations, MwsNetworks, MwsCustomerManagedKeys, MwsPrivateAccessSettings, MwsVpcEndpoint, MwsLogDelivery, MwsNccPrivateEndpointRule and MwsWorkspaces models / stack resources, closing the remaining gaps in everyday workspace-admin and AWS account-provisioning (mws_*) coverage. Also addable inline via MwsNetworkConnectivityConfig.private_endpoint_rules, mirroring the existing workspace_bindings pattern. [#665]
Fixed laktory.dab.build_resources computing workspace_root with OS-native (backslash) separators on Windows, breaking the deployed job's file path on remote Unix compute. [#639]
Fixed DataFrameExpr (SQL expr: pipeline nodes) splitting on a bare ;, causing a ; inside a -- comment to silently truncate/corrupt the query - expr: must now be a single SQL statement, enforced with a clear validation error instead of a confusing backend parse error. [#640]
Fixed laktory/AGENTS.md incorrectly claiming a Pipeline YAML doesn't need orchestrator.type when deploying via DABs; a pipeline with no orchestrator is actually skipped entirely (no resource created, in both DABs and Terraform/Pulumi), now logged as a warning instead of an easy-to-miss INFO log. [#643]
Fixed Pipeline.inject_vars(inplace=False) (the default, used by laktory.dab.build_resources) leaving grandchildren (e.g. a sink two levels deep) with a stale _parent pointing at the pre-copy tree, causing DataSink.checkpoint_path to silently use unresolved ${var.x}/${vars.x} values instead of the injected ones. [#653]
Fixed PipelineNode.execute() always writing the output DataFrame to every sink, including is_quarantine sinks, which silently duplicated the output data into quarantine tables instead of writing the failing rows. [#658]
Fixed a DataQualityExpectation inheriting a NATIVEdataframe_api from a parent node/global setting, which crashed with [NOT_ITERABLE] Column is not iterable since expectations are always checked against a narwhals-wrapped batch - the nested expression now always resolves to NARWHALS. [#659]
Fixed laktory.SQLParser raising "expression ... is not currently supported" for IS NULL, IS NOT NULL, IS TRUE, IS FALSE, and NOT expressions used in type: SQL expectations. [#660]
Fixed a streaming is_quarantine: true sink with no explicit mode: crashing with Mode 'None' is not supported for Spark Streaming DataFrame - it now defaults to APPEND, the only mode that is both valid (COMPLETE/UPDATE don't work for a non-aggregated streaming write to Delta) and correct (MERGE would key off columns quarantined rows typically violate) there. A staticis_quarantine sink still requires an explicit mode, since OVERWRITE vs APPEND depends on the node's own recompute/ingestion pattern. [#661]
Documented the ${var.x} / ${vars.x} alias in laktory/AGENTS.md (previously only on the docs site), and recommended ${var.x} for pipeline YAML in DAB-integrated projects to match DAB's own native bundle-variable syntax. [#642]
Documented that DAB does not expose a computed reference for an artifacts: block's deployed wheel path, and how to reference it via a bundle variable using ${dab_workspace_root}, in laktory/AGENTS.md and docs/concepts/dab.md. [#641]
Documented the DAB-context equivalent of settings.runtime_root (root_path on a pipeline, or the LAKTORY_RUNTIME_ROOT environment variable) for workspaces with public DBFS root access disabled, in laktory/AGENTS.md and docs/concepts/dab.md. [#644]
Documented the local-debugging pattern (Databricks Connect + pl.execute(write_sinks=False, selects=[...])) - including the write_sinks write-safety caveat, the as_stream inspection limitation, and the shared-checkpoint-with-production footgun - in docs/concepts/pipeline.md and laktory/AGENTS.md. [#645]
Documented that dataframe_api is ignored for expectations (always checked via Narwhals) in docs/concepts/dataquality.md and laktory/AGENTS.md. [#659]
Documented mode's implicit-default rules for is_quarantine sinks (defaults to APPEND when streaming, still required when static) in docs/concepts/dataquality.md and laktory/AGENTS.md, and fixed docs/concepts/dataquality.md's own example, which omitted mode: on its (non-streaming) sinks entirely. [#661]
${current_user.X} variable namespace (currently user_name) exposing the live Databricks identity resolved via the SDK, mirroring ${settings.X}. See Variables
settings.runtime_root guidance for DBFS-disabled Databricks workspaces (serverless compute, or workspaces with legacy DBFS features disabled): recommends a Unity Catalog Volume path (/Volumes/{catalog}/{schema}/{volume}/{path}) instead.
inject_vars() no longer crashes with a Pydantic frozen_field error when a frozen field (e.g. a data source's type: literal, such as type: CUSTOM on CustomDataSource) is explicitly set in YAML/code [#628]
depends_on on a LAKEFLOW_JOB pipeline's auto-generated config-file Permissions resource pointing at a nonexistent resource when settings.workspace_root: "user_root" is used, causing terraform plan to fail [#629]
A LAKEFLOW_JOB or LAKEFLOW_DECLARATIVE_PIPELINE pipeline's own job/config definition (task filepath, dependency/library paths, config_filepath) baked in the raw, unresolved settings.workspace_root instead of the "user_root"-resolved value, causing the deployed job to fail with FileNotFoundError / library-install errors [#630]
Checkpoint purge (DataSink and PipelineNode expectations) no longer falls through to a DBFS API probe after already deleting the checkpoint via the plain filesystem - previously this happened unconditionally, which could error out on a DBFS-disabled workspace even though the checkpoint was already correctly removed (e.g. via a Unity Catalog Volume runtime_root)
custom_writer.parent is now correctly assigned on table-based sinks (UnityCatalogDataSink, HiveMetastoreDataSink) - TableDataSink.children_names previously omitted "custom_writer", so laktory_context.sink was always None inside custom writer functions targeting those sinks [#636]
${settings.x} / ${setting.x} variable syntax to reuse a laktory.settings value (e.g. settings.workspace_root) anywhere else in a stack, in both plain substitution and ${{ expr }} contexts - avoids duplicating the same literal in the settings: block and again elsewhere [#618]
settings.workspace_root: "user_root" sentinel value to auto-compute a Databricks user/stack/environment-scoped deployment root (/Users/{you}/.laktory/{stack}/{env}/). Independent of, but designed to nest cleanly with, terraform.backend.databricks_workspace: true - using both resolves the username via a single shared Databricks SDK lookup. See the new Workspace Root docs page [#618]
${{ <expression> }} evaluation now recursively resolves variables that are themselves ${vars.x} indirections, matching the resolution already applied to plain ${vars.x} substitutions - previously the expression evaluator silently saw the raw, unresolved template string instead of its final value [#615]
WorkspaceTree with an explicit path no longer silently drops that path prefix on Windows for files nested in a subdirectory of source. The dirpath used to build each file's target path is now computed via relative_to(...).as_posix() instead of stringifying filepath.parent (which used the OS-native separator and was then wrongly treated as an absolute anchor when joined with path). Notebook.dirpath / WorkspaceFile.dirpath also now normalize a backslash-prefixed value defensively [#616]
settings: block values (workspace_root, build_root, etc.) using ${vars.x} now resolve correctly and deterministically once Stack.inject_vars() runs (build/preview/deploy/destroy/validate), including per-environment variable overrides - previously this relied on an incidental Pydantic side effect that wasn't guaranteed to fire in the right order relative to other fields. A settings: field containing an unresolved ${resources...} reference (which can never resolve here - it's Terraform-native syntax) now raises a clear error instead of silently deploying a broken literal [#617]
The Terraform state path auto-configured by backend.databricks_workspace: true now uses the resolved stack name (after variable injection) instead of the raw, potentially ${vars.x}-templated Stack.name - a stack combining a templated name: with this backend previously had all its environments collide on one unresolved, broken state path. Note: for that narrow combination only, this changes the computed state file location on upgrade; existing state may need a one-time move in the Databricks workspace
TableDataSinkMetadata.set_tags() now uses ALTER TABLE ... SET TAGS (...) / UNSET TAGS (...) (supported since DBR 13.3 LTS) instead of SET TAG ON ... / UNSET TAG ON ... (only recognized on DBR 16.1+), fixing INVALID_PROPERTY_KEY / INVALID_SET_SYNTAX errors when setting table/column tags on Unity Catalog sinks on older Databricks Runtime versions, where the latter DDL form isn't parsed as tag syntax and instead falls through to Spark's generic SET configuration-statement grammar [#621]
CurrentUser (databricks_current_user) now always deploys as a Terraform data block - it only exists as a data source in the databricks provider, so lookup_existing no longer needs to be set explicitly; omitting it previously generated an invalid resource block that only failed later at terraform plan [#619]
Opt-in variable (${vars.x} / ${{ expr }}) rendering of local file content, for environment-specific deployments (e.g. Databricks Apps, Lakeview dashboards): render_vars flag on Dashboard, WorkspaceFile, Notebook, and DbfsFile; render_paths (glob patterns) on WorkspaceTree for bulk opt-in. Resolved content is staged under settings.build_root, original files are never modified
exclude_paths (gitignore-syntax patterns, supports negation) and use_gitignore (auto-honor a source/.gitignore file) on WorkspaceTree, to exclude files/directories from deployment.
PipelineNodeDataSource batch->batch chains now read the upstream dataset (spark.read.table) inside the Lakeflow (LDP) runtime so Lakeflow can infer the dependency edge
Virtual resources (WorkspaceTree, Pipeline) now participate in depends_on in both directions: their own depends_on propagates to the child resources they generate, and a depends_on reference to them expands to all those children
WorkspaceTree now excludes files inside a dot-directory (e.g. .venv/lib/site.py), not just files whose own name starts with . - a negated pattern in exclude_paths (e.g. !.streamlit/) can re-include one when needed
mcp extra now requires mcp>=2 (updated MCP server to the MCPServer API)
File reads/writes now specify an explicit encoding instead of relying on the platform default, fixing UnicodeDecodeError on non-UTF-8 locales (e.g. cp1252 on Windows) when config/source files contain UTF-8 characters. Reads use utf-8-sig to tolerate a BOM; writes use utf-8
PipelineNode.sources - a named dict of data sources replacing the single source field. All existing source: YAML is automatically migrated to sources: {df: ...}.
null_equals_null option for cdc_merge_options [#586]
All laktory-injected Spark conf / pipeline configuration keys now use laktory. namespace prefix (laktory.executor, laktory.requirements, laktory.config_filepath, laktory.pipeline_name)
PipelineNode.execute() now reads all sources entries before invoking the transformer. Upstream nodes referenced via {nodes.X} SQL placeholders are pre-loaded here rather than inside DataFrameExpr
Dtype class now supports time_unit and time_zone properties.
Renamed Databricks Pipelines orchestrator to Lakehouse Declarative Pipeline
Refactored Lakehouse Declarative Pipeline script to use latest API (apply_changes -> create_auto_cdc_flow)
CLI --dbks-job / --dbks-pipeline flags replaced by --databricks-job / --databricks-pipeline
PipelineNode.source field removed - use PipelineNode.sources (dict) instead. YAML using source: is automatically migrated; Python code accessing .source directly must be updated.
Renamed data sink databricks_quality_monitor to databricks_data_profiling_config
Removed pipeline databricks_quality_monitors_enabled flag (now auto-detected)
Pipeline Lakeflow Job orchestrator data profiling configuration task is no longer active by default. Needs to set data_profiling_config_task to True
QualityMonitor resource renamed to DataQualityMonitor to be aligned with Databricks latest naming convention
Lakeflow Job data profiling task key renamed from post-execute to data-profiling-configs
TerraformStack.model_dump(): resource reference substitution (${resources.xxx}) now uses structured dict traversal with str.replace for simple refs and re.escape + lambda for property refs, fixing incorrect matches when resource names contain regex metacharacters (e.g. a dot)
Improved documentation review as per 2026/05/08 audit.
Documentation website
Improved exceptions reporting during validation of data sources and sinks
VariableType to only accept specific format instead of generic string.
All Databricks resource models are now derived from auto-generated files built directly from the Terraform provider schema, ensuring fields stay in sync with the provider and reducing manual maintenance
Support for TableDataSourcereader_kwargs attribute [#513]
CustomWriter class and custom_writer field on BaseDataSink - fully replaces Laktory's built-in write logic with a user-supplied function; Laktory manages the streaming query lifecycle (foreachBatch, trigger, checkpoint) [#514]
CustomDataSource source type and CustomReader class - fully user-supplied read function as a first-class source alongside FileDataSource, UnityCatalogDataSource, etc.
LaktoryContext dataclass - runtime context object (node, pipeline, sink, source) optionally injected into user-supplied functions in CustomWriter, CustomReader, and DataFrameMethod by declaring laktory_context=None in the function signature [#515]
More robust detection of file existence on DBFS and prevention of "com.databricks.sql.io.CloudFileNotFoundException" exception when purging checkpoints.