BaseDataSource

laktory.models.datasources.BaseDataSource

Bases: BaseModel, PipelineChild

Base class for data sources.

PARAMETER	DESCRIPTION
as_stream	If Truesource is read as a streaming DataFrame. Currently only supported by Spark DataFrame backend. TYPE: bool | VariableType DEFAULT: False
drop_duplicates	Remove duplicated rows from source using all columns if True or only the provided column names. TYPE: bool | list[str] | VariableType DEFAULT: None
drops	List of columns to drop TYPE: list | VariableType DEFAULT: None
filter	SQL expression used to select specific rows from the source table TYPE: str | VariableType DEFAULT: None
renames	Mapping between the source column names and desired column names TYPE: dict[str | VariableType, str | VariableType] | VariableType DEFAULT: None
selects	Columns to select from the source. Can be specified as a list or as a dictionary to rename the source columns TYPE: list[str] | dict[str, str] | VariableType DEFAULT: None
type	Name of the data source type TYPE: Literal['CUSTOM', 'DATAFRAME', 'FILE', 'UNITY_CATALOG', 'HIVE_METASTORE'] | VariableType

METHOD	DESCRIPTION
read	Read data with options specified in attributes.

read(**kwargs)

Read data with options specified in attributes.

RETURNS	DESCRIPTION
AnyFrame	Resulting dataframe

Source code in laktory/models/datasources/basedatasource.py

def read(self, **kwargs) -> AnyFrame:
    """
    Read data with options specified in attributes.

    Returns
    -------
    :
        Resulting dataframe
    """
    logger.info(
        f"Reading `{self.__class__.__name__}` {self._id} with {self.dataframe_backend}"
    )
    df = self._read(**kwargs)

    # Convert to Narwhals
    if not isinstance(df, (nw.LazyFrame, nw.DataFrame)):
        df = nw.from_native(df)

    # Post read
    df = self._post_read(df)

    logger.info("Read completed.")

    return df

---

laktory.models.datasources.basedatasource.DataFrameSample

Bases: BaseModel

PARAMETER	DESCRIPTION
fraction	TYPE: float | VariableType DEFAULT: None
n	TYPE: int | VariableType DEFAULT: None
seed	TYPE: int | None | VariableType DEFAULT: None

---

laktory.models.datasources.basedatasource.Watermark

Bases: BaseModel

References

https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#handling-late-data-and-watermarking

PARAMETER	DESCRIPTION
column	Event time column name TYPE: str | VariableType
threshold	How late, expressed in seconds, the data is expected to be with respect to event time. TYPE: str | VariableType