TableDataSource

laktory.models.datasources.TableDataSource

Bases: BaseDataSource

PARAMETER	DESCRIPTION
as_stream	If Truesource is read as a streaming DataFrame. Currently only supported by Spark DataFrame backend. TYPE: bool | VariableType DEFAULT: False
catalog_name	Source table catalog name TYPE: str | None | VariableType DEFAULT: None
drop_duplicates	Remove duplicated rows from source using all columns if True or only the provided column names. TYPE: bool | list[str] | VariableType DEFAULT: None
drops	List of columns to drop TYPE: list | VariableType DEFAULT: None
filter	SQL expression used to select specific rows from the source table TYPE: str | VariableType DEFAULT: None
reader_kwargs	Keyword arguments passed directly to dataframe backend reader. Passed to .options() method when using PySpark. TYPE: dict[str | VariableType, Any | VariableType] | VariableType DEFAULT: {}
reader_methods	DataFrame backend reader methods. TYPE: list[ReaderWriterMethod | VariableType] | VariableType DEFAULT: []
renames	Mapping between the source column names and desired column names TYPE: dict[str | VariableType, str | VariableType] | VariableType DEFAULT: None
schema_name	Source table schema name TYPE: str | None | VariableType DEFAULT: None
selects	Columns to select from the source. Can be specified as a list or as a dictionary to rename the source columns TYPE: list[str] | dict[str, str] | VariableType DEFAULT: None
table_name	Source table name. Also supports fully qualified name ({catalog}.{schema}.{table}). In this case, catalog_name and schema_name arguments are ignored. TYPE: str | VariableType
type	Name of the data source type TYPE: Literal[CUSTOM, DATAFRAME, FILE, UNITY_CATALOG, HIVE_METASTORE] | VariableType

METHOD	DESCRIPTION
read	Read data with options specified in attributes.

ATTRIBUTE	DESCRIPTION
full_name	Table full name {catalog_name}.{schema_name}.{table_name} TYPE: str

full_name property

Table full name {catalog_name}.{schema_name}.{table_name}

read(**kwargs)

Read data with options specified in attributes.

RETURNS	DESCRIPTION
AnyFrame	Resulting dataframe

Source code in laktory/models/datasources/basedatasource.py

def read(self, **kwargs) -> AnyFrame:
    """
    Read data with options specified in attributes.

    Returns
    -------
    :
        Resulting dataframe
    """
    logger.info(
        f"Reading `{self.__class__.__name__}` {self._id} with {self.dataframe_backend}"
    )
    df = self._read(**kwargs)

    # Convert to Narwhals
    if not isinstance(df, (nw.LazyFrame, nw.DataFrame)):
        df = nw.from_native(df)

    # Post read
    df = self._post_read(df)

    logger.info("Read completed.")

    return df