damast.core.polars_dataframe

damast.core.polars_dataframe#

Attributes#

`logger`
`VAEX_HDF5_ROOT`
`VAEX_HDF5_COLUMNS`
`POLARS_TYPE_DICT`

Classes#

`Meta`
`PolarsDataFrame`

Module Contents#

damast.core.polars_dataframe.logger#

damast.core.polars_dataframe.VAEX_HDF5_ROOT: str = '/table'#

damast.core.polars_dataframe.VAEX_HDF5_COLUMNS: str#

damast.core.polars_dataframe.POLARS_TYPE_DICT#

class damast.core.polars_dataframe.Meta#

Bases: type

_base_impl: ClassVar[str] = 'polars'#

__getattr__(attr_name)#

class damast.core.polars_dataframe.PolarsDataFrame(df: polars.LazyFrame | polars.DataFrame)#

_polars_dataframe: PolarsDataFrame#

_dataframe_collected: polars.DataFrame#

_minmax_cache: dict[str, tuple]#

property lazyframe: polars.LazyFrame#

The underlying polars.LazyFrame.

This is the sole point of mutation for the wrapped dataframe - assigning to it (rather than e.g. a plain, private instance attribute) is what lets us keep the collected() cache and the dataframe accessor consistent with the data that is actually stored, instead of silently returning a stale snapshot after an update.

classmethod types() → dict[str, any]#

classmethod resolve_type(type_txt: str)#

classmethod _resolve_parameterized_type(type_txt: str)#

Parse a constructor-call-style dtype repr (e.g. “Datetime(time_unit=’us’)” or “List(Float64)”) back into a polars.datatypes.DataType instance, or return None if type_txt isn’t of that shape.

Uses ast rather than eval to only ever construct a known polars dtype class, with arguments parsed as literals or (recursively) nested dtypes - never arbitrary code.

classmethod _resolve_arg(node: ast.expr)#: Resolve a single argument node of a parameterized dtype repr: a bare dtype name (e.g. “Float64”), a nested parameterized dtype (e.g. “Datetime(time_unit=’us’)” or “List(Float64)”), a dict/list/tuple of such (e.g. “Struct({‘a’: List(String)})”), or a plain literal (str, int, …).

property dataframe: PolarsDataFrame#

Allows to access the underlying dataframe directly.

Note

AnnotatedDataFrame behaves like a polars.LazyFrame, so typically you will not need to access the dataframe through this property.

Returns:: The underlying dataframe

collected()#

is_string(column_name: str) → bool#

is_numeric(column_name: str) → bool#

is_datetime(column_name: str) → bool#

is_bool(column_name: str) → bool#

is_date(column_name: str) → bool#

__getitem__(column_name: str)#

Make dataframe subscriptable and behave more like the pandas.DataFrame.

Parameters:: item – Name of the key when using [] operators
Returns:: item/column from the underlying vaex.dataframe

ensure_column(column_name: str)#: Ensure that a column exist, raise ValueError otherwise

property column_names: list[str]#: Get all column names (without collecting the full dataframe)

dtype(column_name: str) → polars.datatypes.DataType#: Get column dtype (without collecting the full dataframe)

set_dtype(column_name, representation_type) → polars.datatype.DataType#: Set the dtype for a column to the given representation type. Using polars cast functionality :return: The updated object

rescale(column_name: str, factor: float) → None#: Multiply a column’s values by factor in place (e.g. to convert between two equivalent physical units), preserving the column’s existing representation type.

precompute_minmax(column_names: list[str]) → None#

Compute min/max for several columns in a single collect() and cache the results, so that later minmax(column_name) calls for these columns are served from cache instead of each triggering their own collect().

The cache is invalidated automatically whenever lazyframe is reassigned.

minmax(column_name: str) → tuple[any, any]#: Tuple of min and max values of the given column

categories(column_name: str, max_count: int = 100) → list[str]#

minmax_stats(column_names: list[str]) → dict[str, dict[str, any]]#: Tuple of min and max values of the given column

stats(column_name: str) → damast.core.data_description.NumericValueStats#

__getattr__(attr_name)#

Ensure that this object behaves like a polars.LazyFrame.

Parameters:: attr_name – Attribute / Name of column
Returns:: The column data

__setitem__(key, values)#

Set the column for the annotated dataframe, and allow to behave like the polars.Dataframe.

Parameters:

key – Column name
value – Value to set the column to

__len__() → int#

Get the length of the (underlying) dataframe.

Returns:: Length of the dataframe

equals(other: PolarsDataFrame) → bool#

classmethod open(path: str | pathlib.Path, sep=',') → polars.LazyFrame#

classmethod from_vaex_hdf5(path: str | pathlib.Path) → tuple[polars.LazyFrame, damast.core.metadata.MetaData]#: Load hdf5 file and (damast) metadata if found in the file.

NETCDF_BATCH_SIZE: ClassVar[int] = 100000#

classmethod import_netcdf(path: list[str | pathlib.Path]) → tuple[polars.LazyFrame, dict[str, MetaData]]#: Lazily scan NetCDF files - see scan_netcdf() - and extract metadata from their CF attributes, see _metadata_from_cf_attributes().

classmethod scan_netcdf(path: str | pathlib.Path) → tuple[polars.LazyFrame, dict[str, tuple[dict, dict]]]#

Lazily scan a NetCDF file as a table with one row per grid cell - the same layout as xarray.Dataset.to_dataframe(), with the dimensions as leading columns.

Nothing is read until the frame is collected. The grid is then read in slices along its first dimension, so memory is bounded by a slice rather than the whole grid, and rows are filtered/projected/limited per slice. Rows in which every data variable spanning the full grid is missing - e.g. the padding of a sparse (entity x time) grid - are dropped.

Parameters:: path – The NetCDF file
Returns:: The lazyframe, and variable name -> (CF attributes, xarray encoding)

static _netcdf_schema(ds) → polars.Schema#: Columns and dtypes of ds.to_dataframe() without reading data: taken from an empty slice, where object columns (strings) cannot be inferred and default to String.

classmethod _metadata_from_cf_attributes(schema: polars.Schema, variables: dict[str, tuple[dict, dict]], source: str) → damast.core.metadata.MetaData | None#

Create metadata for the columns of a loaded NetCDF file from the CF attributes of its variables: ‘long_name’ becomes the description, ‘units’ the unit - if it can be parsed -, and ‘valid_range’/’valid_min’/’valid_max’ the value range of a numeric column.

‘_FillValue’/’missing_value’ are not mapped: xarray already decodes them to NaN (null in polars), while damast’s missing_value is the value used to replace out-of-range values.

Parameters:: variables – variable name -> (attributes, xarray encoding)
Returns:: The metadata, or None if no variable carries any of these attributes - so that callers can fall back to searching for a spec file or inferring the metadata

static _cf_valid_range(attrs: dict, encoding: dict) → tuple[Any, Any] | None#

(min, max) from the CF ‘valid_range’, or ‘valid_min’/’valid_max’ attributes - an open side becomes -inf/inf. CF defines them in packed units, so they are unpacked like the data via ‘scale_factor’/’add_offset’, which xarray moves into the variable’s encoding.

Returns:: The range, or None if the variable declares none

classmethod import_hdf5(files: str | pathlib.Path | list[str | pathlib.Path]) → tuple[polars.LazyFrame, dict[str, MetaData]]#

Import a dataframe stored as HDF5.

This method tries to load using pandas first, then falls back to reading a vaex-based format using pytables.

classmethod export_hdf5(df: polars.DataFrame | polars.LazyFrame, path: str | pathlib.Path) → pathlib.Path#: Export the dataframe as hdf5. Please use only if really needed, otherwise, stick with the default format (parquet).