damast.core.polars_dataframe

damast.core.polars_dataframe#

Attributes#

`logger`
`VAEX_HDF5_ROOT`
`VAEX_HDF5_COLUMNS`
`POLARS_TYPE_DICT`
`MAX_CATEGORIES`

Classes#

`Meta`
`PolarsDataFrame`

Functions#

`row_position`(→ polars.Expr)	Each row's position, 0-based, as UInt64.
`usable_categories`(→ list \| None)	The distinct values of a string column if they make a sensible list of categories.

Module Contents#

damast.core.polars_dataframe.logger#

damast.core.polars_dataframe.VAEX_HDF5_ROOT: str = '/table'#

damast.core.polars_dataframe.VAEX_HDF5_COLUMNS: str#

damast.core.polars_dataframe.POLARS_TYPE_DICT#

class damast.core.polars_dataframe.Meta#

Bases: type

_base_impl: ClassVar[str] = 'polars'#

__getattr__(attr_name)#

damast.core.polars_dataframe.MAX_CATEGORIES: int = 100#

damast.core.polars_dataframe.row_position() → polars.Expr#

Each row’s position, 0-based, as UInt64.

A running sum over a column of ones rather than with_row_index, row_index or int_range: those use polars’ index type (32 bits in standard builds, so overflowing past 2**32 rows), and unlike them the sum streams instead of holding the frame in memory.

Returns:: The expression, for a frame with at least one column

damast.core.polars_dataframe.usable_categories(column_name: str, values: list, max_count: int = MAX_CATEGORIES) → list | None#

The distinct values of a string column if they make a sensible list of categories.

Parameters:

column_name – The column, for the error message
values – Its distinct values, sorted
max_count – Most values a list of categories may have

Returns:

The values, or None if there are too many or they look like time points

Raises:

ValueError – If a value is not a string

class damast.core.polars_dataframe.PolarsDataFrame(df: polars.LazyFrame | polars.DataFrame)#

_polars_dataframe: PolarsDataFrame#

_dataframe_collected: polars.DataFrame#

_minmax_cache: dict[str, tuple]#

_categories_cache: dict[str, list]#

_schema_cache: polars.Schema | None#

property lazyframe: polars.LazyFrame#

The underlying polars.LazyFrame.

This is the sole point of mutation for the wrapped dataframe - assigning to it (rather than e.g. a plain, private instance attribute) is what lets us keep the collected() cache and the dataframe accessor consistent with the data that is actually stored, instead of silently returning a stale snapshot after an update.

classmethod types() → dict[str, Any]#

classmethod resolve_type(type_txt: str)#

classmethod _resolve_parameterized_type(type_txt: str)#

Parse a constructor-call-style dtype repr (e.g. “Datetime(time_unit=’us’)” or “List(Float64)”) back into a polars.datatypes.DataType instance, or return None if type_txt isn’t of that shape.

Uses ast rather than eval to only ever construct a known polars dtype class, with arguments parsed as literals or (recursively) nested dtypes - never arbitrary code.

classmethod _resolve_arg(node: ast.expr)#: Resolve a single argument node of a parameterized dtype repr: a bare dtype name (e.g. “Float64”), a nested parameterized dtype (e.g. “Datetime(time_unit=’us’)” or “List(Float64)”), a dict/list/tuple of such (e.g. “Struct({‘a’: List(String)})”), or a plain literal (str, int, …).

property dataframe: PolarsDataFrame#

Allows to access the underlying dataframe directly.

Note

AnnotatedDataFrame behaves like a polars.LazyFrame, so typically you will not need to access the dataframe through this property.

Returns:: The underlying dataframe

collected()#

is_string(column_name: str) → bool#

is_numeric(column_name: str) → bool#

is_datetime(column_name: str) → bool#

is_bool(column_name: str) → bool#

is_date(column_name: str) → bool#

__getitem__(column_name: str)#

Make dataframe subscriptable and behave more like the pandas.DataFrame.

Parameters:: item – Name of the key when using [] operators
Returns:: item/column from the underlying vaex.dataframe

ensure_column(column_name: str)#: Ensure that a column exist, raise ValueError otherwise

property schema: polars.Schema#: Schema of the lazyframe - cached, since resolving it can require scanning the input, e.g. for a csv file with full schema inference

property column_names: list[str]#: Get all column names (without collecting the full dataframe)

dtype(column_name: str) → polars.datatypes.DataType#: Get column dtype (without collecting the full dataframe)

set_dtype(column_name, representation_type) → polars.datatype.DataType#: Set the dtype for a column to the given representation type. Using polars cast functionality, or parsing for a string column set to a datetime :return: The updated object

rescale(column_name: str, factor: float) → None#: Multiply a column’s values by factor in place (e.g. to convert between two equivalent physical units), preserving the column’s existing representation type.

precompute_minmax(column_names: list[str]) → None#

Compute min/max for several columns in a single collect() and cache the results, so that later minmax(column_name) calls for these columns are served from cache instead of each triggering their own collect().

The cache is invalidated automatically whenever lazyframe is reassigned.

minmax(column_name: str) → tuple[Any, Any]#: Tuple of min and max values of the given column

precompute_categories(column_names: list[str]) → None#

Compute the unique values of several columns in a single collect() and cache them, so that later categories(column_name) calls for these columns are served from cache instead of each triggering their own collect().

The cache is invalidated automatically whenever lazyframe is reassigned.

categories(column_name: str, max_count: int = MAX_CATEGORIES) → list[str]#

minmax_stats(column_names: list[str]) → dict[str, dict[str, Any]]#: Tuple of min and max values of the given column

stats(column_name: str) → damast.core.data_description.NumericValueStats#

__getattr__(attr_name)#

Ensure that this object behaves like a polars.LazyFrame.

Parameters:: attr_name – Attribute / Name of column
Returns:: The column data

__setitem__(key, values)#

Set the column for the annotated dataframe, and allow to behave like the polars.Dataframe.

Parameters:

key – Column name
value – Value to set the column to

__len__() → int#

Get the length of the (underlying) dataframe.

Returns:: Length of the dataframe

equals(other: PolarsDataFrame) → bool#

CSV_SEPARATOR_CANDIDATES: ClassVar[tuple[str, Ellipsis]] = (';', ',')#

CSV_SNIFF_MAX_LINES: ClassVar[int] = 100#

classmethod sniff_csv_separator(file: str | pathlib.Path, candidates: tuple[str, Ellipsis] = CSV_SEPARATOR_CANDIDATES, max_lines: int = CSV_SNIFF_MAX_LINES) → str#

Guess the column separator of a csv file from its first lines.

The separator a file was written with splits all of its rows into the same number of columns, so the candidate with the largest share of equally wide rows wins, ties going to the one that yields more columns. A candidate that leaves most rows in a single column did not split the file and is never chosen; if that holds for all of them, the first candidate is returned.

Parameters:

file – The csv file whose first lines should be inspected
candidates – Separators to consider, most-preferred first
max_lines – Number of lines to inspect. More lines make a separator that also occurs inside quoted fields easier to rule out - it splits those rows into a width of their own - at the cost of reading further into the file

Returns:

The most likely separator

classmethod scan_csv(files: str | pathlib.Path | list[str | pathlib.Path], separator: str = ',', **kwargs) → polars.LazyFrame#

Lazily scan csv file(s) with damast’s default arguments (DAMAST_CSV_DEFAULT_ARGS).

The schema is inferred once and then passed on explicitly: with full schema inference (infer_schema_length=None) polars would otherwise re-read the entire input on every collect() just to infer the schema again.

Parameters:

files – csv file(s) to scan
separator – Column separator
kwargs – Further arguments for polars.scan_csv, overriding the damast defaults

Returns:

The lazyframe

classmethod open(path: str | pathlib.Path, sep=',') → polars.LazyFrame#

classmethod from_vaex_hdf5(path: str | pathlib.Path) → tuple[polars.LazyFrame, damast.core.metadata.MetaData]#: Load hdf5 file and (damast) metadata if found in the file.

NETCDF_BATCH_SIZE: ClassVar[int] = 100000#

classmethod import_netcdf(path: list[str | pathlib.Path]) → tuple[polars.LazyFrame, dict[str, MetaData]]#: Lazily scan NetCDF files - see scan_netcdf() - and extract metadata from their CF attributes, see _metadata_from_cf_attributes().

classmethod scan_netcdf(path: str | pathlib.Path) → tuple[polars.LazyFrame, dict[str, tuple[dict, dict]]]#

Lazily scan a NetCDF file as a table with one row per grid cell - the same layout as xarray.Dataset.to_dataframe(), with the dimensions as leading columns.

Nothing is read until the frame is collected. The grid is then read in slices along its first dimension, so memory is bounded by a slice rather than the whole grid, and rows are filtered/projected/limited per slice. Rows in which every data variable spanning the full grid is missing - e.g. the padding of a sparse (entity x time) grid - are dropped.

Parameters:: path – The NetCDF file
Returns:: The lazyframe, and variable name -> (CF attributes, xarray encoding)

static _netcdf_schema(ds) → polars.Schema#: Columns and dtypes of ds.to_dataframe() without reading data: taken from an empty slice, where object columns (strings) cannot be inferred and default to String.

classmethod _metadata_from_cf_attributes(schema: polars.Schema, variables: dict[str, tuple[dict, dict]], source: str) → damast.core.metadata.MetaData | None#

Create metadata for the columns of a loaded NetCDF file from the CF attributes of its variables: ‘long_name’ becomes the description, ‘units’ the unit - if it can be parsed -, and ‘valid_range’/’valid_min’/’valid_max’ the value range of a numeric column.

‘_FillValue’/’missing_value’ are not mapped: xarray already decodes them to NaN (null in polars), while damast’s missing_value is the value used to replace out-of-range values.

Parameters:: variables – variable name -> (attributes, xarray encoding)
Returns:: The metadata, or None if no variable carries any of these attributes - so that callers can fall back to searching for a spec file or inferring the metadata

static _cf_valid_range(attrs: dict, encoding: dict) → tuple[Any, Any] | None#

(min, max) from the CF ‘valid_range’, or ‘valid_min’/’valid_max’ attributes - an open side becomes -inf/inf. CF defines them in packed units, so they are unpacked like the data via ‘scale_factor’/’add_offset’, which xarray moves into the variable’s encoding.

Returns:: The range, or None if the variable declares none

classmethod import_hdf5(files: str | pathlib.Path | list[str | pathlib.Path]) → tuple[polars.LazyFrame, dict[str, MetaData]]#

Import a dataframe stored as HDF5.

This method tries to load using pandas first, then falls back to reading a vaex-based format using pytables.

classmethod export_hdf5(df: polars.DataFrame | polars.LazyFrame, path: str | pathlib.Path) → pathlib.Path#: Export the dataframe as hdf5. Please use only if really needed, otherwise, stick with the default format (parquet).