damast.core.deferred

Checks and statistics that a pipeline run collects without executing the pipeline for each of them.

Every collect re-runs the whole lazy plan from the input files, so checking value ranges, counting rows and computing statistics step by step would execute the pipeline once per check. Instead a damast.core.dataprocessing.DataProcessingPipeline run registers them as queries in a PendingQueries, and they are all computed in the single streaming pass that eventually produces the result - e.g. damast.core.dataframe.AnnotatedDataFrame.export().

Functions

current()	The queries of the pipeline run in progress, or None outside a run.
running(pending)	Make pending the run in progress, so that checks register there instead of executing.

Classes

Deferred(queries, finish)	A value computed from one or more queries once they are executed.
PendingQueries()	Queries waiting to be executed together, each with what to do with its result.