How to perform experiments with damast#
One of the main motivation of this library is to facilitate the development and evaluation of Machine-Learning models. Hence, ‘damast’ offers a mini-framework and API to simplify the development of machine learning models. This requires a ‘hopefully’ minimal set of constraints - as what is envisioned by the authors of this library - so that researchers and ML-starters have lower entry barrier into running machine learning.
That being said, we give an example here on a minimal experiment.
!pip install 'damast[ml]'
Requirement already satisfied: damast[ml] in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (0.1.12)
Requirement already satisfied: astropy in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from damast[ml]) (6.1.7)
Requirement already satisfied: cloudpickle in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from damast[ml]) (3.1.1)
Requirement already satisfied: keras>=3.0 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from damast[ml]) (3.11.3)
Requirement already satisfied: matplotlib in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from damast[ml]) (3.10.5)
Requirement already satisfied: numba in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from damast[ml]) (0.61.2)
Requirement already satisfied: numpy>=2 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from damast[ml]) (2.2.6)
Requirement already satisfied: polars>=1.20 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from damast[ml]) (1.32.3)
Requirement already satisfied: psutil in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from damast[ml]) (7.0.0)
Requirement already satisfied: pyais in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from damast[ml]) (2.13.1)
Requirement already satisfied: pyarrow in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from damast[ml]) (21.0.0)
Requirement already satisfied: pydantic>=2.0 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from damast[ml]) (2.11.7)
Requirement already satisfied: ratarmount>=1.1 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from damast[ml]) (1.2.0)
Requirement already satisfied: scikit-learn in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from damast[ml]) (1.7.1)
Requirement already satisfied: tables in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from damast[ml]) (3.10.1)
Requirement already satisfied: torch in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from damast[ml]) (2.8.0)
Requirement already satisfied: tqdm in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from damast[ml]) (4.67.1)
Collecting jax[cpu] (from damast[ml])
Downloading jax-0.6.2-py3-none-any.whl.metadata (13 kB)
Collecting tensorflow (from damast[ml])
Downloading tensorflow-2.20.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.5 kB)
Requirement already satisfied: absl-py in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from keras>=3.0->damast[ml]) (2.3.1)
Requirement already satisfied: rich in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from keras>=3.0->damast[ml]) (14.1.0)
Requirement already satisfied: namex in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from keras>=3.0->damast[ml]) (0.1.0)
Requirement already satisfied: h5py in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from keras>=3.0->damast[ml]) (3.14.0)
Requirement already satisfied: optree in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from keras>=3.0->damast[ml]) (0.17.0)
Requirement already satisfied: ml-dtypes in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from keras>=3.0->damast[ml]) (0.5.3)
Requirement already satisfied: packaging in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from keras>=3.0->damast[ml]) (25.0)
Requirement already satisfied: annotated-types>=0.6.0 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from pydantic>=2.0->damast[ml]) (0.7.0)
Requirement already satisfied: pydantic-core==2.33.2 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from pydantic>=2.0->damast[ml]) (2.33.2)
Requirement already satisfied: typing-extensions>=4.12.2 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from pydantic>=2.0->damast[ml]) (4.15.0)
Requirement already satisfied: typing-inspection>=0.4.0 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from pydantic>=2.0->damast[ml]) (0.4.1)
Requirement already satisfied: ratarmountcore~=0.10.0 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from ratarmountcore[7z,bzip2,fat,gzip,rar,xz,zip,zstd]~=0.10.0->ratarmount>=1.1->damast[ml]) (0.10.1)
Requirement already satisfied: mfusepy~=3.0 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from ratarmount>=1.1->damast[ml]) (3.0.0)
Requirement already satisfied: rarfile~=4.1 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from ratarmountcore[7z,bzip2,fat,gzip,rar,xz,zip,zstd]~=0.10.0->ratarmount>=1.1->damast[ml]) (4.2)
Requirement already satisfied: indexed_zstd<2.0,>=1.2.2 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from ratarmountcore[7z,bzip2,fat,gzip,rar,xz,zip,zstd]~=0.10.0->ratarmount>=1.1->damast[ml]) (1.6.1)
Requirement already satisfied: python-xz~=0.4.0 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from ratarmountcore[7z,bzip2,fat,gzip,rar,xz,zip,zstd]~=0.10.0->ratarmount>=1.1->damast[ml]) (0.4.0)
Requirement already satisfied: rapidgzip~=0.15.0 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from ratarmountcore[7z,bzip2,fat,gzip,rar,xz,zip,zstd]~=0.10.0->ratarmount>=1.1->damast[ml]) (0.15.2)
Requirement already satisfied: indexed_gzip~=1.7 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from ratarmountcore[7z,bzip2,fat,gzip,rar,xz,zip,zstd]~=0.10.0->ratarmount>=1.1->damast[ml]) (1.10.1)
Requirement already satisfied: fast_zip_decryption in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from ratarmountcore[7z,bzip2,fat,gzip,rar,xz,zip,zstd]~=0.10.0->ratarmount>=1.1->damast[ml]) (3.0.0)
Requirement already satisfied: libarchive-c<6.0,~=5.1 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from ratarmountcore[7z,bzip2,fat,gzip,rar,xz,zip,zstd]~=0.10.0->ratarmount>=1.1->damast[ml]) (5.3)
Requirement already satisfied: py7zr~=1.0 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from ratarmountcore[7z,bzip2,fat,gzip,rar,xz,zip,zstd]~=0.10.0->ratarmount>=1.1->damast[ml]) (1.0.0)
Requirement already satisfied: texttable in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from py7zr~=1.0->ratarmountcore[7z,bzip2,fat,gzip,rar,xz,zip,zstd]~=0.10.0->ratarmount>=1.1->damast[ml]) (1.7.0)
Requirement already satisfied: pycryptodomex>=3.20.0 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from py7zr~=1.0->ratarmountcore[7z,bzip2,fat,gzip,rar,xz,zip,zstd]~=0.10.0->ratarmount>=1.1->damast[ml]) (3.23.0)
Requirement already satisfied: brotli>=1.1.0 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from py7zr~=1.0->ratarmountcore[7z,bzip2,fat,gzip,rar,xz,zip,zstd]~=0.10.0->ratarmount>=1.1->damast[ml]) (1.1.0)
Requirement already satisfied: pyzstd>=0.16.1 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from py7zr~=1.0->ratarmountcore[7z,bzip2,fat,gzip,rar,xz,zip,zstd]~=0.10.0->ratarmount>=1.1->damast[ml]) (0.17.0)
Requirement already satisfied: pyppmd<1.3.0,>=1.1.0 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from py7zr~=1.0->ratarmountcore[7z,bzip2,fat,gzip,rar,xz,zip,zstd]~=0.10.0->ratarmount>=1.1->damast[ml]) (1.2.0)
Requirement already satisfied: pybcj<1.1.0,>=1.0.0 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from py7zr~=1.0->ratarmountcore[7z,bzip2,fat,gzip,rar,xz,zip,zstd]~=0.10.0->ratarmount>=1.1->damast[ml]) (1.0.6)
Requirement already satisfied: multivolumefile>=0.2.3 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from py7zr~=1.0->ratarmountcore[7z,bzip2,fat,gzip,rar,xz,zip,zstd]~=0.10.0->ratarmount>=1.1->damast[ml]) (0.2.3)
Requirement already satisfied: inflate64<1.1.0,>=1.0.0 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from py7zr~=1.0->ratarmountcore[7z,bzip2,fat,gzip,rar,xz,zip,zstd]~=0.10.0->ratarmount>=1.1->damast[ml]) (1.0.3)
Requirement already satisfied: pyerfa>=2.0.1.1 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from astropy->damast[ml]) (2.0.1.5)
Requirement already satisfied: astropy-iers-data>=0.2024.10.28.0.34.7 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from astropy->damast[ml]) (0.2025.8.25.0.36.58)
Requirement already satisfied: PyYAML>=3.13 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from astropy->damast[ml]) (6.0.2)
Collecting jaxlib<=0.6.2,>=0.6.2 (from jax[cpu]; extra == "ml"->damast[ml])
Downloading jaxlib-0.6.2-cp310-cp310-manylinux2014_x86_64.whl.metadata (1.3 kB)
Collecting opt_einsum (from jax[cpu]; extra == "ml"->damast[ml])
Downloading opt_einsum-3.4.0-py3-none-any.whl.metadata (6.3 kB)
Requirement already satisfied: scipy>=1.12 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from jax[cpu]; extra == "ml"->damast[ml]) (1.15.3)
Requirement already satisfied: contourpy>=1.0.1 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from matplotlib->damast[ml]) (1.3.2)
Requirement already satisfied: cycler>=0.10 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from matplotlib->damast[ml]) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from matplotlib->damast[ml]) (4.59.1)
Requirement already satisfied: kiwisolver>=1.3.1 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from matplotlib->damast[ml]) (1.4.9)
Requirement already satisfied: pillow>=8 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from matplotlib->damast[ml]) (11.3.0)
Requirement already satisfied: pyparsing>=2.3.1 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from matplotlib->damast[ml]) (3.2.3)
Requirement already satisfied: python-dateutil>=2.7 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from matplotlib->damast[ml]) (2.9.0.post0)
Requirement already satisfied: six>=1.5 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from python-dateutil>=2.7->matplotlib->damast[ml]) (1.17.0)
Requirement already satisfied: llvmlite<0.45,>=0.44.0dev0 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from numba->damast[ml]) (0.44.0)
Requirement already satisfied: bitarray in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from pyais->damast[ml]) (3.7.0)
Requirement already satisfied: attrs in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from pyais->damast[ml]) (25.3.0)
Requirement already satisfied: markdown-it-py>=2.2.0 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from rich->keras>=3.0->damast[ml]) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from rich->keras>=3.0->damast[ml]) (2.19.2)
Requirement already satisfied: mdurl~=0.1 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from markdown-it-py>=2.2.0->rich->keras>=3.0->damast[ml]) (0.1.2)
Requirement already satisfied: joblib>=1.2.0 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from scikit-learn->damast[ml]) (1.5.1)
Requirement already satisfied: threadpoolctl>=3.1.0 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from scikit-learn->damast[ml]) (3.6.0)
Requirement already satisfied: numexpr>=2.6.2 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from tables->damast[ml]) (2.11.0)
Requirement already satisfied: py-cpuinfo in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from tables->damast[ml]) (9.0.0)
Requirement already satisfied: blosc2>=2.3.0 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from tables->damast[ml]) (3.7.2)
Requirement already satisfied: ndindex in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from blosc2>=2.3.0->tables->damast[ml]) (1.10.0)
Requirement already satisfied: msgpack in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from blosc2>=2.3.0->tables->damast[ml]) (1.1.1)
Requirement already satisfied: platformdirs in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from blosc2>=2.3.0->tables->damast[ml]) (4.3.8)
Requirement already satisfied: requests in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from blosc2>=2.3.0->tables->damast[ml]) (2.32.5)
Requirement already satisfied: charset_normalizer<4,>=2 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from requests->blosc2>=2.3.0->tables->damast[ml]) (3.4.3)
Requirement already satisfied: idna<4,>=2.5 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from requests->blosc2>=2.3.0->tables->damast[ml]) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from requests->blosc2>=2.3.0->tables->damast[ml]) (2.5.0)
Requirement already satisfied: certifi>=2017.4.17 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from requests->blosc2>=2.3.0->tables->damast[ml]) (2025.8.3)
Collecting astunparse>=1.6.0 (from tensorflow->damast[ml])
Downloading astunparse-1.6.3-py2.py3-none-any.whl.metadata (4.4 kB)
Collecting flatbuffers>=24.3.25 (from tensorflow->damast[ml])
Downloading flatbuffers-25.2.10-py2.py3-none-any.whl.metadata (875 bytes)
Collecting gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 (from tensorflow->damast[ml])
Downloading gast-0.6.0-py3-none-any.whl.metadata (1.3 kB)
Collecting google_pasta>=0.1.1 (from tensorflow->damast[ml])
Downloading google_pasta-0.2.0-py3-none-any.whl.metadata (814 bytes)
Collecting libclang>=13.0.0 (from tensorflow->damast[ml])
Downloading libclang-18.1.1-py2.py3-none-manylinux2010_x86_64.whl.metadata (5.2 kB)
Collecting protobuf>=5.28.0 (from tensorflow->damast[ml])
Downloading protobuf-6.32.0-cp39-abi3-manylinux2014_x86_64.whl.metadata (593 bytes)
Requirement already satisfied: setuptools in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from tensorflow->damast[ml]) (80.9.0)
Collecting termcolor>=1.1.0 (from tensorflow->damast[ml])
Downloading termcolor-3.1.0-py3-none-any.whl.metadata (6.4 kB)
Requirement already satisfied: wrapt>=1.11.0 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from tensorflow->damast[ml]) (1.17.3)
Collecting grpcio<2.0,>=1.24.3 (from tensorflow->damast[ml])
Downloading grpcio-1.74.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.8 kB)
Collecting tensorboard~=2.20.0 (from tensorflow->damast[ml])
Downloading tensorboard-2.20.0-py3-none-any.whl.metadata (1.8 kB)
Collecting markdown>=2.6.8 (from tensorboard~=2.20.0->tensorflow->damast[ml])
Downloading markdown-3.8.2-py3-none-any.whl.metadata (5.1 kB)
Collecting tensorboard-data-server<0.8.0,>=0.7.0 (from tensorboard~=2.20.0->tensorflow->damast[ml])
Downloading tensorboard_data_server-0.7.2-py3-none-manylinux_2_31_x86_64.whl.metadata (1.1 kB)
Collecting werkzeug>=1.0.1 (from tensorboard~=2.20.0->tensorflow->damast[ml])
Downloading werkzeug-3.1.3-py3-none-any.whl.metadata (3.7 kB)
Requirement already satisfied: wheel<1.0,>=0.23.0 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from astunparse>=1.6.0->tensorflow->damast[ml]) (0.45.1)
Requirement already satisfied: MarkupSafe>=2.1.1 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from werkzeug>=1.0.1->tensorboard~=2.20.0->tensorflow->damast[ml]) (3.0.2)
Requirement already satisfied: filelock in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from torch->damast[ml]) (3.19.1)
Requirement already satisfied: sympy>=1.13.3 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from torch->damast[ml]) (1.14.0)
Requirement already satisfied: networkx in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from torch->damast[ml]) (3.4.2)
Requirement already satisfied: jinja2 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from torch->damast[ml]) (3.1.6)
Requirement already satisfied: fsspec in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from torch->damast[ml]) (2025.7.0)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.8.93 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from torch->damast[ml]) (12.8.93)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.8.90 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from torch->damast[ml]) (12.8.90)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.8.90 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from torch->damast[ml]) (12.8.90)
Requirement already satisfied: nvidia-cudnn-cu12==9.10.2.21 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from torch->damast[ml]) (9.10.2.21)
Requirement already satisfied: nvidia-cublas-cu12==12.8.4.1 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from torch->damast[ml]) (12.8.4.1)
Requirement already satisfied: nvidia-cufft-cu12==11.3.3.83 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from torch->damast[ml]) (11.3.3.83)
Requirement already satisfied: nvidia-curand-cu12==10.3.9.90 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from torch->damast[ml]) (10.3.9.90)
Requirement already satisfied: nvidia-cusolver-cu12==11.7.3.90 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from torch->damast[ml]) (11.7.3.90)
Requirement already satisfied: nvidia-cusparse-cu12==12.5.8.93 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from torch->damast[ml]) (12.5.8.93)
Requirement already satisfied: nvidia-cusparselt-cu12==0.7.1 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from torch->damast[ml]) (0.7.1)
Requirement already satisfied: nvidia-nccl-cu12==2.27.3 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from torch->damast[ml]) (2.27.3)
Requirement already satisfied: nvidia-nvtx-cu12==12.8.90 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from torch->damast[ml]) (12.8.90)
Requirement already satisfied: nvidia-nvjitlink-cu12==12.8.93 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from torch->damast[ml]) (12.8.93)
Requirement already satisfied: nvidia-cufile-cu12==1.13.1.3 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from torch->damast[ml]) (1.13.1.3)
Requirement already satisfied: triton==3.4.0 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from torch->damast[ml]) (3.4.0)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages (from sympy>=1.13.3->torch->damast[ml]) (1.3.0)
Downloading jax-0.6.2-py3-none-any.whl (2.7 MB)
?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/2.7 MB ? eta -:--:--
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.7/2.7 MB 99.3 MB/s 0:00:00
?25hDownloading jaxlib-0.6.2-cp310-cp310-manylinux2014_x86_64.whl (89.9 MB)
?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/89.9 MB ? eta -:--:--
━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━ 57.7/89.9 MB 287.9 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 89.9/89.9 MB 285.7 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 89.9/89.9 MB 215.2 MB/s 0:00:00
?25hDownloading opt_einsum-3.4.0-py3-none-any.whl (71 kB)
Downloading tensorflow-2.20.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (620.4 MB)
?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/620.4 MB ? eta -:--:--
━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.7/620.4 MB 288.6 MB/s eta 0:00:02
━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 116.1/620.4 MB 288.3 MB/s eta 0:00:02
━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 168.8/620.4 MB 279.8 MB/s eta 0:00:02
━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━ 222.3/620.4 MB 275.7 MB/s eta 0:00:02
━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━ 279.4/620.4 MB 275.7 MB/s eta 0:00:02
━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━ 331.4/620.4 MB 270.2 MB/s eta 0:00:02
━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━ 386.9/620.4 MB 268.2 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━ 441.2/620.4 MB 271.1 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━ 497.3/620.4 MB 271.2 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━ 552.9/620.4 MB 270.0 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺ 609.2/620.4 MB 274.9 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 620.2/620.4 MB 274.2 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 620.2/620.4 MB 274.2 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 620.2/620.4 MB 274.2 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 620.2/620.4 MB 274.2 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 620.2/620.4 MB 274.2 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 620.2/620.4 MB 274.2 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 620.2/620.4 MB 274.2 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 620.2/620.4 MB 274.2 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 620.2/620.4 MB 274.2 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 620.2/620.4 MB 274.2 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 620.2/620.4 MB 274.2 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 620.4/620.4 MB 79.5 MB/s 0:00:04
?25hDownloading grpcio-1.74.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.2 MB)
?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/6.2 MB ? eta -:--:--
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.2/6.2 MB 250.4 MB/s 0:00:00
?25hDownloading tensorboard-2.20.0-py3-none-any.whl (5.5 MB)
?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/5.5 MB ? eta -:--:--
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.5/5.5 MB 211.4 MB/s 0:00:00
?25hDownloading tensorboard_data_server-0.7.2-py3-none-manylinux_2_31_x86_64.whl (6.6 MB)
?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/6.6 MB ? eta -:--:--
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.6/6.6 MB 233.5 MB/s 0:00:00
?25h
Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
Downloading flatbuffers-25.2.10-py2.py3-none-any.whl (30 kB)
Downloading gast-0.6.0-py3-none-any.whl (21 kB)
Downloading google_pasta-0.2.0-py3-none-any.whl (57 kB)
Downloading libclang-18.1.1-py2.py3-none-manylinux2010_x86_64.whl (24.5 MB)
?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/24.5 MB ? eta -:--:--
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 24.5/24.5 MB 190.9 MB/s 0:00:00
?25hDownloading markdown-3.8.2-py3-none-any.whl (106 kB)
Downloading protobuf-6.32.0-cp39-abi3-manylinux2014_x86_64.whl (322 kB)
Downloading termcolor-3.1.0-py3-none-any.whl (7.7 kB)
Downloading werkzeug-3.1.3-py3-none-any.whl (224 kB)
Installing collected packages: libclang, flatbuffers, werkzeug, termcolor, tensorboard-data-server, protobuf, opt_einsum, markdown, grpcio, google_pasta, gast, astunparse, tensorboard, jaxlib, jax, tensorflow
?25l
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/16 [libclang]
━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/16 [werkzeug]
━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4/16 [tensorboard-data-server]
━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━ 6/16 [opt_einsum]
━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 8/16 [grpcio]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━ 12/16 [tensorboard]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━ 12/16 [tensorboard]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━ 13/16 [jaxlib]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━ 13/16 [jaxlib]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━ 13/16 [jaxlib]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━ 13/16 [jaxlib]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━ 13/16 [jaxlib]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━ 13/16 [jaxlib]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━ 13/16 [jaxlib]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━ 13/16 [jaxlib]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━ 13/16 [jaxlib]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━ 14/16 [jax]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━ 14/16 [jax]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━ 14/16 [jax]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━ 14/16 [jax]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━ 14/16 [jax]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━ 14/16 [jax]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━ 14/16 [jax]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━ 14/16 [jax]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━ 14/16 [jax]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 15/16 [tensorflow]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16/16 [tensorflow]
?25h
Successfully installed astunparse-1.6.3 flatbuffers-25.2.10 gast-0.6.0 google_pasta-0.2.0 grpcio-1.74.0 jax-0.6.2 jaxlib-0.6.2 libclang-18.1.1 markdown-3.8.2 opt_einsum-3.4.0 protobuf-6.32.0 tensorboard-2.20.0 tensorboard-data-server-0.7.2 tensorflow-2.20.0 termcolor-3.1.0 werkzeug-3.1.3
# The list of modules used for this example
from collections import OrderedDict
from typing import List, Optional
from pathlib import Path
# For performance reasons the underlying data handling library is 'polars'
import polars
# The current development has focused on a keras+tensorflow based Machine Learning setup
import tensorflow as tf
import keras
import damast
from damast.core.transformations import CycleTransformer
# You can define custom units to annotate data, but otherwise astropy units will be used
from damast.core.units import units
# Data ranges can be defined as list, or marked with a lower-bound (min), upper-bound (max)
from damast.core.data_description import MinMax, CyclicMinMax
# The AnnotatedDataFrame combines a data specification and actual 'numeric' data
from damast.core.dataframe import AnnotatedDataFrame
# An AnnotatedDataFrame contains MetaData to describe the data
from damast.core.metadata import MetaData
# Data processing is centered around a DataProcessingPipeline which consists of multiple PipelineElement being run
# in sequence
from damast.core.dataprocessing import DataProcessingPipeline, PipelineElement
# To allow the machine learning process to be simplified, we offer a 'BaseModel' that should be inherited from
from damast.ml.models.base import BaseModel
# The experiment setup
from damast.ml.experiments import Experiment, LearningTask, ForecastTask, ModelInstanceDescription, TrainingParameters
# Allow to generate data for this particular example that uses data from the maritime domain
from damast.domains.maritime.ais.data_generator import AISTestData, AISTestDataSpec
2025-08-26 09:24:46.144538: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-08-26 09:24:50.215345: E external/local_xla/xla/stream_executor/cuda/cuda_platform.cc:51] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)
To illustrate a full experiment, we require a data processing pipeline to be set up. This pipeline will extract all those features, that are necessary to train the Machine Learning model(s). The pipeline will run transformations on the data, as provided here by a LatLonTransformer.
class LatLonTransformer(PipelineElement):
"""
The LatLonTransformer will consume a lat(itude) and a lon(gitude) column and perform
cyclic normalization. It will add four columns to a dataframe, namely lat_x, lat_y, lon_x, lon_y.
"""
@damast.core.describe("Lat/Lon cyclic transformation")
@damast.core.input({
"lat": {"unit": units.deg},
"lon": {"unit": units.deg}
})
@damast.core.output({
"lat_x": {"value_range": MinMax(-1.0, 1.0)},
"lat_y": {"value_range": MinMax(-1.0, 1.0)},
"lon_x": {"value_range": MinMax(-1.0, 1.0)},
"lon_y": {"value_range": MinMax(-1.0, 1.0)}
})
def transform(self, df: AnnotatedDataFrame) -> AnnotatedDataFrame:
lat_cyclic_transformer = CycleTransformer(features=["lat"], n=180.0)
lon_cyclic_transformer = CycleTransformer(features=["lon"], n=360.0)
_df = lat_cyclic_transformer.fit_transform(df=df)
_df = lon_cyclic_transformer.fit_transform(df=_df)
df._dataframe = _df
return df
The selected example model here, will require the above listed features as input - and provide a likewise-shaped output (for illustration purposes).
class Baseline(BaseModel):
"""
This is a placeholder ML model that illustrates the minimal
requirements.
"""
input_specs = OrderedDict({
"lat_x": {"length": 1},
"lat_y": {"length": 1},
"lon_x": {"length": 1},
"lon_y": {"length": 1}
})
output_specs = OrderedDict({
"lat_x": {"length": 1},
"lat_y": {"length": 1},
"lon_x": {"length": 1},
"lon_y": {"length": 1}
})
def __init__(self,
name: str,
features: List[str],
timeline_length: int,
output_dir: Path,
targets: Optional[List[str]] = None):
self.timeline_length = timeline_length
super().__init__(name=name,
output_dir=output_dir,
features=features,
targets=targets)
def _init_model(self):
features_width = len(self.features)
targets_width = len(self.targets)
self.model = tf.keras.models.Sequential([
keras.layers.Flatten(input_shape=[self.timeline_length, features_width]),
keras.layers.Dense(targets_width)
])
class BaselineA(Baseline):
"""Placeholder Model to illustrate the use of multiple models"""
pass
class BaselineB(Baseline):
"""Placeholder Model to illustrate the use of multiple models"""
pass
This example operates with synthetic, i.e. automatically generated data which is specific to the maritime domain. You will see a previous of the first 10 columns when running the following cell.
import tempfile
import shutil
tmp_path = Path(tempfile.gettempdir()) / "test-output-ais_preparation"
if tmp_path.exists():
shutil.rmtree(tmp_path)
tmp_path.mkdir(parents=True)
pipeline = DataProcessingPipeline(name="ais_preparation",
base_dir=tmp_path) \
.add("cyclic", LatLonTransformer())
features = ["lat_x", "lat_y", "lon_x", "lon_y"]
data = AISTestData(1000)
adf = AnnotatedDataFrame(dataframe=data.dataframe,
metadata=MetaData.from_dict(data=AISTestDataSpec.copy()))
dataset_filename = tmp_path / "test.hdf5"
adf.save(filename=dataset_filename)
adf.head(10)
/home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages/damast/core/polars_dataframe.py:406: PerformanceWarning:
your performance may suffer as PyTables will pickle object types that it cannot
map directly to c-types [inferred_type->mixed,key->block2_values] [items->Index(['date_time_utc', 'source'], dtype='object')]
pandas_df.to_hdf(path, key=DAMAST_HDF5_ROOT)
naive plan: (run LazyFrame.explain(optimized=True) to see the optimized plan)
SLICE[offset: 0, len: 10] DF ["mmsi", "lon", "lat", "date_time_utc", ...]; PROJECT */11 COLUMNS
A central idea to the experiment framework lies in providing a means for a consistent input and output to perform experiments. Hence, define a LearningTask (here: ForecastTask) that collects the learning parameters that define this task.
forecast_task = ForecastTask(
label="forecast-ais-short-sequence",
pipeline=pipeline, features=features,
models=[ModelInstanceDescription(BaselineA, {}),
ModelInstanceDescription(BaselineB, {}),
],
group_column="mmsi",
sequence_length=5,
forecast_length=1,
training_parameters=TrainingParameters(epochs=1,
validation_steps=1)
)
The actual experimentation takes a single LearningTask as input and it will output
experiment = Experiment(learning_task=forecast_task,
input_data=dataset_filename,
output_directory=tmp_path)
report = experiment.run()
with open(report, "r") as f:
print(f.read())
INFO:damast.core.dataframe:Loading hdf: files=[PosixPath('/tmp/test-output-ais_preparation/test.hdf5')]
INFO:damast.core.dataframe:No metadata provided or found in files - searching now for an existing spec file
INFO:damast.core.dataframe:Found metadata: ['/tmp/test-output-ais_preparation/test.spec.yaml']
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
File ~/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages/damast/core/pipeline.py:20, in Pipeline.run(self, df)
19 df = step.fit_transform(df)
---> 20 AnnotatedDataFrame.ensure_type(df)
21 except Exception as e:
File ~/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages/damast/core/dataframe.py:96, in AnnotatedDataFrame.ensure_type(cls, obj)
95 if not isinstance(obj._dataframe, DataFrame):
---> 96 raise ValueError(f"AnnotatedDataFrame._dataframe is not of type {DataFrame}")
ValueError: AnnotatedDataFrame._dataframe is not of type <class 'polars.lazyframe.frame.LazyFrame'>
During handling of the above exception, another exception occurred:
RuntimeError Traceback (most recent call last)
Cell In[7], line 4
1 experiment = Experiment(learning_task=forecast_task,
2 input_data=dataset_filename,
3 output_directory=tmp_path)
----> 4 report = experiment.run()
6 with open(report, "r") as f:
7 print(f.read())
File ~/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages/damast/ml/experiments.py:589, in Experiment.run(self, logging_level, report_filename)
586 permitted_values = filtered_groups.select(group_column).unique().collect()[:,0]
587 adf._dataframe = adf.filter(pl.col(group_column).is_in(permitted_values.implode()))
--> 589 features = self.compute_features(adf)
590 train_group, test_group, validate_group = \
591 self.compute_train_test_validate_groups(features,
592 group=group_column,
593 ratios=self._split_data_ratios)
595 experiment_dir = self.create_experiment_directory(base_dir=self.output_directory,
596 label=self.label)
File ~/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages/damast/ml/experiments.py:421, in Experiment.compute_features(self, adf)
411 def compute_features(self,
412 adf: AnnotatedDataFrame) -> AnnotatedDataFrame:
413 """
414 Compute features for an experiment by running the designated pipeline on input dataframe.
415
(...)
419 :param adf: The input dataframe
420 """
--> 421 adf_with_features = self.learning_task.pipeline.transform(adf)
423 self.learning_task.pipeline.save(dir=self.output_directory)
424 self.learning_task.pipeline.save_state(
425 df=adf_with_features,
426 dir=self.output_directory)
File ~/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages/damast/core/dataprocessing.py:852, in DataProcessingPipeline.transform(self, df)
847 if df.is_empty():
848 raise RuntimeError(
849 f"{self.__class__.__name__}.transform: there is no data available to transform"
850 )
--> 852 adf = pipeline.transform(df=in_df)
853 assert isinstance(adf, AnnotatedDataFrame)
855 adf.validate_metadata(validation_mode=damast.core.ValidationMode.UPDATE_METADATA)
File ~/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages/damast/core/pipeline.py:14, in Pipeline.transform(self, df)
13 def transform(self, df: AnnotatedDataFrame):
---> 14 return self.run(df=df)
File ~/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages/damast/core/pipeline.py:23, in Pipeline.run(self, df)
21 except Exception as e:
22 msg = ''.join(tc.format_exception(e)[-2:])
---> 23 raise RuntimeError(f"Step #{idx} in pipeline ({step.__class__.__name__}) failed: name_mappings: {step.name_mappings}\n\
24 {msg}")
25 return df
RuntimeError: Step #0 in pipeline (LatLonTransformer) failed: name_mappings: {}
File "/home/runner/work/damast/damast/.tox/build_docs/lib/python3.10/site-packages/damast/core/dataframe.py", line 96, in ensure_type
raise ValueError(f"AnnotatedDataFrame._dataframe is not of type {DataFrame}")
ValueError: AnnotatedDataFrame._dataframe is not of type <class 'polars.lazyframe.frame.LazyFrame'>
The outputs of an experiment are collected inside a dedicated (timestamped) folder. This folder will also contain a subfolder for each of the parametrized models that defines a LearningTask.
last_experiments = sorted([str(f) for f in Path(experiment.output_directory).glob(pattern="*") if f.is_dir()])
print("Last experiment in: ", last_experiments[-1])
experiment_folder = sorted([str(f) for f in Path(last_experiments[-1]).glob(pattern="*")])
file_listing = '\n'.join(experiment_folder)
print("Contents:\n")
print(file_listing)
Once the training is running it can be monitored using tensorboard:
tensorboard --logdir=<experiments-directory>