workflow#

hydromodpy.workflow provides composable steps and an immutable PipelineState payload that flows from one step to the next. The Project facade and every workflow launcher (simulation, calibration, batch, comparison, testbed) reuse these steps so the same TOML always drives the same lifecycle.

Sub-modules#

  • workflow/internals/state.py – PipelineState[TPayload] generic dataclass, immutable, specialised by step (ValidatedState, ResolvedState, GeographicState, LoadedState, MeshedState, SetupState, OpenStoreState, SolverRanState, ExtractedState, DerivedState, ExportedState). Transit through state.advance(...).

  • workflow/internals/step.py – Step[TIn, TOut] Protocol (name, run(state) -> state, optional config_sections).

  • workflow/internals/checkpoint.py – CheckpointStore with HMAC-SHA256 signed pickle plus zstd compression.

  • workflow/internals/ledger.py – DuckDB ledger of executed steps.

  • workflow/internals/dependencies.py – earliest_affected_step resolves the first pipeline index affected by a dotted-path override (used by calibration to skip shared phases).

  • workflow/internals/derived.py – DerivedRegistry for ordered post-extraction calculations.

  • workflow/internals/manifest.py – WorkflowManifest summary for the lockfile.

  • workflow/runner.py – Pipeline runner that chains steps, manages the ledger, the optional checkpoint store, and typed errors (StepError exposes step_name and run_id).

  • workflow/steps/ – the canonical step library.

  • workflow/pipelines/ – pre-assembled pipelines for the standard workflows.

Twelve canonical steps#

Defined under workflow/steps/ and re-exported from workflow/steps/__init__.py:

ValidateStep, ResolveStep, LoadDataStep, BuildGeographicStep, BuildMeshStep, SetupProcessStep, PrepareSolverStep, RunSolverStep, ExtractStep, DeriveStep, ExportStep, DisplayStep.

Step contract#

class XyzStep:
    name: ClassVar[str] = "xyz"
    tin: ClassVar[type] = InputPayloadType
    tout: ClassVar[type] = OutputPayloadType
    config_sections: ClassVar[tuple[str, ...]] = ("flow.param",)

    def run(self, state: PipelineState) -> PipelineState:
        ...
        return state.advance(
            step_index=...,
            step_name=self.name,
            payload=...,
        )

The config_sections annotation declares which TOML subtrees the step reads. earliest_affected_step walks every step’s declarations to decide which phases can be skipped per calibration trial.

Key public symbols#

  • hydromodpy.workflow.internals.state.PipelineState

  • hydromodpy.workflow.internals.step.Step

  • hydromodpy.workflow.internals.checkpoint.CheckpointStore

  • hydromodpy.workflow.internals.dependencies.earliest_affected_step

  • hydromodpy.workflow.internals.derived.DerivedRegistry

  • hydromodpy.workflow.runner.Pipeline

  • hydromodpy.workflow.steps.{ValidateStep, ResolveStep, ..., DisplayStep}

The v1 StepsLedger module is removed; workflow steps are now recorded in the project catalog.duckdb under the workflow_steps table (see Storage Layout).

Layer-matrix neighbours#

  • Allowed targets: core, schema, config, physics, data, spatial, simulation, solver, calibration, results, display, analysis, reporting and workflow.

  • Allowed sources: project and cli.

See also#