Cutting Nextflow cost on spot instances without losing a run

Spot instances are cheap until a task gets reclaimed mid-run and the pipeline has no cheap way to recover. At pipeline scale — dozens of parallel tasks across a multi-hour DAG — reclamation isn't a rare event you can ignore, it's a scheduling condition you have to design for. The question isn't whether it happens, it's what it costs when it does.

The core problem

A naive setup treats a reclaimed instance as a failed pipeline: restart from the top, lose everything already computed. On a long DAG, that turns an occasional interruption into the dominant cost, because you keep re-paying for work that already finished once.

Patterns that help

Keep task granularity small. If a single process step represents hours of compute, losing it to reclamation is expensive. Breaking a monolithic step into smaller per-sample or per-chunk processes means a reclamation only costs the smallest unit of work Nextflow can retry independently.

Use Nextflow's built-in retry, not a custom wrapper. errorStrategy combined with maxRetries will resubmit a killed task without any custom orchestration:

process.errorStrategy = { task.exitStatus in [137, 143] ? 'retry' : 'terminate' }
process.maxRetries = 3
process.executor = 'awsbatch'

Exit codes 137 and 143 correspond to SIGKILL and SIGTERM, which is how most cloud schedulers signal a spot reclamation — matching on those specifically, rather than retrying every failure, avoids masking real bugs as transient infrastructure noise.

Run with -resume. Nextflow's work directory caches completed task outputs by content hash. A pipeline restarted with -resume after a partial failure only re-executes the tasks that didn't finish, not the whole DAG — this is the single biggest lever, and it's built in rather than something you have to engineer.

Diversify instance types in the pool. Cloud providers reclaim spot capacity per instance type/size, in waves tied to demand for that specific type. A pool spread across several comparable instance types is less likely to see a correlated mass-reclamation than a pool of one type at one size.

What this actually saves

Our own cloud-native variant calling case study reports a runtime and cost reduction against a "legacy pipeline" baseline that isn't documented or published yet — which is why that figure is flagged TODO-VERIFY on the project page. The patterns above are the mechanism that makes a number like that plausible in principle; they are not a substitute for publishing the actual baseline and benchmark that would let someone check it.