Efficient & Scalable Builds with AWS Parallel Cluster + Spack

Combining:

1. Spack
2. AWS Parallel Cluster
3. Slurm
4. Lustre / AWS FSx for Linux

Benefits:

• Accurate, reproducible builds of a wide range of community and custom C++ and Python packages applications. E.g.,
  • Easily deploy VNC server to HPC example
• Builds against multiple compilers / compiler versions / CPU ABIs. E.g.:
  • Building versions of Python 3.9, 3.10, 3.11, 3.12 and 3.13 each compiled with all of GCC v11, v12, v13, v14, v15 (all which are themselves compiled automatically) is a two-line configuration job, and the resulting compilation can be parallelised across the whole SLURM cluster
• Full rebuild from sources right through to the base packages (CPython, numpy, etc)
• 1000s of packages built < 1h, 1000s of cores deployed on single compilation job
• Scale down to low cost / fast scale up