Diverse users running across a diverse set of architectures, it seems Spack was built just for this!

I want to share my experiences with Spack, here my focus is on CP2K. My build environment is a Cray XC50 running Thunder X2 Arm CPUs.

Spack is a source-based package manager and build system, all builds take place in self-contained environments. Every installed package is defined by a spec, the spec is a string that represents the enabled features, compiler, etc.. that results in a unique package. The process of determining the packages unique hash is called concretization.

You can find all of this explained much better in the public documentation or by following the Spack Tutorial

Getting started

> spack compiler find

This populates ~/.spack/cray/compilers.yaml with all our compilers, Spack checks typical locations as well as your available modules.

First attempt

> spack install cp2k@8.1 +openmp +mpi +plumed %gcc@9.2.0 smm=blas ^fftw +openmp ^openblas threads=openmp

+openmp is default

+mpi to enable MPI support

+plumed to enable PLUMED

%gcc@9.2.0 uses GCC 9.2.0, loaded via a module by Spack

smm=blas is required as libxsmm not available on arm64, noted in spec .

^fftw +openmp enables OpenMP for the FFTW dependency

^openblas threads=openmp uses OpenMP for threadding in OpenBLAS, I found the threads= options in its spec file but I could have just run spack info openblas.

I saw some differing results when I loaded modules into my environment before running Spack so I try avoid loading any when building packages.

Now, this build fails to concretize and Spack shows us what it tried to do and which packages are conflicting:

> spack install cp2k@8.1 +openmp +mpi +plumed %gcc@9.2.0 smm=blas ^fftw +openmp ^openblas threads=openmp
==> Error: Conflicts in concretized spec "cp2k@8.1%gcc@9.2.0~cosma~cuda~cuda_arch_35_k20x~cuda_blas~cuda_fft~elpa+libint~libvori+libxc+mpi+openmp~pexsi+plumed~sirius~spglib cuda_arch=none lmax=5 smm=blas arch=cray-cnl7-aarch64/kizgfsf"
List of matching conflicts for spec:

    cp2k@8.1%gcc@9.2.0~cosma~cuda~cuda_arch_35_k20x~cuda_blas~cuda_fft~elpa+libint~libvori+libxc+mpi+openmp~pexsi+plumed~sirius~spglib cuda_arch=none lmax=5 smm=blas arch=cray-cnl7-aarch64
        ^fftw@3.3.9%gcc@9.2.0+mpi+openmp~pfft_patches precision=double,float arch=cray-cnl7-aarch64
        ^libint@2.6.0%gcc@9.2.0+fortran tune=cp2k-lmax-5 arch=cray-cnl7-aarch64
...
...
1. "^openblas threads=none" conflicts with "cp2k+openmp"

You can see the concretized spec of my cp2k installation is

cp2k@8.1%gcc@9.2.0~cosma~cuda~cuda_arch_35_k20x~cuda_blas~cuda_fft~elpa+libint~libvori+libxc+mpi+openmp~pexsi+plumed~sirius~spglib cuda_arch=none lmax=5 smm=blas arch=cray-cnl7-aarch64

This includes any defaults for every variant the package supports, I prefer to leave it up to the maintainer for variants I’m not familiar with, but if you want highly reproducible environments you will probably want to specify everything.

The error ^openblas threads=none" conflicts with "cp2k+openmp suggests the openblas dependency won’t support threading, which is required because I’m building cp2k with openmp support.

packages.yaml

OpenBLAS isn’t tricky to compile, so I opted to compile it manually:

> make
> make PREFIX=/software/arm64/apps/openblas/0.3.15 install

Now I need to tell Spack about my installation of OpenBLAS, this kind of package is called an External and they are listed in ~/.spack/packages.yaml:

packages:
  autoconf:
    externals:
    - spec: autoconf@2.69
      prefix: /usr
  automake:
    externals:
    - spec: automake@1.15.1
      prefix: /usr
  bash:
    externals:
    - spec: bash@4.4.23
      prefix: /
...

I added OpenBLAS and a few Cray libraries to my packages.yaml:

  mpi:
    externals:
    - spec: "mpich@7.7.16"
      modules:
      - cray-mpich/7.7.16
    buildable: False
  openblas:
    externals:
    - spec: "openblas@0.3.15"
      modules:
      - apps/openblas/0.3.15
  fftw:
    externals:
    - spec: "fftw+openmp"
      modules:
      - cray-fftw
  cray-libsci:
    externals:
    - spec: "cray-libsci"
      modules:
      - cray-libsci
    buildable: False

Each external has a spec that Spack matches against, for example if I ask for fftw explicitly without openmp support then it won’t use my fftw external as I have listed +openmp in its spec.

I used buildable: False to tell Spack to never build an alternative for some packages. I want to use the supported system libraries where possible, instead of some apps using cray-libsci and others building a custom libsci, as an example.

JSONDecodeError

This bug slowed me down, I didn’t find out the cause and it has since cleared itself up. It may be related to loading spack on different clusters with a shared home directory and it ended up loading something confusing in compilers.yaml? 🤷‍♂️

==> cp2k: Executing phase: 'edit'
==> Error: JSONDecodeError: Expecting value: line 1 column 1 (char 0)

/lustre/software/aarch64/spack/var/spack/repos/builtin/packages/cp2k/package.py:300, in edit:
        297            ldflags  += ['-hnoomp']
        298
        299        if '@7:' in spec:  # recent versions of CP2K use C++14 CUDA code
  >>    300            cxxflags.append(self.compiler.cxx14_flag)
        301            nvflags.append(self.compiler.cxx14_flag)
        302
        303        ldflags.append(fftw.libs.search_flags)

I opened bug #23570 to track it.

Missing build dependency

I got the following error from plumed: RuntimeError: Cannot generate configure: missing dependencies ['m4']

Spack seems to think m4 isn’t available, it turned out that the package’s package.py was missing the type='build' on the line where m4 is listed as a build depdenency. This was fixed in #23953 .

Build

Lastly I had difficulty building netlib-scalapack until I realised that cray-libsci provides the LAPACK/ScaLAPACK capabilities cp2k requires. I manually set the dependency package with ^cray-libsci.

> spack install cp2k@8.1 +openmp +mpi +plumed %gcc@9.2.0 smm=blas ^cray-libsci
==> Warning: Missing a source id for cray-libsci@20.06.1
==> Warning: Missing a source id for perl@5.26.1
==> Warning: Missing a source id for mpich@7.7.16
==> cray-libsci@20.06.1 : has external module in ['cray-libsci']
[+] /opt/cray/pe/libsci/20.06.1/GNU/8.1/aarch64 (external cray-libsci-20.06.1-lcxhogmgo4zwfosmyp6wddo7xvga4ybx)
==> fftw@3.3.9 : has external module in ['cray-fftw']
[+] /opt/cray/pe/fftw/3.3.8.5/arm_thunderx2 (external fftw-3.3.9-bwhmlzy6eik3ajeqobol527ww6s42vcn)
[+] /usr (external autoconf-2.69-ememqml5q3hrhatuxevmdaeyuocnaqnk)
[+] /usr (external automake-1.15.1-edymz5en4xwxwybe76d3535re7euacgl)
[+] /usr (external bzip2-1.0.6-2fqflhapqghfvebfk5kvogp7g4xulquw)
[+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/zlib-1.2.11-cx7za7abtiy5ezumlkfxsffzfxuj3xq6
[+] /usr (external libtool-2.4.6-utlxqeicabegxrlreyo3biinw7v4vswn)
[+] /usr (external m4-1.4.18-og3jty4rquctsfxnxzwaufttw6bylag4)
[+] /usr (external perl-5.26.1-lk6rvqusbsxtnumqre6ete2tvgxghgxx)
==> mpich@7.7.16 : has external module in ['cray-mpich/7.7.16']
[+] /opt/cray/pe/mpt/7.7.16/gni/mpich-gnu/8.2 (external mpich-7.7.16-ifjkd3v6a3hbtqmxekneycwh3wgrhc4k)
[+] /usr (external pkg-config-0.29.2-ueeszbkidqgqjkns6gh6glgaxqws2x6r)
[+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/gsl-2.6-vtuwngsrcrtb5uod2hbteacifcxich5c
[+] /usr (external python-3.6.10-ifrblj537zg2nvq5baegeotax4h5dzm3)
[+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/boost-1.76.0-qyx6cquksudterwipejedj3cbfmutmae
[+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/gmp-6.2.1-6wmrr6qexxutxbxctk5rq5d4qg5w2g3z
[+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/libxc-4.3.4-tur67qh7oxjbjjtmxmgi2hjty6dt4rue
[+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/py-setuptools-50.3.2-3rr4jqvqhjhjiclh2tkgl23x6uuj6aae
[+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/libint-2.6.0-3ym5vlezh7eisciqwhpeu7p74vqymata
[+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/py-cython-0.29.22-q4r6465kru3jpvf3ac2lliljzs4cb3an
[+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/plumed-2.6.2-vkrvu45ewjb3lvj5kf4uaj3x77pf3x6e
[+] /lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/cp2k-8.1-tnejptyboq56nppfikolzk5emc6vf5sb

Success!

In the end I don’t think my OpenBLAS build was necessary, looks like libsci handled that too!

My final Spack package spec:

> spack find -vdx cp2k
==> 1 installed package
-- cray-cnl7-aarch64 / gcc@9.2.0 --------------------------------
cp2k@8.1~cosma~cuda~cuda_arch_35_k20x~cuda_blas~cuda_fft~elpa+libint~libvori+libxc+mpi+openmp~pexsi+plumed~sirius~spglib cuda_arch=none lmax=5 smm=blas
    cray-libsci@20.06.1~mpi~openmp+shared
    fftw@3.3.9+mpi+openmp~pfft_patches precision=double,float
    libint@2.6.0+fortran tune=cp2k-lmax-5
        boost@1.76.0+atomic+chrono~clanglibcpp~container~context~coroutine+date_time~debug+exception~fiber+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave cxxstd=98 patches=57a8401dee8f52b0342e0c8147a5b2db834e8d8f3fbcbbc5950016bd3e9e1ef0 visibility=hidden
            bzip2@1.0.6~debug~pic+shared
            zlib@1.2.11+optimize+pic+shared
        gmp@6.2.1
    libxc@4.3.4~cuda+shared cuda_arch=none
    mpich@7.7.16~argobots+fortran+hwloc+hydra+libxml2+pci+romio~slurm~verbs+wrapperrpath device=ch4 netmod=ofi pmi=pmi
    plumed@2.6.2+gsl+mpi+shared arrayfire=none optional_modules=all
        gsl@2.6~external-cblas

Playtime

Now with Spack loaded, the simplest way to load this build of cp2k into my environment is with the following:

> spack load cp2k@8.1+mpi+openmp+plumed

I could instead load cp2k or cp2k@8.1 and Spack will check against all the builds and try find match. Right now both of these commands will succeed, since I only have one build. It is best to be more specific, so I specifically loaded cp2k version 8.1 with MPI, OpenMP & Plumed support.

> which cp2k
cp2k.popt        cp2k.psmp        cp2k_shell.psmp
> which cp2k.popt
/lustre/software/aarch64/spack/opt/spack/cray-cnl7-aarch64/gcc-9.2.0/cp2k-8.1-tnejptyboq56nppfikolzk5emc6vf5sb/bin/cp2k.popt

Thoughts

Getting started with spack took some time, mostly to wrap my head around the concepts of specs, variants, externals.. You should also be prepared to contend with build bugs, but that’s nothing new!

I found the Spack community to be responsive and helpful: https://github.com/spack/spack#community

Contributing bug fixes and issues is easy, they have provided commands to print the relevant debug info, logs and discover the package maintainers which allows you to directly ping them in Github.