Installing software with EasyBuild as a user

Posted on: 14 March 2017
By: bbrydsoe

Installing software modules with EasyBuild as a user

In most cases HPC2N have already installed, or can install, the software you need, these instructions are for the rare occasions when HPC2N can't install something for reasons like licensing or similar.

The official documentation for EasyBuild also have instructions for how to do this, but we recommend that you use our documentation primarily and fall back on theirs when needed, since we have some simplifications that makes life easier.
Note! In particular if you have read the official EasyBuild instructions there will have been mention of a configuration file. When installing software as a user, you do NOT need an EasyBuild configuration file, and it can even cause problems to have one.

This guide will let you install software (and create a module for easy loading) either to your $HOME directory, or some other directory of your choice specified by the $PROJECT environment variable, under a subdirectory called easybuild. The enviroment variable $PROJECT can be used to install software into the projects storage directory tree, for instance, a project with a storage directory named "our-storage" would do:

export PROJECT=/proj/nobackup/our-storage

before following these instructions, and for ease of use add it to the respective users $HOME/.bashrc.
Below we we will mostly use $HOME in the text which can be replaced by $PROJECT when so desired.

After you have installed software with EasyBuild, your $HOME/easybuild, or your $PROJECT/easybuild, will contain:

ebfiles_repo - easybuild configuration files for software you have already built
modules - the modules for the software you have built
software - the software you have built
sources - sources tar files downloaded by the build

Installing software

Load the SoftInst module corresponding to whether you want to install into $HOME or $PROJECT.
1. For installing into $HOME use:
```
ml SoftInst/user
```
2. For installing into $PROJECT use:
```
ml SoftInst/project
```
You can check your EasyBuild config settings with one of these commands:
```
eb --show-config
eb --show-full-config
```
Check if the software you want exists as an EasyBuild software package. This can be done in two ways:
1. Go to the EasyBuild software list
2. Run the command
```
eb --search '^software'
```
  where software is the name of the software you are looking for.
Pick a toolchain to build with. You should always choose one that is installed on the system you are working on to build with, and not build a new one. If the software you want to install does not have an EasyBuild configuration for one of our exisiting toolchains, then try to build with one of them anyway, using the flag --try-toolchain=<toolchain> ,<version>. The newest of the installed versions is recommended. You can get a list of toolchains that the software in question has been successfully built with using either of the methods mentioned under 3. Look at
```
ml av
```
to see which toolchains are available on Kebnekaise.
Do a "dry-run" first to see if it will build or if there are errors:
```
eb <software-easybuild-config.eb> --dry-run
```
where <software-easybuild-config.eb> is the easybuild configure file you get from either the software list page or from running
```
eb --search '^software'
```
The format of the name of an EasyBuild configure file is <name-of-software>-<version>-<toolchain>-<version-of-toolchain>.eb.
1. Example: GPAW-21.6.0-foss-2021a.eb is for the software GPAW, version 21.6.0. The configuration file is for the toolchain foss, version 2021a.
If all goes well, install the software. Note that you should not give the full path to the software's easybuild configure file, only its name:
```
eb <software-easybuild-config.eb>
```
- In order to try with a different toolchain (e.g. if Kebnekaise does not have the one installed that the software defaults to), then do:
```
eb  <software-easybuild-config.eb> --try-toolchain=<toolchain>,<version>
```
- The software may have a number of prerequisites which needs to be installed as well. In order to do this automatically, do
```
eb  <software-easybuild-config.eb> --robot
```
  or
```
eb  <software-easybuild-config.eb> --try-toolchain=<toolchain>,<version> --robot
```
  to use a different toolchain than the default for the software.
If you get any errors, contact support@hpc2n.umu.se.
It will take some time to install the software, any prerequisites, and to create the module.
If all went well, you will now have a software module in $HOME/easybuild/modules which can be loaded, just like the system-wide installed modules. NOTE: You need to load the toolchain it was built with first.

Example, building GPAW with foss/2021a

Pick a toolchain from those offered on the above page with listed software. Or check which configurations exist for a specific software by using

eb --search '^gpaw'

Here we will do the latter:

b-an01 [~]$ eb --search '^gpaw'
== found valid index for /hpc2n/eb/software/EasyBuild/4.4.2/easybuild/easyconfigs, so using it...
 * /hpc2n/eb/custom/easyconfigs/g/GPAW/GPAW-0.11.0-customize.patch
 * /hpc2n/eb/custom/easyconfigs/g/GPAW/GPAW-0.11.0-intel-2017.01-Python-2.7.12.eb
 * /hpc2n/eb/custom/easyconfigs/g/GPAW/GPAW-0.11.0-intel-2017a-Python-2.7.13.eb
 * /hpc2n/eb/custom/easyconfigs/g/GPAW/GPAW-0.11.0-iomkl-2017.01-Python-2.7.12.eb
 * /hpc2n/eb/custom/easyconfigs/g/GPAW/GPAW-1.3.0-customize.patch
 * /hpc2n/eb/custom/easyconfigs/g/GPAW/GPAW-1.3.0-foss-2017b-Python-2.7.14.eb
 * /hpc2n/eb/custom/easyconfigs/g/GPAW/GPAW-1.3.0-foss-2019a-Python-2.7.15.eb
 * /hpc2n/eb/custom/easyconfigs/g/GPAW/GPAW-1.3.0-intel-2017b-Python-2.7.14.eb
 * /hpc2n/eb/custom/easyconfigs/g/GPAW/GPAW-1.3.0_numpy_lstsq_rcond.patch
 * /hpc2n/eb/custom/easyconfigs/g/GPAW/GPAW-1.4.0-foss-2018a-Python-3.6.4.eb
 * /hpc2n/eb/custom/easyconfigs/g/GPAW/GPAW-1.4.0-foss-2019a-Python-2.7.15.eb
 * /hpc2n/eb/custom/easyconfigs/g/GPAW/GPAW-1.5.2-foss-2019a-Python-3.7.2.eb
 * /hpc2n/eb/software/EasyBuild/4.4.2/easybuild/easyconfigs/g/GPAW-setups/GPAW-setups-0.8.7929.eb
 * /hpc2n/eb/software/EasyBuild/4.4.2/easybuild/easyconfigs/g/GPAW-setups/GPAW-setups-0.9.9672.eb
 * /hpc2n/eb/software/EasyBuild/4.4.2/easybuild/easyconfigs/g/GPAW-setups/GPAW-setups-0.9.11271.eb
 * /hpc2n/eb/software/EasyBuild/4.4.2/easybuild/easyconfigs/g/GPAW-setups/GPAW-setups-0.9.20000.eb
 * /hpc2n/eb/software/EasyBuild/4.4.2/easybuild/easyconfigs/g/GPAW/GPAW-0.9.0-blas-lapack-libs.patch
 * /hpc2n/eb/software/EasyBuild/4.4.2/easybuild/easyconfigs/g/GPAW/GPAW-1.4.0-customize.patch
 * /hpc2n/eb/software/EasyBuild/4.4.2/easybuild/easyconfigs/g/GPAW/GPAW-1.4.0-foss-2018b-Python-3.6.6.eb
 * /hpc2n/eb/software/EasyBuild/4.4.2/easybuild/easyconfigs/g/GPAW/GPAW-1.4.0-intel-2018b-Python-3.6.6.eb
 * /hpc2n/eb/software/EasyBuild/4.4.2/easybuild/easyconfigs/g/GPAW/GPAW-1.4.0-silence-numpy-warning-8e072ac8.patch
 * /hpc2n/eb/software/EasyBuild/4.4.2/easybuild/easyconfigs/g/GPAW/GPAW-19.8.1-customize-intel.patch
 * /hpc2n/eb/software/EasyBuild/4.4.2/easybuild/easyconfigs/g/GPAW/GPAW-19.8.1-foss-2018b-ASE-3.18.0-Python-3.6.6.eb
 * /hpc2n/eb/software/EasyBuild/4.4.2/easybuild/easyconfigs/g/GPAW/GPAW-19.8.1-foss-2019a-Python-3.7.2.eb
 * /hpc2n/eb/software/EasyBuild/4.4.2/easybuild/easyconfigs/g/GPAW/GPAW-19.8.1-intel-2018b-ASE-3.18.0-Python-3.6.6.eb
 * /hpc2n/eb/software/EasyBuild/4.4.2/easybuild/easyconfigs/g/GPAW/GPAW-19.8.1-intel-2019a-Python-3.7.2.eb
 * /hpc2n/eb/software/EasyBuild/4.4.2/easybuild/easyconfigs/g/GPAW/GPAW-20.1.0-Add-Easybuild-configuration-files.patch
 * /hpc2n/eb/software/EasyBuild/4.4.2/easybuild/easyconfigs/g/GPAW/GPAW-20.1.0-Wrap-pragma-omp-simd-in-ifdef-_OPENMP-blocks.patch
 * /hpc2n/eb/software/EasyBuild/4.4.2/easybuild/easyconfigs/g/GPAW/GPAW-20.1.0-foss-2019b-Python-3.7.4.eb
 * /hpc2n/eb/software/EasyBuild/4.4.2/easybuild/easyconfigs/g/GPAW/GPAW-20.1.0-intel-2019b-Python-3.7.4.eb
 * /hpc2n/eb/software/EasyBuild/4.4.2/easybuild/easyconfigs/g/GPAW/GPAW-20.10.0-foss-2019b-ASE-3.20.1-Python-3.7.4.eb
 * /hpc2n/eb/software/EasyBuild/4.4.2/easybuild/easyconfigs/g/GPAW/GPAW-20.10.0-foss-2020b.eb
 * /hpc2n/eb/software/EasyBuild/4.4.2/easybuild/easyconfigs/g/GPAW/GPAW-20.10.0-intel-2019b-ASE-3.20.1-Python-3.7.4.eb
 * /hpc2n/eb/software/EasyBuild/4.4.2/easybuild/easyconfigs/g/GPAW/GPAW-20.10.0-intel-2020b.eb
 * /hpc2n/eb/software/EasyBuild/4.4.2/easybuild/easyconfigs/g/GPAW/GPAW-20.10.0-test-mpi.patch
 * /hpc2n/eb/software/EasyBuild/4.4.2/easybuild/easyconfigs/g/GPAW/GPAW-21.1.0-foss-2020b-ASE-3.21.1.eb
 * /hpc2n/eb/software/EasyBuild/4.4.2/easybuild/easyconfigs/g/GPAW/GPAW-21.1.0-intel-2020b-ASE-3.21.1.eb
 * /hpc2n/eb/software/EasyBuild/4.4.2/easybuild/easyconfigs/g/GPAW/GPAW-21.6.0-foss-2021a.eb

We intend to build with foss/2021a. So let's pick GPAW-21.6.0-foss-2021a.eb

Let us take a look inside the configuration file (GPAW-21.6.0-foss-2021a.eb). The path is found from eb --search '^gpaw' and looking inside the file gives:

b-an01 [~]$ cat /hpc2n/eb/software/EasyBuild/4.4.2/easybuild/easyconfigs/g/GPAW/GPAW-21.6.0-foss-2021a.eb
easyblock = "PythonPackage"

name = 'GPAW'
version = '21.6.0'

homepage = 'https://wiki.fysik.dtu.dk/gpaw/'
description = """GPAW is a density-functional theory (DFT) Python code based on the projector-augmented wave (PAW)
 method and the atomic simulation environment (ASE). It uses real-space uniform grids and multigrid methods or
 atom-centered basis-functions."""

toolchain = {'name': 'foss', 'version': '2021a'}
toolchainopts = {'usempi': True, 'openmp': False}

source_urls = [PYPI_LOWER_SOURCE]
sources = [SOURCELOWER_TAR_GZ]
patches = [
    ('GPAW-20.1.0-Add-Easybuild-configuration-files.patch', 1),
]
checksums = [
    '5bb805bf514a7b04e3fdfce6f7864d150032badc5cd2805c57513af982d7a9cc',  # gpaw-21.6.0.tar.gz
    # GPAW-20.1.0-Add-Easybuild-configuration-files.patch
    'a12440bf63af70b891a63989b0f048bb8ebf4f60499020ea09259937f04cd042',
]

dependencies = [
    ('Python', '3.9.5'),
    ('SciPy-bundle', '2021.05'),
    ('ASE', '3.22.0'),
    ('libxc', '5.1.5'),
    ('libvdwxc', '0.4.0'),
    ('GPAW-setups', '0.9.20000', '', True),
]

prebuildopts = 'GPAW_CONFIG=doc/platforms/Linux/EasyBuild/config_foss.py'
preinstallopts = prebuildopts

download_dep_fail = True
use_pip = True
sanity_pip_check = True

sanity_check_paths = {
    'files': ['bin/gpaw%s' % x for x in ['', '-analyse-basis', '-basis', '-plot-parallel-timings',
                                         '-runscript', '-setup', '-upfplot']],
    'dirs': ['lib/python%(pyshortver)s/site-packages']
}

moduleclass = 'chem'

We're building with foss/2021a. Let us do a "dry-run" first to see if it will build or if there are errors, this will also list which prerequisites (dependencies) are already installed and which are missing:
```
eb GPAW-21.6.0-foss-2021a.eb --dry-run
```
No errors! Let's build!
We are building including any prerequisites:
```
eb GPAW-21.6.0-foss-2021a.eb --robot
```
When the software is installed and the module created, you can load it with:
```
ml foss/2021a
ml GPAW/21.6.0
```

Using the self-installed software

Your self-built software module will be located in $HOME/easybuild/modules/all and can be loaded, just like the system-wide installed modules.
Example for the GPAW we installed above:

ml foss/2021a
ml GPAW/21.6.0

If you install something that doesn't use a compiler toolchain it will end up in the Core module tree, and to use that you will need to explicitly tell the module system how to find them by doing: