Installing Python modules

Python modules can usually be installed through python, with the tool 'python setup.py', but some module can be difficult, and are easier to install in a stand-alone environment. In these cases we suggest using Virtualenv.

Installing with Virtualenv

Accessing Virtualenv

1. Using Virtualenv with HPC2N Python modules
   1. Virtualenv is installed with each of the loadable Python modules and is accessible after the Python module is loaded. It is highly recommended to use these versions of Virtualenv instead of installing yourself.
2. Load the module containing the Python version that you want to use, see the support pages for how to use the module system if you are unfamiliar with it.
3. It is advisable to also load any modules that provide any requirements for the python module you are going to install, this includes python-module requirements, like numpy/scipy (from SciPy-bundle) and many others. Using "ml spider python-module-name" will show if there is one installed at all, if there is one but not for the Python version you want to use, ask support@hpc2n.umu.se to install it, specifying which module and for which Python version.
4. Now you want to create your first virtual environment. Here I call it 'vpyenv', and put it in your Public, but you can call it anything, of course. Run the following to initialize the environment:
   

   virtualenv --system-site-packages $HOME/Public/vpyenv

Installing modules in Virtualenv

1. In order to install Python modules in the environment, you first need to activate it. Change directory to the environment you created before, and run:
   

   source bin/activate

   You can deactivate it with

   deactivate

2. It will now look like this (remember I called my environment 'vpyenv', and put it in my Public directory):
   

   (vpyenv)t-mn01 [~/Public/vpyenv]$

3. Load any modules that contain pre-installed dependencies for your software.
   1. You can also save that set of modules as a collection to make it easy to use the virtual environment later.
4. You can now install python modules like this (example, spacy):
   

   pip install --no-cache-dir --no-build-isolation spacy

   The module will be downloaded and installed. The "--no-cache-dir" option is required to avoid it from reusing earlier installations from the same user in a different environment. The "--no-build-isolation" is to make sure that it uses the loaded modules from the module system when building any Cython libraries.

Installing with setup.py

Aside from building and installing, you will usually need to set the correct environment before you can use the python module. There may be specifics to building a certain modules (including dependencies or possibly running a configuration script), so you should always check if there is an INSTALL or README file included with the module.

In general, python modules can be installed like this:

1. Download the python module and untar it.
2. Load any site modules needed (OpenMPI, Lapack, BLAS...). You should use the gcc-versions if such exist.
3. Also load any python-module requirements as per point 3 under "Accessing Virtualenv" above.
4. cd into the python modules source directory and run:
   

   python setup.py build

5. Install the python module. It is important that the installation directory is readable from batch jobs. A good choice is to install the python module(s) in $HOME/Public/python-modules, that way they are accessible from batch jobs and are easy to keep track of.
   

   python setup.py install --prefix=<path to your install directory>

6. Add this line to your ~/.bashrc (change as needed - set to where you have installed the module):

   export PYTHONPATH=$PYTHONPATH:<path to your install directory>/lib/python2.7/site-packages
   

7. Check the installation. You need to first open a new shell. Then launch python and type:
   

   import <python-module>