Python on the HPC

A very powerful, easy to use and object-oriented scripting language.

Homepage Version(s): 3.6.8, 3.10.4, 2.7.17

Python is a very powerful, easy to use, object-oriented scripting language. The language has a numerous packages that are designed for a myriad of different purposes.

Using Python on the HPC#

Currently, Python version 3 is available on the HPC:

1 2	`$ python --version Python 3.6.8`

The Python3 environment module is loaded by default upon login. If you need another version of Python (newer or older), use the module system to load it:

To use newer versions of Python, load the appropriate module:

$ module load python/3.10.4

The following have been reloaded with a version change:
  1) python/3 => python/3.10.4

$ python --version
Python 3.10.4

If your code uses Python 2, you will need to load the appropriate module:

$ module load python/2

The following have been reloaded with a version change:
  1) python/3.10.4 => python/2

$ python --version
Python 2.7.17

Python environments#

Our recommended way to use Python on the HPC is to create a custom environment using either virtualenv, Conda, or Anaconda. All three tools are available and pre-installed on the HPC. If you are doing data science work, you will probably need to use Anaconda or Conda. If you are unsure, refer to this LinkedIn article.

If you are confident that your Python script or app does not use any third-party libraries (i.e., you do not need to use pip), you can use the default system-wide Python environment.

Virtualenv#

The simplest way to create an isolated Python environment is to use virtualenv, which comes pre-packages with Python. Each environment contains a copy of all Python runtimes and libraries, and is isolated to its own directory. You can have as many virtual environments as you need.

The following example shows how to create a Python virtual environment named myapp in the default version of Python:

Creating a Virtualenv#

# Create a Python virtualenv named 'myapp'
$ virtualenv -p python ~/myapp

# Use pip3 to install packages (e.g., pycrypto)
~/myapp/bin/pip3 install pycrypto

# Run Python in virtual environment
~/myapp/bin/python3

Activating a Virtualenv#

As you can see from the above, you need to specify the full path each time you execute a Python-related command in your new Virtualenv. If you want to make your Virtualenv Python environment the default, you can run source ~/myapp/bin/activate:

# Make myapp Python commands the default
source ~/myapp/bin/activate

# You will know that the virtualenv is activated, because it will prepend the prompt 
(myapp) [~]$  

# Now, you can be sure that any commands reference the 'myapp' runtime
$ which python
~/myapp/bin/python

$ which pip
~/myapp/bin/pip

To deactivate the environment, type deactivate at the prompt:

1
2
3

(myapp) $ deactivate
$ which python
/usr/local/bin/python3/python

Tip

If you know that you will be using your custom Python virtualenv exclusively, or most of the time, you can autoload it upon login by appending the activate script to your .bashrc file:

1	`echo -e "\n\n# Activate myapp virtual environment\nsource ~/myapp/bin/activate" >> ~/.bashrc`

Conda#

To use Conda, our recommended platform for data science work, refer to our dedicated Conda page.

Jupyter Notebooks#

To use Jupyter Notebooks, our recommended development environment for Python collaboration, refer to our dedicated Jupyter Notebooks page

Python HPC Jobs#

You submit Python jobs to the HPC the same way you would with other jobs. Be sure to include the activation commands for any virtual environments or Conda environments.

#!/bin/bash

#SBATCH -n 1
#SBATCH -A genacc_q
#SBATCH -t 01:00:00
#SBATCH --mail-type=ALL

# Here, we load any environment modules we need (recommended)
module load python/3

# Here, we activate our virtual environment (substitute the appropriate conda command) 
source ~/myapp/bin/activate

# Here we perform the execution of our script
python3 my_python_script.py

Last update: July 7, 2023