Skip to content

Python on the HPC

A very powerful, easy to use and object-oriented scripting language.

Homepage Version(s): 3.6.8, 2.7.18

Python is a very powerful, easy-to-use, object-oriented scripting language. The language has a numerous packages that are designed for a myriad of different purposes.

Using Python on the HPC#

Currently, Python version 3 is available on the HPC:

1
2
$ python --version
Python 3.6.8

The Python3 environment module is loaded by default upon login. If you need another version of Python (newer or older), use the module system to load it:

Using newer versions of Python on the HPC#

The RCC includes the most recent Python version from the default vendor-provided package repository (3.6.8 at the time of writing).

If you need a newer version of Python, we recommend using Conda to install it. Below is a brief example of installing the latest version (3.12 at the time of writing).

Warning

The below example assumes that you have already configured your shell environment for Conda. If you haven't, refer to our initial setup guide before running the below example.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
# Create a new conda environment
(base) [USER@h22-login-26 ~]$ conda create -n my_python_app

# Activate the new conda environment
(base) [USER@h22-login-26 ~]$ conda activate my_python_app

# Install Python
(my_python_app) [USER@h22-login-26 ~]$ conda install python

# Follow the prompts...

# Check Python version
(my_python_app) [USER@h22-login-26 ~]$ python --version

If you need a specific version rather than the latest, you can create a Conda environment using the following syntax:

1
2
3
4
5
6
7
8
# Create the new Python environment using a Specific Python version
(base) [USER@h22-login-26 ~]$ conda create -n my_python_app python=3.8

# Follow the prompts...

(base) [USER@h22-login-26 ~]$ conda activate my_python_app
(my_python_app) [USER@h22-login-26 ~]$ python --version
Python 3.8.18

For more information about Conda environments, refer to our Conda documentation.

Using Python v2 on the HPC#

If your code uses Python 2, you will need to load the appropriate module:

1
2
3
4
5
6
7
$ module load python/2

The following have been reloaded with a version change:
  1) python/3.6.8 => python/2

$ python --version
Python 2.7.17

Python environments#

Our recommended way to use Python on the HPC is to create a custom environment using either virtualenv, Conda, or Anaconda. All three tools are available and pre-installed on the HPC. If you are doing data science work, you will probably need to use Anaconda or Conda. If you are unsure, refer to this LinkedIn article.

If you are confident that your Python script or app does not use any third-party libraries (i.e., you do not need to use pip), you can use the default system-wide Python environment.

Virtualenv#

The simplest way to create an isolated Python environment is to use virtualenv, which comes pre-packages with Python. Each environment contains a copy of all Python runtimes and libraries, and is isolated to its own directory. You can have as many virtual environments as you need.

The following example shows how to create a Python virtual environment named myapp in the default version of Python:

Creating a Virtualenv#

1
2
3
4
5
6
7
8
# Create a Python virtualenv named 'myapp'
$ python -m venv ~/myapp

# Use pip3 to install packages (e.g., pycrypto)
~/myapp/bin/pip3 install pycrypto

# Run Python in virtual environment
~/myapp/bin/python3

Activating a Virtualenv#

As you can see from the above, you need to specify the full path each time you execute a Python-related command in your new Virtualenv. If you want to make your Virtualenv Python environment the default, you can run source ~/myapp/bin/activate:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
# Make myapp Python commands the default
source ~/myapp/bin/activate

# You will know that the virtualenv is activated, because it will prepend the prompt 
(myapp) [~]$  

# Now, you can be sure that any commands reference the 'myapp' runtime
$ which python
~/myapp/bin/python

$ which pip
~/myapp/bin/pip

To deactivate the environment, type deactivate at the prompt:

1
2
3
(myapp) $ deactivate
$ which python
/usr/local/bin/python3/python

Tip

If you know that you will be using your custom Python virtualenv exclusively, or most of the time, you can autoload it upon login by appending the activate script to your .bashrc file: 

1
echo -e "\n\n# Activate myapp virtual environment\nsource ~/myapp/bin/activate" >> ~/.bashrc

Conda#

To use Conda, our recommended platform for data science work, refer to our dedicated Conda page.

Jupyter Notebooks#

To use Jupyter Notebooks, our recommended development environment for Python collaboration, refer to our dedicated Jupyter Notebooks page

Python HPC Jobs#

You submit Python jobs to the HPC the same way you would with other jobs. Be sure to include the activation commands for any virtual environments or Conda environments.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
#!/bin/bash

#SBATCH -n 1
#SBATCH -A genacc_q
#SBATCH -t 01:00:00
#SBATCH --mail-type=ALL

# Here, we load any environment modules we need (recommended)
module load python/3

# Here, we activate our virtual environment (substitute the appropriate conda command) 
source ~/myapp/bin/activate

# Here we perform the execution of our script
python3 my_python_script.py