Python on the HPC
A very powerful, easy to use and object-oriented scripting language.
Python is a very powerful, easy to use, object-oriented scripting language. The language has a numerous packages that are designed for a myriad of different purposes.
Using Python on the HPC#
Currently, Python version 3 is available on the HPC:
The Python3 environment module is loaded by default upon login. If you need another version of Python (newer or older), use the module system to load it:
To use newer versions of Python, load the appropriate module:
If your code uses Python 2, you will need to load the appropriate module:
Python environments#
Our recommended way to use Python on the HPC is to create a custom environment using either virtualenv, Conda, or Anaconda. All three tools are available and pre-installed on the HPC. If you are doing data science work, you will probably need to use Anaconda or Conda. If you are unsure, refer to this LinkedIn article.
If you are confident that your Python script or app does not use any third-party libraries (i.e., you do not need to use pip
), you can use
the default system-wide Python environment.
Virtualenv#
The simplest way to create an isolated Python environment is to use virtualenv, which comes pre-packages with Python. Each environment contains a copy of all Python runtimes and libraries, and is isolated to its own directory. You can have as many virtual environments as you need.
The following example shows how to create a Python virtual environment named myapp in the default version of Python:
Creating a Virtualenv#
Activating a Virtualenv#
As you can see from the above, you need to specify the full path each time you execute a Python-related command in your
new Virtualenv. If you want to make your Virtualenv Python environment the default,
you can run source ~/myapp/bin/activate
:
To deactivate the environment, type deactivate
at the prompt:
Tip
If you know that you will be using your custom Python virtualenv exclusively, or most of the time, you can
autoload it upon login by appending the activate script to your .bashrc
file:
Conda#
To use Conda, our recommended platform for data science work, refer to our dedicated Conda page.
Jupyter Notebooks#
To use Jupyter Notebooks, our recommended development environment for Python collaboration, refer to our dedicated Jupyter Notebooks page
Python HPC Jobs#
You submit Python jobs to the HPC the same way you would with other jobs. Be sure to include the activation commands for any virtual environments or Conda environments.