Skip to content

Jupyter Notebooks

Jupyter Notebook is a web application for creating and sharing computational Python documents.

Homepage

This application is available in Open OnDemand.

We provide an interactive app for Jupyter Notebook using the Open OnDemand HPC web portal.

Run Jupyter Notebooks using Open OnDemand#

From the Open OnDemand dashboard, select Interactive Apps → Jupyter Notebook. You will see something similar to the following screen:

Jupyter Notebook Job Submission Interface

  • Set the desired parameters, and then click the Launch button.

  • Indicate the Python version and the path of the virtual environment. This application will initiate a session using a Conda environment on one or multiple nodes.

  • Python Version: Specify the desired Python version for your job execution. This can be specified as either a Python version (e.g., python/2, python/3) or an Anaconda version on the HPC system.

  • Path to Jupyter Virtual Environment: Enter a new or existing path to a Conda Jupyter Notebook environment

    • If such an environment does not exist at the given path, the system will generate one for your project. If you are building a new environment, you should specify a value of at least 8 for the number of cores. Your job will remain in Starting mode while the environment is configured. This can take up to 10 minutes.

Jupyter Notebook Job Submission Starting

  • If you are connecting to an already existing Jupyter Notebook environment, your environment will load immediately after your job starts.

Manually configuring a Jupyter Notebook environment#

You have the option of manually building and configuring a Jupyter Notebook environment, rather than using Open OnDemand to automatically build it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# Load the anaconda environment module
$ module load anaconda

# If you are on a compute node, you will also need to load the webproxy module (ignore this step if you're on a 
# login node)
$ module load webproxy

# Run the initialization step. Substitute your shell if you are using a shell besides bash.
$ conda init bash

# Re-load your runtime environment. Substitute the runtime configuration file for your shell if you are using a shell besides bash.
$ source ~/.bashrc

# You will now see '(base)' prepended before your prompt
(base) $ 

# Create the Jupyter notebook environment. Substitute your own name for 'my_jupyter_notebook'
(base) $ conda create -n my_jupyter_notebook

# Activate the Jupyter Notebook environment
(base) $ conda activate my_jupyter_notebook
(my_jupyter_notebook) $

# You can now use tools like _"conda install"_ to install packages in your environment. For example, if you want to install
# the JupyterLab IDE:
(my_jupyter_notebook) $ conda install jupyterlab

To run your newly created notebook on Open OnDemand (see above), take note of the path that the environment location: that was printed during the conda create output:

1
2
3
## Package Plan ##

  environment location: /gpfs/home/USERNAME/.conda/envs/my_jupyter_notebook

Enter this path into the Path to Jupyter Virtual Environment field in Open OnDemand:

Custom path to Jupyter notebook

Accessing Files Outside Your Home Directory#

By default, Jupyter sessions launched through Open OnDemand are restricted to the user's home directory. This limitation is due to Jupyter's internal configuration, which prevents access to directories outside the home path. Consequently, files stored in /gpfs/research or other locations outside your home directory cannot be directly accessed or browsed within Jupyter.

If you need to access your research storage from within Jupyter, you can create a symbolic link from your home directory to your research folder.

Workaround Steps#

  1. Open a terminal (either through Open OnDemand or SSH).
  2. Create a symbolic link from your research directory to your home directory using the following syntax:
1
$ ln -s /gpfs/research/<RESEARCH_VOLUME> /gpfs/home/<FSUID>/<LINKNAME>

For example:

1
ln -s /gpfs/research/scratch/abc25d /gpfs/home/abc25d/scratch

This creates a symbolic link called scratch in your home directory that points to /gpfs/research/scratch/abc25d.
Once created, you can navigate to scratch in Jupyter as if it were a normal folder, and access your research data.

Using R with Jupyter Notebooks#

Our Jupyter Notebook Open OnDemand app automatically loads "R" and installs the necessary dependencies to use R with Jupyter Notebooks.

Select the version of R that you would like to use on the job submission screen:

Select the version of "R" you would like to use with Jupyter Notebooks

Fill out the remainder of the form, and then follow the instructions above to submit your job. When the job starts, press the "New" button at the upper-right hand corner of the Jupyter notebook screen, and select "R":

Select the version of "R" you would like to use with Jupyter Notebooks

Manually install the "R" kernel in Jupyter#

If you need further customization, you can manually install the "R" kernel in your Jupyter Notebook environment.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
# Load requisite environment modules
$ module load anaconda R

# Activate the Conda environment for your existing Jupyter Notebook (see 'Manually configuring a Jupyter Notebook environment' above)
$ conda activate jupyterenv

# Run R after the Jupyter environment is active
(jupyterenv) $ R

# Install IRKernel by running the following command within R
> install.packages('IRkernel')
> IRkernel::installspec()

You can now run "R" in your Jupyter Notebook session.