VS Code remote setup

This is originally a partial fork of VS Code setup by Pedro Ribeiro, Programmer Analyst in the Department of Computational Biomedicine at Cedars-Sinai.

Main

VSCode and it's python packages have several useful features for developing code in any language, including R and Python. With it you can use remote terminals, git/github GUI for version control, markdown editors, debugging with breakoints, Python linting, jupyter notebooks, interactive Python code, etc all from the same application. I recommend checking out the following resources to get started:

Resources: Getting Started With Python in VSCode Python Debugging with VSCode Jupyter with VSCode Workspaces Version Control/Git

VSCode on the HPC via SSH

To use VSCode on the HPC, you need to install the ssh package. It can be easily found by just searching 'ssh' in the packages menu.
This should add a new icon to on the left menu bar. Clinking that icon will open up the ssh menu.
You must hover over the "SSH Targets" dropdown menu to reveal a + symbol.
Clicking that will open a menu where you can ssh into a remote node. This should be one of the submit nodes listed above. Type in ssh <username>@Csclprd3-s001v.csmc.edu. (The other submit nodes will also work)
After typing in your password, a window should open up that is hosted on the HPC submit node. From here you can directly edit files on the HPC, run files, utilize a terminal, etc. To open up to a specific folder or workspace you can go to "File - Open Folder". To view the terminal you can click "View - Terminal". You can open up as many terminals as you wish by clicking the + on the right hand side of the terminal menu.

I would recommend organizing your projects as VSCode workspaces as described here: Workspaces

In short they are folders where you can customize certain specifications.

VSCode ssh jump directly into compute node with Slurm

**Note!! This method should NOT be done on the old cluster, csclprd3. UGE does not support this feature and will not clean up your orphaned processes after you exit. **

VSCode allows you to ssh through a login node into another node within the cluster. This allows you to work interactively within a compute node without having to pipe several jupyter sessions around.

With Slurm on esplhpccompbio, users can request an interactive node using the salloc -t 8:00:00 command (with similar resource allocation flags as sbatch and srun). It is helpful to put a time limit (8 hours in this example) to make sure the interactive job closes when you are not using it if you forget to close it manually. Next, Slurm will print to your terminal the name of the allocated compute node. You can now ssh into this node to work interactively. When the job terminates, it should clean up any loose processes you may have launched. (The reason why this does not work with UGE is that it does not have a mechanism for closing these orphaned processes, leading to resources being comsumed and never freed)

The following steps describe how to ssh into a compute node via VSCode.

First, you have to edit the ~/.ssh/config file to add the compute node to your list of ssh targets. This can be found by clicking the Remote Explorer on the left menu of VSCode, then clicking the gears when hovering next to "SSH Targets"

These lines should already be present in the file if you have already ssh into the cluster previously. "Host" and "HostName" correspond to the name of the login node.

Host esplhpccompbio-lv01.csmc.edu
  HostName esplhpccompbio-lv01.csmc.edu
  User <username>

Next, add these lines into the same file. The "ProxyJump" option corresponds to the login node you would like to use, which should be included in the file in the format above. "Host" and "HostName" in this section will correspond to the name of the compute node.

Host <compute node>
  HostName esplhpc-cp014
  user <compute node>
  ProxyJump esplhpccompbio-lv01.csmc.edu

For example, if the allocated node were esplhpc-cp014

  HostName esplhpc-cp014
  user <username>
  ProxyJump esplhpccompbio-lv01.csmc.edu

You will have to edit this file with the name of the new compute node each time you are allocated a different one.

Finally, in "Remote Explorer" in VSCode, you should see a link to ssh into the compute node. You will be prompted for your password twice; once for the login node, and again for the compute node.

Note! As when using sbatch, resource allocation is not virtualized for your environment. It is important that you make sure not to use more than your requested resource allocation. For example, if you only requested 5 threads, make sure not to run python scripts with n_jobs set to -1 as that will use all the threads on the node rather than just what was allocated. Same goes for memory usage.

Jupyter Notebooks with VSCode

VScode will start a local jupyter session by default. To connect it to the interactive node you have to take the following steps:

Open up your .ipynb notebook file.
On the bottom right there is a link called "Jupyter Server: ...", click that. Alternatively search for "Jupyter: Specify a local or remote Jupyter server for connections" in the Command Palette (Cmd+Shift+P) or (ctrl+Shift+P).
Select "Existing"
Copy the localhost link from the previous section.
The bottom right indicator should now say "Jupyter Server: remote" to confirm you are on the compute node.
You can now select which environment to run in the top right on your notebook. It should say "Choose Kernel" if you have not chosen one yet, or the environment name along with the python version if you had. (Note that one kernel will be labeled remote, however as long as the indicator at the bottom says "Jupyter Server: remote", they will all be run remotely. This remote kernel just refers to the environment from which you ran the Jupyter session.)

VSCode also allows you to run any .py file interactively as if it were a jupyter notebook. You can separate the code into chunks with the line # %%. You can run each chunk individually with the "Run cell" in the GUI or with ctrl-Enter.

The same steps as before can be used to connect this window to the remote note. To confirm it is connected, run:

import socket
print(socket.gethostname())

If it did not connect correctly you can try setting the URI again. You may need to close the interactive window and/or reset the kernel as well. Running the jupyter command from the same environment as you want to use can help because sometimes VSCode has trouble switching environments on a remote session.

Useful Plugins

VSCode installs and uses plugins where it is hosted. When ssh into the HPC, it will not use the locally installed plugins, so you will need to reinstall whatever plugins you want to use. Click on the plugin to learn more about the features they offer.

Other Recommended Resources

PreviousJupyter Notebook NextVim setup

Last updated 1 year ago