Accessing Tier-3 Resources through the Bastion Host

Accessing Tier-3 Resources through the Bastion Host

All connections to the internal user interface nodes and any other nodes must pass through the t3hop01.psi.ch gateway. This is done by using the gateway as a jump host. The gateway enforces multi factor authentication (MFA) by requiring your password + a code that is generated following the TOTP (Time-based One-Time Password) standard. TOTP code generators are now widely used and you will find such generators available as apps for your Android or Iphone. The TOTP generator gets initialized by sharing an initial secret with the service, often in form of a QR code that you can scan with your mobile phone's camera, or alternatively you can type in a key.

Obtaining test access for the new Tier-3 resources

Currently we require that you have an account for the old Tier-3 resources (i.e. for the pre-2025-May part of the system). The old system will be decommissioned and resources moved to the new system, as soon as the test phase is over. NOTE: The two old GPU systems are only reachable from the old Tier-3 UI and slurm, the new system only offers the new RH9 based multicore nodes.

Write a mail to cms-tier3@lists.psi.ch with the request to get access. We use this occasion to also revise the account validity and to which group users belong

first name:
last name:
group: (one out of Caminada / Canelli / deCosa / Dissertori / Kilminster / Wallny - tell us if we miss a group) 
tier-3 account name:
account expiration date:    (DD.MM.YYYY format. Date when you'll be leaving the research group)

After we vetted the correctness, you will receive a new password that is valid for a limited number of authentications and only for a limited duration. You will need to change it when you first log in. (Note: this will NOT replace your password for the old Tier-3 nodes - the old system is separate).

Resources in the new Tier-3 part

All systems are on RHEL9

t3ui05
t3wn80-90: Worker nodes with 2 x AMD EPYC 7763 64-Core, 512 GB RAM, 7 TB /scratch (RAID 0 over 2 NVMe)
new Slurm 24.11.3

we do not yet have any GPU nodes in the new cluster. We will move the old ones over once this cluster is tested when we decommission the old RHEL7 configuration.

Memory resource control is activated also on the new t3uiX nodes to prevent UI abuse. It may be that we will have to adapt for your usage, please provide feedback while testing should the memory restrictions prove too limiting.

Initial login to the gateway, changing the initial password and generating the MFA access credentials

You need to have your mobile phone with an application able to generate TOTP codes ready, so that you can set it up with your credentials.

ssh -o PreferredAuthentications=keyboard-interactive YOUR-T3-USER@t3hop01.psi.ch

You will see a menu that allows you to change your password and also to set up your MFA credentials.

Choose "change password" and go through the password renewal process.

Password quality enforcement requirement: 10 character minimal length, must contain at least one each of uppercase, lowercase, number, special characters. If your new password does not meet those criteria, you will need to repeat the process.

When you successfully managed to change your password, you will enter the Enroll for MFA credentials mode. This will display a QR code inside of the terminal that you can scan with your mobile's TOTP app. Alternatively you could also enter into your application the secret printed on the bottom of the page (equivalent to the QR code).

You will be asked for an initial code displayed by your mobile app to confirm whether it was initialized correctly with the secret. After this step, you're account is ready for MFA.

The MFA credential setup will only be offered if you have no MFA credentials yet. Otherwise, only a password change will be offered. If at a later point you need to re-initialize your MFA credentials (e.g. due to loss of the phone, etc.), you will have to contact the Tier-3 admins on the mailing list in order to get a reset.

Connecting to the user interface hosts by jumping through the bastion host connection

Here we show ssh commands with all options being given explicitly on the command line. It is easier to put the options into your ~/.ssh/config file, as shown in a later section of this document. You also jump there and make those changes immediately, it will save some typing.

To connect to the bastion host and configuring re-use of the connection (ControlMaster Option):

ssh -o PreferredAuthentications=keyboard-interactive -o ControlMaster=auto -o ControlPath=~/.ssh/connections/%h_%p_%r YOURUSER@t3hop01.psi.ch

NOTE: The ControlPath directory given must exist, you may have to create it! If your ~/.ssh/config file already contains such a definition, please re-use the setting that you made there. The directory is used by SSH to manage the current connections, it will place there one file for each existing master connection to a host, and it will remove that file after disconnect.

While the master connection to the bastion host stays open, you can jump through it to the Tier-3 user interfaces without having to re-authenticate to the bastion host. For jumping to a user interface, use an ssh command like the following.

ssh -J YOUR-T3-USER@t3hop01.psi.ch YOUR-T3-USER@t3ui05.psi.ch

Setting up SSH keys

You can place ssh keys on the user interface hosts, so while you will always have to do password + MFA code authentications to connect to the bastion host, connections to the UIs thereafter can be passwordless using the SSH keys.

If you already have an SSH keypair in your ~/.ssh/ directory you can re-use that (each keypair will show up as two files in the form of {KEYNAME} and {KEYNAME}.pub). If you have no keypair yet, you can create one. This command chooses a key type and strength appropriate for 2025 (ed25519).

ssh-keygen -t ed25519 -a 100 -f ~/.ssh/id_ed25519 -C "your_email@example.com"

If you require compatibility with old systems, you may create an RSA key instead, but RSA keys are considered unsafe.

ssh-keygen -t rsa -b 4096 -a 100 -f ~/.ssh/id_rsa -C "your_email@example.com"

In order to place the public part of the key on a target server, it is easiest to use the following kind of SSH command. The -i option allows to specify a specific key (your filename may differ).

ssh-copy-id -i ~/.ssh/id_ed25519.pub -o ProxyJump=YOUR-T3-USER@t3hop01.psi.ch YOUR-T3-USER@t3ui05.psi.ch

# or easier, if you already configured your ~/.ssh/config file
ssh-copy-id -i ~/.ssh/id_ed25519.pub t3ui05.psi.ch

After this, you will be authenticated by that key when connecting to the target server.

Configuring persistent options in the SSH config file for bastion and UI nodes

This is an example configuration that you can put into your ~/.ssh/config file. If your config file already contains configuration settings, you will need to correctly add the new host rules and make sure that the logic is consistent with your configuration.

Note that you will have to create the directory ~/.ssh/connections into which SSH will place files for each currently active connection (they get deleted after the connection is closed)

# Activate ControlMaster mode. All connections to a particular host will go through the first connection to that host, requiring
# only an authentication for that initial connection
ControlMaster auto
# Keep master connection open until 300s after closing 
ControlPersist 300
# Path for temporary files used by ControlMaster. The directory ~/.ssh/connections must exist!
ControlPath ~/.ssh/connections/%h_%p_%r

host t3hop01.psi.ch
     user YOUR-T3-USER
     PreferredAuthentications keyboard-interactive

host t3ui05.psi.ch
    user YOUR-T3-USER
    ProxyJump t3hop01.psi.ch
    # you must put here the path of the SSH key you chose (note, it's not the *.pub file, but the private one)
    IdentityFile ~/.ssh/id_ed25519

Explanations of some Options:

ControlPath: For each active connection under ControlMaster, there will be one file created in this directory, and it will be erased when the connection terminates.
PreferredAuthentications keyboard-interactive: The MFA authentication process needs this option to generate the additional prompt for the TOTP code.

These settings will allow you to just use simple ssh commands to correctly initiate the bastion connection and the UI connections without having to type the options every time.

ssh t3hop01.psi.ch
ssh t3ui05.psi.ch

Connection sharing for Windows clients using Putty

Consult the PSI general documentation

(Section: "SSH to one or more internal systems using “Connection sharing”) which describes how to set up connections to the hop.psi.ch system. Replace that hostname by t3hop01.psi.ch for the Tier-3 access.

-- DerekFeichtinger - 2025-05-13