A handy trick with batched processes on an HPC is that you can start an interactive session in a running session. Here’s an example where I needed to check if I was nearing the maximum allowed memory:
Here I list my active jobs to get the jobid
, run bash on that node, and list the processes by memory usage.
[daniel.kick@Atlas-login-1 BLUP_G]$ squeue -u daniel.kick
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
2491772 atlas BLUP-W daniel.k R 39:56 1 Atlas-0025
[daniel.kick@Atlas-login-1 BLUP_G]$ srun --pty --jobid 2491772 bash
[daniel.kick@Atlas-0025 BLUP_G]$ htop