Useful shortcuts for unix system and Slurm workload manager

Over time to make my life easier, I have developed several shortcuts for convenience in execution and navigation in the Unix system and Slurm workload manager. I usually put these commands in separate files in a folder and then add that folder as an alias in .bashrc file.

To request compute nodes for a specified duration in hours from a list of partitions in HPC:

Save the following script in the file named ddnode_hr_core and copy the file to a folder named HPC_shortcuts in your home directory. Then, add export PATH=$PATH:~/shortcuts in your .bashrc file. Then, restart the terminal, type ddnode_hr_core hoursNeeded numberCPU in your terminal (here hoursNeeded numberCPU are the input - a numerical value), then, enter the corresponding number for the partition.

#!/bin/bash
echo "1- mecfd_q"echo "2- engineering_q"echo "3- engineering_long"echo "4- genacc_q"echo "5- backfill"echo "6- backfill2"read -p "Partition? " P
case $P in1) PARTITION="mecfd_q" ;;2) PARTITION="engineering_q" ;;3) PARTITION="engineering_long" ;;4) PARTITION="genacc_q" ;;5) PARTITION="backfill" ;;6) PARTITION="backfill2" ;;*) echo Invalid option! exit 1 ;;esacsrun --pty -t $1:00:00 -n $2 -p $PARTITION /bin/bash

This shortcut returns the directory of the job running in HPC:

Save this script in the file named jobDir, and copy it to the HPC_shortcuts folder as described above.

Command :

jobDir 123456789

Output:

/path/to/the/script/used/for/submitting/the/job

Script:

# Shortcut to the directory of the job running in slumr workload managerscontrol show job $1 | awk -F= '/Command=/{print $2}'

Find the age of the files sorted by last modified time:

This shortcut returns the latest 4 files sorted by "time modified" and the age of the file. Save the script in the file named lls and copy it to the shortcut folder.

Command:

lls

Output:

0:13:42 visitlog.py

2:07:42 visit0053.value.Z

2:07:42 visit0053.png

2:09:06 visit0052.value.Z

Script:

#!/bin/sh
for ifile in 2 3 4 5dofileName=$(ls -lt | grep -v '^d' | grep -v '^l' | awk '{print $9}' | head -n $ifile | tail -n 1)
#echo "$fileName"
time1=$(date +%s) time2=$(date -r $fileName +%s)i=$((time1-time2))#echo $delta#formatedTime=$(date -d $delta +%S:%M:%H)((sec=i%60, i/=60, min=i%60, hrs=i/60))timestamp=$(printf "%d:%02d:%02d" $hrs $min $sec)echo $timestamp " " $fileName#echo "$formatedTime"done

Useful for Mac: this shortcut removes the extra text pasted when copying a directory in Cyberduck

Extra step: add alias ccd="source ~/shortcuts/ccd.sh" in .bashrc file

# To make life easier with cyberduck copy directory optionURL=$1cd "${URL%/*}"

Shortcut to get the list of jobs running on the cluster by the current user in slurm workload manager

#!/bin/bash# Shortcut to get the jobs runningsqueue -u $USER -o "%.10i %.9P %.20j %.2t %.10M %.4D %.4C %R"

Calls sinfo command for the selected partition with some extra information (mecfd_q is the name of the partition in FSU RCC cluster)

#!/bin/bash
# sample output# NODELIST NODE STATE CPUS S:C:T MEMORY TMP_DISK WEIG AVAIL_FE REASON # hpc-m35-1-1 1 mixed 28 2:14:1 128341 20470 1 YEAR2017 none
echo "1- mecfd_q"echo "2- engineering_q"echo "3- engineering_long"echo "4- genacc_q"echo "5- backfill"echo "6- backfill2"echo "7- mecfd18_q"read -p "Partition? " P
case $P in1) PARTITION="mecfd_q" ;;2) PARTITION="engineering_q" ;;3) PARTITION="engineering_long" ;;4) PARTITION="genacc_q" ;;5) PARTITION="backfill" ;;6) PARTITION="backfill2" ;; 7) PARTITION="mecfd18_q" ;;*) echo Invalid option! exit 1 ;;esac
sinfo -p $PARTITION -N -o "%.12N %.5c %.12C %.15T %16E"

Shows the list of the jobs running in a specific partition

#!/bin/bash
echo "1- mecfd_q"echo "2- engineering_q"echo "3- engineering_long"echo "4- genacc_q"echo "5- backfill"echo "6- backfill2"echo "7- mecfd18_q"read -p "Partition? " P
case $P in1) PARTITION="mecfd_q" ;;2) PARTITION="engineering_q" ;;3) PARTITION="engineering_long" ;;4) PARTITION="genacc_q" ;;5) PARTITION="backfill" ;;6) PARTITION="backfill2" ;;7) PARTITION="mecfd18_q" ;; *) echo Invalid option! exit 1 ;;esac
squeue -p $PARTITION -o "%.18i %.12j %.8u %.2t %.10M %.6D %.6C %R"

Perform PWD and DATE commands in each sub-directory to list the last modified time of a specific file (binary-tahoe)

for d in ./*/; do (cd "$d" && echo "`date -r binary-tahoe` ${PWD##*/}"); done

Page updated

Google Sites

Report abuse