Welcome! This workshop is designed for users that are new or reviewing basic Linux principles. This workshop also covers details on connecting to CS servers.

Note, the portal SSH login cluster servers are not GPU accelerated systems.

There are no network or VPN requirements for connecting to the SSH portal login cluster. These servers can be accessed on or off grounds, and do not require a VPN connection to access.

To SSH to the portal login cluster, from your personal computer, open a terminal and enter the following command, replacing <userid> with your computing ID.

Temporary passwords are sent via email when your CS account is created initially. After logging in for the first time, you will be required to change your password.

On Linux / MacOS, one application for connecting to portal via SSH is named Terminal

~$ ssh <userid>@portal.cs.virginia.edu

On Windows, two applications for connecting to portal via SSH are Command Prompt or PowerShell

C:\Users\username> ssh <userid>@portal.cs.virginia.edu

Note, for the first time you connect via SSH, you will see the following output

Are you sure you want to continue connecting (yes/no)?

Enter yes to continue

For more information, see our wiki page about our (Logging in via SSH)

Note, the NoMachine login cluster servers are not GPU accelerated systems.

There are no network or VPN requirements for connecting to the NoMachine desktop GUI cluster. These servers can be accessed on or off grounds, and do not require a VPN connection to access.

After downloading the NoMachine client application from the (NoMachine Website), open the NoMachine application and follow the steps listed below

1. Select Add
2. Under Address enter the following. Note, 'Name' can be anything
   1. Name: UVA CS Remote Linux Desktop
   2. Host: nxfront.cs.virginia.edu
   3. Port: 4000
   4. Protocol: NX
3. Select Add to complete the adding of the CS NoMachine host information
4. Double left-click, or right click to select “Start Connection”
5. Then ender your user credentials, replacing <userid> with your computing ID
   1. Username: <userid>
   2. Password: <cs password>
6. For the first time connecting, select New desktop
7. Select Create a new virtual desktop

Thereafter, you'll be prompted for a few configuration options that are user specific.

For more information, see our wiki page about our (NoMachine cluster)

Note, the RDP Windows server is not a GPU enabled system.

Network Requirements:

• If you are on grounds connected to the eduroam Wi-Fi, then you can connect directly
  • Eduroam information can be found here: https://in.virginia.edu/eduroam

• If you are off grounds, a UVA VPN is required to connect
  • UVA VPN information can be found here: https://in.virginia.edu/vpn

For windows systems, the application Remote Desktop Connection or RDP is installed by default. To connect to the remote desktop server

1. Ensure that you are either on grounds connected to eduroam or using a UVA VPN
2. Open the Remote Desktop Connection application
3. Select the Show Options drop down to expand the Logon settings
4. Enter the following information, note the domain name CSDOM\ that precedes your userid
   1. Computer: winsrv01
   2. User name: CSDOM\<userid>
5. Select Connect and enter your password

For more information, see our wiki page about our general purpose (Windows Server)

Remote development is possible for most common software. Generally, you simply need to enter your CS credentials to log in.

• VSCode (Windows, MacOS, Linux)
  • Instructions: https://code.visualstudio.com/docs/remote/ssh

• JetBrains (Windows, MacOS, Linux) * Instructions: https://www.jetbrains.com/remote-development/ For further reading about SSH connections, see our (SSH wiki page) For further reading about remote development, see our (Remote Development wiki page) For further reading about data transfer, see our (Data Transfer wiki page) —- —- ===== 1: Basic Linux Commands and Usage ===== ==== 1.1: Filesystem Navigation ==== Below are some example commands that can be run within a terminal over an SSH connection or using the application “Terminal” while using a NoMachine desktop GUI. Note, for all applicable examples, replace <path> with the location of a file or directory To print the current working directory (i.e. your current location) <code> ~$ pwd </code> To change/move to another directory <code> ~$ cd <path> </code> Note, the tilde character ~ specifies to use your home directory location. Further, simply using the command ~$ cd will move you to your home directory. The following are all equivalent to move to your home directory <code> ~$ cd OR ~$ cd ~ OR ~$ cd ~/ OR ~$ cd /u/<userid> </code> When using a dot . character, this specifies to reference relative to your current location (the current working directory) <code> ~$ cd ./my_directory </code> When using two dot .. characters, this specifies to reference the parent directory relative to your current location (the current working directory) <code> ~$ pwd /u/<userid>/my_directory ~$ cd ../ ~$ pwd /u/<userid> </code> The relative path is the location relative to the current working directory. For example, assuming the current working directory is /u/<userid>, and the directory /u/<userid>/my_directory exists, then either of the following can be used to change the working directory to my_directory <code> ~$ cd ./my_directory/ OR ~$ cd my_directory/ </code> The absolute path is the full path to a file or directory location, regardless of the current working directory. For example, if a directory with the path /u/<userid>/my_directory/ exists, and provided that the directory exists on the system, the following can be used regardless of the current working directory <code> ~$ cd /u/<userid>/my_directory/ </code> ==== 1.2: Listing Files and Directories ==== To list all files and directories under a given path <code> ~$ ls <path> ~$ ls cs_workshop/ dir0 file0 file1 </code> To list all files and directories in the current working directory <code> ~$ ls . </code> To list all files and directories of a directory relative to your current working directory <code> ~$ ls ./my_directory </code> To list all files and directories of a directory regardless of your current working directory, use the absolute path <code> ~$ ls /u/<userid>/my_directory </code> The following modifiers can also be applied to alter the output of an ls command To list one filename or directory on each output line <code> ~$ ls -1 ~/my_directory dir0 file0 file1 </code> To list one file or directory for each line, including file/directory permissions, ownership, and modified timestamps <code> ~$ ls -lh ~/my_directory total 10 drwx—— 2 <userid> <groupname> 2 Jun 11 00:01 dir0 -rw——- 1 <userid> <groupname> 4 Jun 11 00:02 file0 -rw——- 1 <userid> <groupname> 6 Jun 11 00:03 file1 </code> Note, the flag h is added to make the file and directory sizes print in human readable format. Without this, using only -l will show file and directory sizes in Bytes * The total is the disk usage of the listed files/directories in blocks. Each block is 1024 bytes * The first column is the file/directory permissions * The second column is the number of links to the file/directory * The third column is the owner of the file/directory * The fourth column is the group of the file/directory * The fifth column shows the size of the file/directory. For directories, this is not the sum of the contents * The sixth, seventh, and eighth columns show the modification date * The nineth column shows the file/directory name To list ALL files and directories, including hidden files or hidden directories <code> ~$ ls -al ~/cs_workshop/ total 23 drwx—— 3 <userid> <groupname> 6 Jun 11 00:01 . drwx–x— 36 <userid> <groupname> 68 Jun 11 00:01 .. drwx—— 2 <userid> <groupname> 2 Jun 11 00:01 dir0 -rw——- 1 <userid> <groupname> 4 Jun 11 00:02 file0 -rw——- 1 <userid> <groupname> 6 Jun 11 00:03 file1 -rw——- 1 <userid> <groupname> 7 Jun 11 00:05 .hidden_file0 </code> To list ALL files and directories, placing the most recent at the top <code> ~$ ls -alt ~/cs_workshop/ total 23 drwx–x— 36 <userid> <groupname> 68 Jun 11 00:01 .. -rw——- 1 <userid> <groupname> 7 Jun 11 00:05 .hidden_file0 -rw——- 1 <userid> <groupname> 6 Jun 11 00:03 file1 -rw——- 1 <userid> <groupname> 4 Jun 11 00:02 file0 drwx—— 3 <userid> <groupname> 6 Jun 11 00:01 . drwx—— 2 <userid> <groupname> 2 Jun 11 00:01 dir0 </code> To show information about a specific file or directory <code> ~$ stat <path> ~$ stat ~/cs_workshop </code> ==== 1.3: File and Directory Manipulation ==== Note, for all applicable examples, replace <src path> with the starting location or source, replace <dest path> with the destination location To create an empty directory <code> ~$ mkdir <path> ~$ mkdir ~/my_new_directory/ </code> To create an empty file <code> ~$ touch <path> ~$ touch ~/my_new_directory/my_new_file </code> To move an individual file <code> ~$ mv <src path> <dest path> ~$ mv ~/my_old_directory/my_file ~/my_new_directory/ </code> To rename a file <code> ~$ mv <path to current file> <path to new file> ~$ mv ~/my_new_directory/my_file ~/my_new_directory/new_filename </code> To move a directory and its contents, for example, moving the directory ~/java_programs/ into the directory ~/code/ <code> ~$ mv <src path> <dest path> ~$ mv ~/java/ ~/code_projects/ </code> To rename a directory <code> ~$ mv <path to current directory> <path to new directory> ~$ mv ~/my_old_directory ~/my_new_directory </code> To copy a file Note, if you specify the full path and/or a different file name, the file will copy to the path under the name specified <code> ~$ cp <src path> <dest path> preserve the filename ~$ cp ~/my_directory/my_file ~/my_other_directory/ ~$ cp ~/my_directory/my_file ~/my_new_directory/my_file change the filename ~$ cp ~/my_directory/my_old_file ~/my_directory/my_new_file </code> To copy a directory and its contents to another directory, include the recursive option -r <code> ~$ cp -r <src path> <dest path> ~$ ls ~/code_projects/ java python c ~$ cp -r ~/code_projects/ ~/code_submission/ ~$ ls ~/code_submission/ code_projects </code> To copy a directory's contents (not the directory itself), include the recursive option and a trailing /. at the end of the directory name <code> ~$ cp -r <src path>/. <dest path> ~$ ls ~/code_projects/ java python c ~$ cp -r ~/code_projects/. ~/code_submission/ ~$ ls ~/code_submission/ java python c </code> ==== 1.4 View File or Directory Sizes and Usage ==== To view the total usage of an individual file or directory <code> ~$ du -h <path> </code> To view all usage of files and directories, showing usage for each, in a given directory (i.e. your home directory) <code> ~$ du -h –max-depth=1 <path> ~$ du -h –max-depth=1 /u/<userid> </code> ===== 2: Processes and Output ===== A process in Linux is started when a command is entered. In other words, running a command such as ~$ ls -al ~/my_directory starts a short-lived process. When a command is entered to start a process, it is started in the foreground. Meaning, the terminal will wait until the command or process completes before becoming usable again. ==== 2.1: Stopping Processes ==== To stop a process in the foreground, hold the CTRL key, and select the C key (CTRL + C). On MacOS, this is the Command Key and . (Command Key + .). This example starts an infinite loop in the foreground, then CTRL + C is entered to stop the process <code> ~$ while true; do echo “42”; sleep 1; done 42 42 42 ^C </code> To stop a process in the background, using the respective PID number <code> ~$ kill <pid> </code> For example, to stop ./my_python_script.py which has a PID of 12344 <code> ~$ kill 12344 </code> A job number of a process can also be used. For example, to stop ./my_python_script.py which has a job number of 1 <code> ~$ kill %1 </code> Note, these examples send a signal to the process called a SIGTERM, or graceful stop. If a process doesn't stop from a SIGTERM, a forceful terminate signal SIGKILL can be sent using -9 in the kill command. Process cleanup does not take place when a SIGKILL is used. This is used as a last resort option, when a process isn't responding to a SIGTERM <code> ~$ kill -9 12344 </code> ==== 2.2: Moving Processes to the Background ==== At times, it can be helpful to run a process in the background instead of the foreground. The benefit here is that the terminal becomes usable while the process is running so that other commands may be entered or to be able to start additional processes. To launch a process in the background, without output redirection, append the & character to a command <code> ~$ ./<command or program> & </code> Alternatively, you can move a process from the foreground to the background by pressing CTRL + z <code> ~$ ./my_python_script.py ctrl + z, entered [1]+ Stopped ./my_python_script.py </code> Then, you can resume a process in the background <code> ~$ bg 1 [1]+ ./my_python_script.py </code> Note, without output redirection (described below), all output from background processes will be sent to either standard out (STDOUT), or standard error (STDERR), which is written to your terminal's foreground. ==== 2.3: View Processes ==== To list all running processes belonging to your user along with each respective Process ID (PID) number <code> ~$ ps -aux | grep <userid> <userid> 12344 0.3 0.4 19164 3552 pts/1 R+ 15:32 0:00 ./my_python_script.py <userid> 12345 0.0 0.0 19164 3552 pts/1 R+ 15:45 0:00 ps -aux <userid> 12346 0.0 0.0 19164 3552 pts/1 R+ 15:45 0:00 grep <userid> </code> Notice that there are two processes, one for the command ps -aux (PID=12345), and the other for grep <userid> (PID=12346). When a process has been moved to the background, you can list these as jobs as well <code> ~$ jobs [1]+ Running ./my_python_script.py </code> —- —- ===== 3: Editing Files from the Terminal ===== There are several options for editing files in a terminal. Two common text editors are nano and vim For example, to open a file for editing using nano <code> ~$ nano my_file </code> Note, vim is a powerful tool and has a learning curve, be sure to look up a reference cheat sheet To exit without saving from editing a file with vim, hold shift, then press the colon / semi-colon key :, then enter q! <code> ~$ vim my_file </code> There is a command named vimtutor that can be used for practicing and learning vim interactively <code> ~$ vimtutor </code> —- —- ===== 4: Resource Usage Monitoring ===== To view the resources you're using on a given system, you can either start a process in the background, or open a second terminal to the server that your process is running on to run some of the following commands listed below. ==== 4.1: CPU & Memory Monitoring ==== This command will show the respective CPU and Memory usage of your processes. Note, you can also replace the environment variable $USER with your userid <code> ~$ top -U $USER OR ~$ top -U <userid> </code> Another option is to list processes that you are using <code> ~$ ps -aux | grep <userid> </code> ==== 4.2: GPU Monitoring ==== Note, this section is for GPU enabled systems only. Login clusters such as portal and NoMachine do not have GPUs. The GPUs available in CS are manufactured by Nvidia only. The information below pertains specifically to Nvidia GPUs. The most common command for viewing GPU usage and information is <code> ~$ nvidia-smi </code> More in-depth output can be obtained using nvtop, which requires loading the software module <code> ~$ module load gcc ~$ module load nvtop ~$ nvtop </code> —- —- ===== 5: Specialized Commands Unique to CS ===== ==== 5.1: storagecheck (home or project directory usage) ==== Note, this data is updated hourly. To view home directory usage <code> ~$ storagecheck -h </code> To view project directory usage, enter the path to the project directory <code> ~$ storagecheck -p /p/<project directory name> </code> —- —- ===== 6: Advanced Linux Commands and Usage ===== This section contains information for more advanced options and usage for Linux. ==== 6.1 File and Directory Permissions ==== Files/directories stored in a HOME directory (/u) are assigned default permissions and rarely need to be modified. Files/directories stored in a PROJECT directory (/p) are assigned default permissions and rarely need to be modified. Files/directories stored in SCRATCH/VOLATILE storage (/bigtemp) are NOT assigned default permissions and may need to be modified. For a given file, we'll use example output from a stat command, omitting several details to focus on file permissions <code> ~$ stat my_file Access: (0700/-rwx——) Uid: (12345/ <userid>) Gid: (54321/<groupname>) </code> Note, the syntax for Access: (0700/-rwx------) shows both forms of a file or directory's permissions. The first '0' is for the file's SUID/SGID/Sticky bit. For the purpose of this explanation, we will ignore this part and focus on the last three digits. The first dash '-' shows if the referenced item is a file (denoted with a dash '-' character) or directory (denoted with a 'd' character). A linux file or directory has three octals to determine access. The three octals for access/permissions, using the above example, we'll separate each octal <code> rwx—— ⇒ rwx — — ⇒ rwx — — | | | The first octal | | is the OWNER's access, | | which is <userid> | | | | The second octal | is the GROUP's access, | members of <groupname> | | The third octal is for OTHER/EVERYONE else (global) access </code> File permissions may be represented using characters * r for Read (open a file) * w for Write (write to a file) * x for Execute (run a program or list a directory) Equivalently, file permissions octals may be represented numerically * 4 for Read (open a file) * 2 for Write (write to a file) * 1 for Execute (run a program or list a directory) * The resulting permissions are the sum of the allowed access Using the above example for my_file, rewritten using numeric values for the octals, we'll separate each octal <code> 700 ⇒ 7 0 0 ⇒ 7 0 0 | | | The OWNER (<userid>) has | | Read (4 or r), | | Write (2 or w), | | and Execute (1 or x) | | | | The GROUP <groupname> has | no access | | Other (everyone else) has no access </code> Using the above example for my_file, where the permissions are rwx------, we'll separate each octal <code> rwx—— ⇒ rwx — — ⇒ rwx — — | | | The OWNER (<userid>) has | | Read (4 or r), | | Write (2 or w), | | and Execute (1 or x) | | | | The GROUP <groupname> has | no access | | Other (everyone else) has no access </code> For another example, observe the file index.html <code> ~$ stat index.html Access: (0604/-rw—-r–) Uid: (12345/ <userid>) Gid: (54321/<groupname>) </code> Separating the octals, we can see that <code> ⇒ rw- — r– ⇒ rw- — r– | | | The OWNER (<userid>) has | | Read (4 or r), | | and Write (2 or w) | | | | The GROUP <groupname> has | no access | | Other (everyone else) has Read (4 or r) </code> ==== 6.2: Change File or Directory Permissions ==== To change the permissions of a respective file or directory, the form is to set each of the three respective octals using numeric values <code> ~$ chmod <permission octals> <path> set so that only the owner can read/write ~$ chmod 600 ~/my_file set so that only the owner can read/write/execute ~$ chmod 700 ~/my_program.py set so that only the owner and group members can read/write to a file ~$ chmod 660 /p/my_project/shared_file set so that the owner can read/write, and make it globally accessible (i.e. readable by a web server) ~$ chmod 604 ~/public_html/index.html </code> ==== 6.3: Redirect Output ==== At times, it can be helpful to redirect output of a command or process to a log file To redirect STDOUT (standard output) <code> ~$ <command or program> > ~/output.log ~$ ./a.out > ~/output.log </code> To redirect STDERR (standard error) <code> ~$ <command or program> 2> ~/error.log ~$ ./a.out 2> ~/error.log </code> To redirect both STDOUT and STDERR <code> ~$ <command or program> > ~/full_output.log 2>&1 ~$ ./my_python_script.py > ~/full_output.log 2>&1 </code> To suppress or ignore output, you can send either or both streams to /dev/null as shown <code> ~$ <command or program> > /dev/null 2>&1 ~$ ./my_python_script.py > /dev/null 2>&1 </code> —- —-