Differences

This shows you the differences between two versions of the page.

--- linux_environment_modules [2020/06/11 14:58]
pgh5a [Using Modules]
+++ linux_environment_modules [2023/04/12 17:40] (current)
@@ Line 1: / Line 1: @@
-====== Environment Modules ======
+===== Software Modules =====
+[[http://modules.sourceforge.net|Software Modules]] make it easy for you to use software installed on our Linux systems. We provide a wide range of software across all our servers so that **you don't have to install software on your own.**  We also offer different versions of software, letting users choose which they want to use.
-[[http://modules.sourceforge.net|Environment Modules]] are designed to manage custom package installations.  Software in our department is all installed to a network filesystem mounted over NFS where each package available is installed to it's own directory.  This allows us to manage a wide range of software across different Linux distributions without users having to deal with any configuration.  We can include different versions of software, letting users choose which they want to use.
+\\
-===== Available Modules =====
+==== Virtual Environments ====
+For most Python related packages (i.e. pip install ...), it is often more applicable to utilize a virtual environment. This can be done with [[ https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html | Anaconda]]
+or [[ https://docs.python.org/3/library/venv.html | venv]]. The former requires loading an anaconda module, and the latter requires the python3 module.
-Please see [[linux_modules:start|Available Modules]] for a listing of available software.
+\\
-===== Using Modules =====
+==== Jupyter Notebook ====
+A Jupyter Notebook interface can be loaded using a virtual environment (as described above). This is best done using a [[ https://www.cs.virginia.edu/wiki/doku.php?id=nx_lab | NX virtual desktop]]. Once you've logged into your NX virtual desktop, load python3 or anaconda3 as a [[https://www.cs.virginia.edu/wiki/doku.php?id=linux_environment_modules#using_software_modules | software module]]:
+<code>
+[labstudent@labsrv03 ~]$ module load anaconda3
+</code>
+Then, load Jupyter Notebook:
+<code>
+[labstudent@labsrv03 ~]$ Jupyter Notebook
+</code>
+Follow the link given to you by the Notebook Server to access the interface:
+<code>
+[W 12:52:23.489 NotebookApp] Loading JupyterLab as a classic notebook (v6) extension.
+[I 12:52:23.492 NotebookApp] Serving notebooks from local directory: /u/labstudent
+[I 12:52:23.492 NotebookApp] Jupyter Notebook 6.5.4 is running at:
+[I 12:52:23.492 NotebookApp] http://localhost:8888/?token=eb9345040d71bbeb1eddc75fb88504f98e25beb207907f6d
+[I 12:52:23.492 NotebookApp]  or http://127.0.0.1:8888/?token=eb9345040d71bbeb1eddc75fb88504f98e25beb207907f6d
+[I 12:52:23.492 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
+[C 12:52:23.634 NotebookApp]
+    To access the notebook, open this file in a browser:
+        file:///u/labstudent/.local/share/jupyter/runtime/nbserver-20880-open.html
+</code>
-On any Linux system, you can run ''%%module avail%%'' to get a list of available software.
+\\
+==== How Modules Work ====
+Environment modules modify Environment Variables of the Linux Shell. Environment variables typically are paths to executables, version specifications, or terminal colors. For example, when an SSH session is initialized, variables are set such as the ''%%PATH%%'' variable for the location of various executable commands such as ''%%vim%%'', ''%%nano%%'', and ''%%ssh%%''.
+To view environment variables currently set for your session, execute the ''%%printenv%%'' command:
 <code>
-[pgh5a@portal04 ~]$ module avail
+[abc1de@portal04 ~]$ printenv
-------------------------------- /sw/centos/Modules/systemModfiles -------------------------------------------------------------
+HOSTNAME=portal01
-slurm-client         slurm-client-testing
+... output omitted ...
+SHELL=/bin/bash
+PATH=/sw/centos/slurm/current/bin:/sw/centos-7.4/Modules/5.0.1/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
+PWD=/u/abc1de
+LANG=en_US.UTF-8
+MODULEPATH=/sw/centos/Modules/modulefiles
+HOME=/u/abc1de
+... output omitted ...
+</code>
---------------------------------- /sw/centos/Modules/modulefiles --------------------------------------------------------------
+\\
-afl               cmake-3.10        cudnn-7.3.1       gdb-8.1           libav             netdata           php7              ruby
-altera_pro        cmake-3.15        cudnn-7.5.0       ghex              libtorch          nios2eds          php7.1.10         ruby2.5.1
-anaconda3         cs6620            dmd               git               libtorch-1.3.1    nvtop             python            sbt
-anaconda3-2019.10 cuda-toolkit      dmd-2.088.0       gradle            lua               openmpi           python2           scala
-antlr             cuda-toolkit-10   doxygen           hashcat           lua-5.3.4         openmpi2.1.5      python2.7.15      sqlite3
-atom              cuda-toolkit-10.1 eclipse           java              lxd               openmpi4.0.0      python3           storm
-boost             cuda-toolkit-8.0  emacs             java11            modelsim_ae       openmpi4.0.2      python3.6.2       svn
-boost-1.67.0      cuda-toolkit-9.0  fio               java8             modelsim_ase      parallel          python3.8.0       torch
-clang-llvm        cuda-toolkit-9.2  gcc               java8u161         nano              perl              qsys              verilator
-clang-llvm-6.0.0  cudnn             gcc-5.5.0         java9             nccl              perl-5.26         quartus
-clang-llvm-7.1.0  cudnn-7.0.5       gcc-6.3.0         java9.0.4         nccl-10.0         perl-6            R
-cmake             cudnn-7.1.1       gcc-7.1.0         lammps            nccl-9.2          php               rstudio
-------------------------------- /sw/linux-any/Modules/modulefiles -------------------------------------------------------------
+==== Software Module Requests ====
-ant               cplex             cplex-opl         dmd-2.088.0       ripgrep           scons-3.0.1       vscode
+If a module is needed but not listed, please send an email to cshelpdesk@virginia.edu with the following:\\
-antlr             cplex-concert     cplex-studio-all  go                sbt               tmux
+. In the subject line, write "Software Module Request: // software name and the version //"\\
-apktool           cplex-cpoptimizer dmd               matlab            scons             tmux-2.7
+. In the body of the email, please include information about the software being requested (version and any associated links)
+\\
+==== Using Software Modules ====
+Type ''%%module avail%%'' to get a list of available software //(this list is just an example, more are available)//.
+<code>
+[abc1de@portal04 ~]$ module avail
+---- /sw/centos/Modules/modulefiles ---
+anaconda3       cuda-toolkit-11.7.0     cudnn-8.4.1     git-2.38.1
+java17-17.0.1   maven-3.8.3             nano-6.0        openmpi-4.1.4
+python3-3.9.9   ruby-3.0.3              vscode-1.61.2
 </code>
-Load the software module using''%%module load%%''.  Now the software is ready to use.
+Load the software module using ''%%module load%% <module name>''. Now the software is ready to use.
 <code>
-pgh5a@portal04 ~ $ module load python3              # Load the module for use in your current session
+abc1de@portal04 ~ $ module load python3              # Load the module for use in your current session
-pgh5a@portal04 ~ $ which python
+abc1de@portal04 ~ $ which python
 /sw/centos/python/3.6.2/bin/python
-pgh5a@portal04 ~ $ python                           # When you run `python` you get the version that has been loaded
+abc1de@portal04 ~ $ python                           # When you run `python` you get the version that has been loaded
 Python 3.6.2 (default, Oct  6 2017, 13:03:05)
 [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)] on linux
@@ Line 53: / Line 88: @@
 <code>
-pgh5a@portal ~ $ module load matlab
+abc1de@portal ~ $ module load matlab
-pgh5a@portal ~ $ module load php
+abc1de@portal ~ $ module load php
-pgh5a@portal ~ $ module list
+abc1de@portal ~ $ module list
 Currently Loaded Modulefiles:
 ) matlab    2) python3   3) php
 </code>
-''%%module display%%'' will show specific information about the environment variables setup by a module.
+''%%module display <module name>%%'' will show specific information about the environment variables setup by a module.
 <code>
-pgh5a@portal ~ $ module display python
+abc1de@portal ~ $ module display python
 -------------------------------------------------------------------
 /sw/centos/Modules/modulefiles/python:
@@ Line 73: / Line 108: @@
 prepend-path	 MANPATH /sw/centos/python/current/share/man
 -------------------------------------------------------------------
-pgh5a@portal ~ $ echo $LD_LIBRARY_PATH
+abc1de@portal ~ $ echo $LD_LIBRARY_PATH
 /sw/centos/python/3.6.2/lib
 </code>
-Use ''%%module switch%%'' to change one version of software for another and ''%%module unload%%'' to unload modules when you are done or no longer want this software.
+Use ''%%module switch <current module name> <new module name>%%'' to change one version of software for another and ''%%module unload <module name>%%'' to unload modules when you are done or no longer want this software.
 <code>
-pgh5a@portal ~ $ module switch python3 python2
+abc1de@portal ~ $ module switch python3 python2
-pgh5a@portal ~ $ module unload php
+abc1de@portal ~ $ module unload php
 </code>
-===== Things to keep in mind =====
+\\
 ==== Shell Scripting/Slurm Batch ====
+If you write shell scripts or slurm batch files that make use of these software packages, remember that you must still load modules from inside of your script.  For example, your script must contain the line ''%%module load python3%%'' before you can use python from inside of your script.
-If you write shell scripts or slurm batch files that make use of these software packages, remember that you must still load modules from inside of your script.  For example, your script must contain the line ''%%module load python3%%'' before you can use python from inside of your script.
-=== Initialize Modules System ===
-To make modules work from inside SLURM or scripts, you must including the following line in your script **before** loading any modules:
-<code bash>
-source /etc/profile.d/modules.sh
-</code>