Site Tools


linux_environment_modules

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
linux_environment_modules [2018/02/14 21:16] rootlinux_environment_modules [2023/09/18 15:50] (current) – external edit 127.0.0.1
Line 1: Line 1:
 +===== Software Modules =====
 +[[http://modules.sourceforge.net|Software Modules]] make it easy for you to use software installed on our Linux systems. We provide a wide range of software across all our servers so that **you don't have to install software on your own.**  We also offer different versions of software, letting users choose which they want to use. 
 +
 +\\
 +
 +==== Virtual Environments ====
 +For most Python related packages (i.e. pip install ...), it is often more applicable to utilize a virtual environment. This can be done with [[ https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html | Anaconda]]
 +or [[ https://docs.python.org/3/library/venv.html | venv]]. The former requires loading an anaconda module, and the latter requires the python3 module.
 +
 +\\
 +
 +==== Jupyter Notebook ====
 +A Jupyter Notebook interface can be loaded using a virtual environment (as described above). This is best done using a [[ https://www.cs.virginia.edu/wiki/doku.php?id=nx_lab | NX virtual desktop]]. Once you've logged into your NX virtual desktop, load python3 or anaconda3 as a [[https://www.cs.virginia.edu/wiki/doku.php?id=linux_environment_modules#using_software_modules | software module]]:
 +<code>
 +[abc1de@labsrv03 ~]$ module load anaconda3
 +</code>
 +Then, load Jupyter Notebook:
 +<code>
 +[abc1de@labsrv03 ~]$ Jupyter Notebook
 +</code>
 +Follow the link given to you by the Notebook Server to access the interface:
 +<code>
 +[W 12:52:23.489 NotebookApp] Loading JupyterLab as a classic notebook (v6) extension.
 +[I 12:52:23.492 NotebookApp] Serving notebooks from local directory: /u/abc1de
 +[I 12:52:23.492 NotebookApp] Jupyter Notebook 6.5.4 is running at:
 +[I 12:52:23.492 NotebookApp] http://localhost:8888/?token=eb9345040d71bbeb1eddc75fb88504f98e25beb207907f6d
 +[I 12:52:23.492 NotebookApp]  or http://127.0.0.1:8888/?token=eb9345040d71bbeb1eddc75fb88504f98e25beb207907f6d
 +[I 12:52:23.492 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
 +[C 12:52:23.634 NotebookApp] 
 +    
 +    To access the notebook, open this file in a browser:
 +        file:///u/<youruserid>/.local/share/jupyter/runtime/nbserver-20880-open.html
 +    for example...
 +        file:///u/abc1de/.local/share/jupyter/runtime/nbserver-20880-open.html
 +</code>
 +
 +==== How Modules Work ====
 +Environment modules modify Environment Variables of the Linux Shell. Environment variables typically are paths to executables, version specifications, or terminal colors. For example, when an SSH session is initialized, variables are set such as the ''%%PATH%%'' variable for the location of various executable commands such as ''%%vim%%'', ''%%nano%%'', and ''%%ssh%%''.
 +
 +==== Using Software Modules ====
 +Type ''%%module avail%%'' to get a list of available software //(this list is just a small example, many more modules are available)//.
 +
 +<code>
 +[abc1de@portal04 ~]$ module avail
 +---- /sw/ubuntu/Modules/modulefiles ---
 +anaconda3       cuda-toolkit-11.7.0     cudnn-8.4.1     git-2.38.1 
 +java17-17.0.1   maven-3.8.3             nano-6.0        openmpi-4.1.4 
 +python3         ruby-3.0.3              vscode-1.61.2
 +</code>
 +
 +Load the software module using ''%%module load%% <module name>''. Now the software is ready to use.
 +
 +<code>
 +abc1de@portal04 ~ $ module load python3              # Load the module for use in your current session
 +abc1de@portal04 ~ $ which python3
 +/sw/ubuntu-22.04/python/current/bin/python3
 +abc1de@portal04 ~ $ python3                          # When you run `python` you get the version that has been loaded
 +Python 3.10.12 (main, Jun 14 2023, 10:18:49) [GCC 11.3.0] on linux
 +Type "help", "copyright", "credits" or "license" for more information.
 +>>>
 +</code>
 +
 +''%%module list%%'' will show what modules you have loaded.
 +
 +<code>
 +abc1de@portal ~ $ module load matlab
 +abc1de@portal ~ $ module load php
 +abc1de@portal ~ $ module list
 +Currently Loaded Modulefiles:
 +  1) matlab    2) python3   3) php
 +</code>
 +
 +''%%module display <module name>%%'' will show specific information about the environment variables setup by a module.
 +
 +<code>
 +abc1de@portal ~ $ module display python
 +-------------------------------------------------------------------
 +/sw/ubuntu/Modules/modulefiles/python:
 + 
 +module-whatis Python current 
 +prepend-path PATH /sw/ubuntu/python/current/bin 
 +prepend-path LD_LIBRARY_PATH /sw/ubuntu/python/current/lib 
 +prepend-path LD_INCLUDE_PATH /sw/ubuntu/python/current/include 
 +prepend-path MANPATH /sw/ubuntu/python/current/share/man 
 +-------------------------------------------------------------------
 +abc1de@portal ~ $ echo $LD_LIBRARY_PATH 
 +/sw/ubuntu/python/3.10.12/lib
 +</code>
 +
 +Use ''%%module switch <current module name> <new module name>%%'' to change one version of software for another and ''%%module unload <module name>%%'' to unload modules when you are done or no longer want this software.
 +
 +<code>
 +abc1de@portal ~ $ module switch python3 python2
 +abc1de@portal ~ $ module unload php
 +</code>
 +
 +==== Software Module Requests ====
 +If a module is needed but not listed, please send an email to cshelpdesk@virginia.edu with the following:\\
 +1. In the subject line, write "Software Module Request: // software name and the version //"\\
 +2. In the body of the email, please include information about the software being requested (version and any associated links)
 +
 +==== Shell Scripting/Slurm Batch ====
 +If you write shell scripts or slurm batch files that make use of these software packages, remember that you must still load modules from inside of your script.  For example, your script must contain the line ''%%module load python3%%'' before you can use python from inside of your script. 
 +==== Using modules in a non-interactive script ====
 +When executing a shell script non-interactively, initialization files are not loaded from /etc/profile.d. This prevents the initialization of the Environment Modules system and will prevent you from loading software modules.
 +
 +To fix this, simply include the following line in your scripts:
 +
 +<code>
 +source /etc/profile.d/modules.sh
 +</code>