Some programs, but not many, are available on all nodes, all the time. Most programs are made available by modules that users can load. Some modules are freely available to all cluster accounts, while some are restricted to individual user groups.
The modules that are loaded when you log in will depend on the settings in your account. Most likely, these settings will be placed in the "hidden" files ".bashrc" and ".bash_profile".
We ask for a list of currently loaded modules by running
$ module list
An account that follows the suggestions for a CRMDA account that are offered below will see the following.
$ module list
Currently Loaded Modules: 1) legacy 4) openmpi/1.10 7) icu/59.1 2) emacs/24.5 5) java/1.8.0_131 8) tcltk/8.6.6 3) compiler/gcc/6.3 6) xz/5.2.3 9) Rstats/3.3
Other useful module keywords
1. module avail
What else is available, and where is it found?
2. module spider <modulename>
Some modules are not listed in the "avail" output because they require us to load other modules first.
3. module load <modulename>
If there is only one module named <modulename>, that will be loaded. If module avail shows several versions, with names like Rstats/3.0, Rstats/3.1, then user is free to either specify a version or not. If no version is specified, the module system loads the newest (highest numbered) module.
4. module purge
removes all modules
5. module unload <modulename>
This will unload a module, and everything that it caused to be loaded. We used to have a module called "stats" that loaded a family of program modules, unloading had this effect:
$ module unload stats R is unloaded Rstudio is unloaded Rpackages is unloaded SAS is unloaded spss is unloaded stata is unloaded openmpi is unloaded emacs is unloaded
Where are the modules?
The system administrators have a family of modules. User groups, like CRMDA, are also allowed to create modules and make them available. The CRMDA modules are in a folder named "/panfs/pfs.local/work/crmda/tools/modules". In order to use our modules, one can run the following that adds our collection to the user environment:
$ module use /panfs/pfs.local/work/crmda/tools/modules
After running that command, then the CRMDA module folder is listed in the collection of module folders in the output from "module avail". The output will look something like this
$ module avail --------- /panfs/pfs.local/work/crmda/tools/modules --------- icu/59.1 openmpi/1.10.7 (D) java/1.8.0_131 (D) Rstats/3.3crc mplus/7.3 (D) Rstats/3.3 openmpi/1.6.5c Rstats/3.4 (D) openmpi/1.6.5 tcltk/8.6.6 openmpi/1.7.5 xz/5.2.3 ----------- /panfs/pfs.local/software_old/modules ----------- abacas/1.3.1 abaqus/2016 abyss/1.3.4 abyss/1.3.6 abyss/1.5.2 abyss/1.9.0 abyss/2.0.2 (D) [snip] ---- /panfs/pfs.local/software/install/modulefiles/Core ----- abundantotu/0.93b legacy (L) anaconda/4.3 matlab/r2016b annovar/2017Jun01 molden/5.7 bcftools/1.4.1 mplus/7.3_combo blast+/2.6.0 muscle/3.8.31 blast/2.2.26 namd/2.12_ibverbs breakdancer/1.4.5 namd/2.12_multicore (D) bwa/0.7.15 netcdf/4.1.1 canu/1.5 netcdf/4.5.0-rc1 (D) cmake/3.8.2 orca/18.104.22.168 (D) compiler/gcc/4.9 pear/0.9.10 compiler/gcc/5.4 perl/5.22 compiler/gcc/6.3 (D) picard/2.9.2 [snip]
Notice that the CRMDA module folder is at the top of the list, so if we give the same name to a module that is also in another directory, then our version is found first and it is used.
In case you worry "this might become too confusing, how will I remember to load the correct module paths?" we understand. In fact, we agree with the worry and we have a proposal to simplify this for CRMDA users. Please keep reading, we will explain everything.
What do modules do? Why do they work?
There some of the "never mind the man behind the curtain" magic going on here. We learned how to compile software into the folder "/panfs/pfs.local/work/crmda/tools" and we configure the files in the modules folder to know where they need to look. It just works, we hope.
If you really must know, here is an explanation. Suppose a program is compiled and installed in the Unix way, which means that everything for the program goes into a higher level folder, with subdirectories "bin" for the executable, "lib" for the shared library, "include" for header files, and so forth. The module framework adjusts the user's session (the 'environment') so that the "bin" folder is found in the user's path, and the "lib" folder is added to the LD_LIBRARY_PATH. When the user types the program's name, it is found as if it were installed in the larger operating system.
Consider, for example, our installation of tcltk, which is a required component in interactive use of R. In the folder "/panfs/pfs.local/work/crmda/tools/tcltk", one should see a folder where tk is installed, "8.6.6", and tcl (a scripting language) is in "tcl8.6.6". Inside those folders, you should see the subdirectories with names like "bin" and "lib". In the module tcltk, which loads both tcl and tk (look for /panfs/pfs.local/work/crmda/tools/modules/tcltk/8.6.6.lua), we see the tk bin and lib directories are added to the path.
Set the User Environment Correctly
We learned this lesson the hard way--failure. The full explanation of the situation is offered in this Timeline Blog posts Rstats/3.3 and Rstats/3.4 updates: dealing with OpenMPI and Infiniband library concerns and: R modules: Super Exciting New Updates. As it turned out, the modules we needed to run R parallel job were not being found automatically when cluster nodes were called to do calculations. And some nodes received the wrong instructions.
The fix was to adjust the settings in our user accounts, to control the environment. We need to edit both the user ".bashrc" and ".bash_profile" files. Two steps were necessary. First, allow modules in the CRMDA collection to be found,
module use /panfs/pfs.local/work/crmda/tools/modules
so that package load statements will succeed:
module load Rstats
Finally, due to some unexpected problems with the CRC cluster, we need to protect against errors that arise when users run R programs using the MPI framework. That piece looks like this:
OMPI_MCA_btl=^openib export OMPI_MCA_btl
This is necessary, unfortunately. We learned that the version of OpenMPI that is available in the cluster will cause all R programs to crash when they attempt to use the Infiniband communication devices that connect the nodes. We avoid that by inserting a statement in the user environment that prevents Infiniband devices from being used. This last change is technical in nature and we do not expect the R-using members of the CRMDA community to understand the details.
Because it seemed likely that users would make mistakes when inserting those elements into their .bashrc and .bash_profile files, we suggest an "easy" shortcut. This adds the CRMDA bin folder to the user's path, and then it runs a script that we prepare
export PATH=/panfs/pfs.local/work/crmda/tools/bin:$PATH source crmda_env.sh
That should handle everything. Users who want to keep fine-grained control are welcome to insert the commands from crmda_env.sh into their .bashrc and .bash_profile. The content of crmda_env.sh can be reviewed by inspecting it, just look in the folder "/panfs/pfs.local/work/crmda/tools/bin". At the current time, we do not plan to make this very fancy. There will be one file named "crmda_env.sh" that loads the settings that we believe are save, and there will be a "bleeding edge" version crmda_env-test.sh. Once we believe the test version is good, then "crmda_env.sh" will be renamed as "crmda_env-old.sh" and the test version will become "crmda_env.sh".