External Storage: An Introduction

Storage Limits

As of July 7, 2015, we allocate 20GB storage for each user account. 

Users who are employees or affiliated with CRMDA may be granted access to additional resources. There are several folders under  /crmda, that may become accessible to users with various types of accounts.  Many, but not all user accounts, will be created with access to read and write in the folder /crmda/users. Only CRMDA employees will have access to /crmda/projects, /crmda/procedures, /crmda/archive.

The Basic Idea

Users want edit files in their own computers and transfer onto the cluster so they can run them. Programs generate output that has to be downloaded. 

There are 2 primary ways of thinking about the transfer process. 

1. A "file transfer program" uses a secure copy protocol (scp) to move files.

2. A remote file server can be "mounted" so it appears "as if" it is attached as a disk drive on the user's computer. (In CRMDA, the remote share "/crmda" appears as drive R:\, for example.

The ACF login node we recommend is login2.acf.ku.edu. The user's home directory (same as "folder"), as is customary in Linux systems, is /home/username. All users, whether they are staff or affiliates with CRMDA, have these home folders.


Directories worth Knowing:

/home/username

/crmda

Users who are employees or affiliated with CRMDA may be granted access to additional resources. There are several folders under  /crmda, that may become accessible to users with various types of accounts.  Many, but not all user accounts, will be created with access to read and write in the folder /crmda/users. Only CRMDA employees will have access to /crmda/projects, /crmda/procedures, /crmda/archive.

Within each home folder, there are some directories that are automatically created. One is /home/username/data. That's a separate place that is optimized for accessing large files.

On login2.acf.ku.edu, we also have many other folders, some programs, etc. One noteworthy folder is called

/crmda

Under /crmda, there are directories for users and workgroups.

The basic idea is that the material under /crmda should be

  * easily "shareable" among staff and team members

and

  * easily accessible OUTSIDE of the ACF system (probably via transfer.acf.ku.edu).


These storage devices exist behind a firewall and the staff has created a rubric through which users can access the storage. This note attempts to explain the many ways in which this can be done. As usual, I suggest you use what you like, and try to remember the other methods are available.

There are two protocols through which these shares can be accessed.

1. Secure Shell protocol (SSH)

2. Windows SMB (or "Samba") protocol

The user's files under /home/username are accessible only via the SSH protocol.

The files under /crmda can be accessed by either SSH or Samba protocols.

That means the user has many choices about how to access that storage, and the convenience of each type will depend on the kind of computer that the user has.  Below you will find help in both Windows and Linux systems. 


From Windows

Use a GUI file transfer program.

The only fully workable MS Windows program for interacting with ACF is WinSCP (the portable version works fine). At the current time, Filezilla is not able to change ownership of files and folders, but otherwise it works well and hopefully that capability will be added.


To see my crmda folder, for example, I start "Winscp"

 host:  login2.acf.ku.edu
  Username: pauljohn
  Password: *****
  Port: 22

It is important to choose the SCP connection option. That's the one that allows us to change ownership of files.

That will home directory on ACF. If you want to navigate to /crmda, click "up" (or two periods, "..") in the file system.

Try it and see!

This will transfer files without formally "mapping" the drive as a device in your system. It is faster, and makes "automatic synchronization" possible with a file transfer program like Unison (which exists on all platforms, as far as I know) or rsync.

A program like rsync has options that allow it to bring all the files on crmda "up to date" without doing a lot of extra work. It is not necessary to copy a whole directory if only one file is changed.

The program Unison is nice because it can compare 2 folders and make a two-way exchange to put both of them up to date. I use this to keep my course folders synced on several computers.

Mount a Shared Folder

We log in at "login2.acf.ku.edu", but there is a separate access point to attach shared directories, "transfer.acf.ku.edu." That offers the Windows CIFS protocol (formerly "SMB", or "Samba").

In Windows Explorer, find the "My Computer".  There's a "Map Network Drive" button. Choose a drive letter. Click on "Reconnect at logon" if you want the drive mapped permanently. DO NOT choose that option on a laptop computer if you are leaving our network. In the host box, type

\\transfer.acf.ku.edu\crmda\workgroups

\\transfer.acf.ku.edu\crmda\users\your-name-here

If you drop files into /crmda/users/username or /crmda/workgroups/groupname, then those new files will be owned by you, but the group for the files should be "crmda". That means if the group has read/execute permissions, others can see your stuff.

You can adjust permissions, it is not a complicated process, but no detailed instructions have yet been written.

Here's the thumbnail sketch.

Caution: the Windows Explorer "right click" menu does not interact perfectly well with file ownership and permissions settings on the ACF system. In order to change permissions or ownership of a folder or file, either it is necessary to 1) log into ACF with a terminal client and use commands like "chown" or "chmod" or 2) run the windows program WinSCP and connect to either login2.acf.ku.edu (or login2.acf.ku.edu, we hope) using SCP mode. That will allow a right-click menu that can change user and group ownership and permissions as well. We are working on a separate writeup about permissions and it will illustrate those steps.


From Linux

A GUI file transfer program

Free GUIs like Filezilla or gftp will work exactly the same way in Linux as in Windows.

The server that hosts the storage is either "transfer.acf.ku.edu" or "login2.acf.ku.edu". The reason to prefer transfer.acf.ku.edu is that it is a restricted purpose access point that is usually not crowded with users.

Use Gnome File Manager Nautilus to view & access files

There are many different "access protocols" and the ACF server--where the crmda shares and the linux user home folders exist, offers several of them.

If you are using a Gnome desktop, you have access to nautilus, the file manager. It is the thing that shows your home folders files when you choose "Places" in the top menu bar. You can also open nautilus by opening a terminal and typing

$ nautilus

By default, nautilus does not open a "URL bar" where you can type. But it does have a little icon that looks like a pencil and if you click that, a URL bar opens up. Even more recently, they have hidden that, but under the right side icon that looks like 3 dashes there is a menu item called "Enter location" that will make it possible to type an address. Recently, this syntax worked:

ssh://pauljohn@transfer.acf.ku.edu:22/home/pauljohn

That uses the "secure shell file system access" method, which I note below can also be accessed from the command line as "sshfs".


If you want to access the crmda via the "smb" (MS Windows) protocol, you could type this in the nautilus URL bar:

smb://transfer.acf.ku.edu/crmda


In my opinion, that is a bit strange, because you are using a Linux system to pretend it is a Windows system in order to get an imitation of a Windows service from a Linux server :). But whatever twists your wicket!

You can use the command line to browse the CRMDA remote storage shares in Linux.

These are more methods of pretending you are in a Windows system. If your server happens to only allow connections through the Windows Server SMB protocol, then it will be useful to understand how you can talk to it.

smbclient is an ftp-like interface for browsing Windows shares

  ex:  smbclient \\\\transfer.acf.ku.edu\\crmda -U HOME\\pauljohn

(Yes, all those \ are required, I can explain about 'escapes' some other time.) You will get an smb: \> prompt. Type help and you will get a command list.

From here you can use get and put and other ftp-like commands to transfer files. To be honest, I hardly ever use this strategy. I usually use smbclient simply to try to find out what Windows shares are offered on a system. I would add the option "-L" to list the resources. For example, to day I observed:

$ smbclient -L \\\\transfer.acf.ku.edu\\crmda -U HOME\\pauljohn

$ smbclient -L \\\\transfer.acf.ku.edu\\crmda -U HOME\\pauljohn
Enter HOME\pauljohn's password:
Domain=[HOME] OS=[Unix] Server=[Samba 3.6.23-14.el6_6]

        Sharename       Type      Comment
        ---------       ----      -------
        data            Disk      data directory
        scratch         Disk      scratch directory
        work            Disk      work directory
        research        Disk      research directory
        mai             Disk      MAI Cluster Storage
        cluster_data    Disk      MAI Cluster Storage
        crmda           Disk      Dropbox Directory
        IPC$            IPC       IPC Service (Transfer Server)
        pauljohn        Disk      Home Directory (/home/pauljohn)
Domain=[HOME] OS=[Unix] Server=[Samba 3.6.23-14.el6_6]

        Server               Comment
        ---------            -------
        TRANSFER-ACF         Transfer Server
        WIN

        Workgroup            Master
        ---------            -------
        HOME                 TRANSFER-ACF
        ITTCKU               HYDRA

Mount the share

You can also "fully" mount the CRMDA remote storage shares in Linux. This is the equivalent of "mapping a network drive" in Windows. The only difference is that instead of using a drive letter like D:\, in Linux we create a directory and use that as the mount point. For example, create a directory called "testmount"

 mkdir testmount

To mount the share on "testmount", this syntax is used:

  ex  mount -t cifs //transfer.acf.ku.edu/crmda testmount -o
       "user=pauljohn,domain=HOME"

Some systems have a "shortcut" command for mounting cifs

  ex:  mount.cifs //transfer.acf.ku.edu/crmda testmount -o
       "user=pauljohn,domain=HOME"

(Replace pauljohn with your username, of course. The first person who comes in and says their login to "pauljohn" failed is going to be the target of vicious ridicule.)

On some recent Linux distributions, that will be allowed only for the root user. That poses some frustrating problems. There are workarounds in the vfs (virtual file system) support available in many desktop environments. In the Thunar file manager of the XFCE desktop, for example, one can enter the URL

smb://transfer.acf.ku.edu/crmda

and an menu that asks for the user name and domain (which I enter as HOME\pauljohn and HOME). After that, the share is mounted. In the Gnome, this can be done with the file manager "Nautilus". Where it says network:///, (the address bar at the top), we want to put the address

smb://transfer.acf.ku.edu/crmda

Then a little panel pops up asking for a username, domain name, and password. The domain is HOME.

Now, in all honestly, I don't know that I would really want to mount it as a windows share under linux. The file server is a Linux system, so when you access it "as if" it were a windows share, the file system is getting translated from Linux to a Windows-compatible format, and then on your Linux system you are translating it back. Sometimes that changes file name capitalization, or at least it used to.

A Linux desktop system can access the files in a more direct way. I would just as soon mount it with a SSH file system with this command:

$ mkdir testmount

$ sshfs transfer.acf.ku.edu:/crmda/users/pauljohn testmount

That causes contents of /crmda/users/pauljohn to appear on my local system as a folder.

I do that exact same kind of access on my user home partition with

$ mkdir myhome

$ sshfs transfer.acf.ku.edu: myhome

If for some reason this fails, try to mount a share on login2.acf.ku.edu.  It makes the same storage available, but transfer.acf.ku.edu is usually faster.

When finished, to disconnect, it is necessary to remember the somewhat more complicated

$ fusermount -u myhome

$ fusermount -u testmount

Control Ownership And Permission

The instructions on Linux systems have a section on file ownership. I'd suggest you start by reviewing that material, in particular Linux File Permissions. That will explain how the permissions look within the ACF Linux system.

At the current time, the MS Windows Explorer (the common file manager) cannot set ownership or permissions on mounted ACF shares. It is necessary to use other means to change them.

Basically, files have "owners", and the rights are assigned to "groups" and "others". The owner should have rights to read, write, and execute. The group should default to "crmda", but that can be re-assigned. (It is necessary to ask the system administrators to create officially named user groups before they can be used.) If you want other users to be able to read your materials, the "other" group needs to have "read" permissions on files and directories must be both "read" and "execute." The "execute" bit is required for directory browsing.

As of Spring, 2011, the only program for MS windows that can assign ownership and adjust permissions is WinSCP, and that must be started in "SCP" mode.

In order to change owner or group information, it may be most convenient to log into ACF, navigate to the correct location, and run a command like this to change the ownership on a particular file:

$ chown myUserName.myGroupName fileName

That can also be applied to a directory, recursively:

$ chown -R myUserName.myGroupName dirName

To change a thing called "dirName" so that it is owned by "myUserName" and the group will be "myGroupName". The -R option has the effect of applying the change recursively.


CRMDA Calendar

Like us on Facebook
One of 34 U.S. public institutions in the prestigious Association of American Universities
Nearly $290 million in financial aid annually
44 nationally ranked graduate programs.
—U.S. News & World Report
Top 50 nationwide for size of library collection.
—ALA
23rd nationwide for service to veterans —"Best for Vets," Military Times
Equity & Diversity Calendar

KU Today