Insights and Overviews about Linux/UNIX
Unix systems were developed in the 1970s, before the personal computer revolution began. Linus Torvalds created a version of the Unix kernel that could run on personal computers in the early 1990s. That is known as the Linux kernel today. Torvalds made the Linux kernel an open source, "free software" program which has since become the dominant operating system for web servers and high performance computing. (See Just for Fun: The Story of an Accidental Revolutionary, by Linus Torvalds and David Diamond). The open source movement wrapped the Linux kernel together with a battery of open source/free programs made available by the GNU Foundation and others, resulting in the GNU/Linux operating system, which most people that are not members of the GNU Foundation simply refer to as Linux.
The Linux operating system has several benefits. Linux is an open framework, it is not owned and dominated by a single corporation. There are Linux distributions that are available free of charge. Generally, Linux is more secure. It is easier to develop software in Linux, most programmers will probably agree.
It is now possible to order computers from the factory with the Linux operating system, but it is also easy to install Linux in place of an existing operating system. At the University of Kansas, the RedHat Linux distribution is used for web servers and enterprise computing. In the CRMDA labs, we run CentOS, a free variant of RedHat. Many of the users who maintain their own laptops prefer to use the Ubuntu Linux or Mint distributions. These may include some conveniences to users, especially newer video drivers.
If you are wondering "is Linux right for my computer?" you may need to spend some time to find out. In the CRMDA labs, we have several Linux desktop workstations that KU students and faculty can take for a test drive.
There is no denying the fact that your perspective on computers will change if you use Linux on a regular basis. Years ago, the head of a student user group called KULUA (Kansas University Linux User Association) circulated this cute essay about The Personal Evolution of a Windows User into a Linux user. That still fits the situation today. Also it meshes with the Famous Airplane Analogy about the transition from Windows to Linux. One political scientist who works in CRMDA wrote essay "Live Report From Linux Land" that was published in The Political Methodologist in 1997.
Log In and Explore in a Terminal
If you have a Linux workstation, sit down and log in. Otherwise, use a one of the remote desktop strategies we described. Connect to login2.acf.ku.edu, open a terminal, and run "qxlogin".
Type in some commands!
$ ls -la
$ mkdir i_dont_know_what_im_doing
$ cd i_dont_know_what_im_doing
$ echo "This is really interesting" > my_first_file.txt
If you want explanations of these things, please check our writeup on a user's first session on Advanced Computing Facility Linux cluster at KU. We also have a longer Linux command cheat sheet.
The Unix File System
The way files are stored has undergone dramatic changes since the 1990s. In Microsoft DOS and Windows 95, there were no permission frameworks. A three year old child could erase the whole hard drive. Malicious users could erase each other's homework. In those days, the Linux users pointed with ridicule at the seemingly ridiculous Win95 framework. When it introduced Windows 2000, Microsoft changed posture and introduced the NTFS file system. Users are allowed to own files and users are not allowed to quite-so-easily destroy the operating system. More importantly, like Linux, it protects user privacy by preventing one user from destroying files owned by another. The Macintosh, of course, is a BSD Unix system, and its file permission framework is essentially the same as Linux.
The convergence of the operating systems does not stop there. All of these systems name a directory for user folders and they expect users to keep their work in that confined space. Where the Linux and Unix tradition calls the main folder for user accounts "/home", Macintosh calls it "/user" and Windows calls it "C:\users".
Suppose we are in a terminal. Run this to move to the user's home directory
$ ls -la
$ ls /usr/bin
The full names of all files and directories begin with "/". The "/usr/bin" directory is the place where the system keeps most of its programs. "/home" is where user accounts are stored.
Your files are protected from other users, and the system's files are protected from you. The standard Unix/Linux system categorizes users into 3 groups: the individual User, the Groups of users in which each individual is included, and Others. We have described file permissions in a separate document.
The shell is the connector between what the user types and what the operating system hears. The most widely used Linux Shell is "BASH", which stands for Bourne-Again Shell. One the environment variables is the shell's name. You should check:
Many small Linux systems today (netbooks) use a smaller, lighter shell such as "dash".
Most shells supply tab completion. This is even true of the Windows Command Box, since Windows XP.
Start typing a long command or file name, then hit the TAB key. The system will try to finish typing the name for you. For example, type "/usr/bin/fi" and hit TAB. You may see a list of legal completions:
$ /usr/bin/fi fiascotopnm fig2ps file-roller findhyph fixfmps fixtpps fig2dev fig2ps2tex filezilla firefox fixpsditps fixwfwps fig2eps fig4latex filterdiff fitstopnm fixpspps fixwpps fig2mpdf file find fixcvsdiff fix-qdf fixwwps fig2pdf file-rename find2perl fixdlsrps fixscribeps
Add "re" and TAB again:
This is not only a good way to avoid tedious typing, it is also a GREAT way to avoid typographical errors in long file names. All you do is hit the magic button and the system will complete the command for you.
Shells remember: History
Shells have a history feature. They remember what you typed. Usually, you can go backwards by hitting the "up arrow", or possibly "Control up". No need to keep re-typing.
Run a Script
Type some commands into a file called "whatever.sh". Then "run" it by telling the system to do so, as in
$ sh ./whatever.sh
(to run in the generic "sh" shell). To run in the BASH shell,
$ bash ./whatever.sh
There are a number of scripting languages, we don't care to go into too much detail on that in this document.
There is a manual page for the BASH shell. From the command line, this
$ man bash
prints out a long page describing all of the options.
It may be important to know the following: users can have many shells open at once. Try this exercise. Open two different terminals. In each one, type some commands. In each one, use the "up arrow" to review the history of your commands. Those histories are separate.
On Unix/Linux systems, files are often packed with tar and compressed with the GNU gzip program. See Linux:CompressedFiles.
Graphical Programs on an X Server
If you log into a Unix/Linux system, and you are in the "console"--the place where you type in commands, then you have access only to the "command line interface." You type stuff in, see the results in letters and numbers.
If you need more, you need an X-Server to display graphical information. An X11-Server (or X, or X-Server) is a program that displays pretty pictures, boxes, menus, and such. If you are logged into a Unix/Linux system, and you run a program that requires graphics, it may fail unless you have an X-Server running.
If your personal computer is a Linux or Unix system, there is almost certainly an X11 server already running. The X11 server is the technology that drives the "desktop", it allows you to view windows and move them about. Your system may also be configured to automatically forward graphical programs from the remote system onto your desktop. If it is not, it will be necessary to tell the secure shell program that you want to allow X11 forwarding. That is done by changing the ssh login command from this:
ssh -X login2.acf.ku.edu
For MS Windows workstations, there are many X servers available. Xming Xserver » is a free (or inexpensive) is a free/inexpensive program that we have tested and found satisfactory. Its use is demonstrated by some multi-media presentations (yes, you can get them on youtube!). See the basic demonstrations in Xserver movie 1 » and movie 2 » . These two movies illustrate two different ways to trigger the X11 forwarding from the remote system onto the user's desktop.
You probably already have an X-Server running if you are logged into a Unix/Linux workstation. If you see pictures, icons, and stuff like that, you are seeing them on an X-Server's display, so that means you have an X-Server.
If you log into a remote Unix/Linux system with "ssh -X", and then you run a graphical program from that computer, its graphics will automatically get "projected" across the network onto the monitor that you are using. If you log in with Putty or some other client program, the graphics will not be automatically forwarded to your terminal, but it is pretty easy to set up. In Putty, for example, one can select pre-connection options, one of which is to enable X11 forwarding.
How Can You Edit Files on a Unix/Linux System
There are many good answers to this question. I am afraid to start giving too much information for fear it will make people think it is too bewildering. Nevertheless, check out EditingFiles.
Networking that All Smart People Understand
These days, we need to use "secure shell" (SSH) the programs to move files around. From the command line, the favorites are "rsync" and "scp". There are graphical interfaces for this such as "WinSCP" for Windows, or "Filezilla" for all platforms.
Use scp to copy a file on othersystem to the current location.
scp otherusername@othersystem:desired_file_name_name .
The "period" means "current directory". If you use the -r option with scp, it will copy recursively down through a directory structure.
The program rsync is a smarter version of scp. rsync can compare 2 directories and copy over the parts that are altered. This speeds up file transfers across long distances. That's how the R network CRAN is propagated from one system to another. The rsync program comes with a more-readable-than-average manual page (run "man rsync").
Where to Learn More?
When the Internet was young--before there was Google--a good cheat sheet was a precious commodity. Perhaps today, with the incredible power of search engines, there is not much use in keeping copies of bookmarks or documents.
On the off chance you are traditional, perhaps you will appreciate links to some documents we have found especially helpful.
- On DigitalOcean.com, there is a 4 part series Getting Started With Linux that is very nicely donw. It begins with "An Introduction to the Linux Terminal", and then surveys file management, permissions, and input/output.
- "Getting To Know Bash"
- The nixCraft Website, offering "Linux and Unix tutorials for new and seasoned sysadmin." Keep this linke handy: The Linux Shell Scripting Tutorial, by Vivek Gite.
- My friend Benedikt Stefansson put together some nice notes to help with the command line and BASH shell ».
If you want to be a script writer--and everybody will at some point--there are some important choices to make. One can write in the language of the shell itself, such as the generic shell scripting language that all shells agree to follow, or one can write using specialized functions in BASH that will only work on systems that have the BASH shell. People who write elaborate shell scripts are usually very determined and hard-nosed, they are about as close to the operating system as you can get. The bookstore, or library, will have handy manuals for this kind of work. In CRMDA, we have referred to Beginning Linux Programming by Neil Matthew and Richard Stones and Unix Power Tools by Jerry Peek, Tim O'Reilly, and Mike Loukides. There is a good page called Heiner's SHELLdorado for wanna-be scritpt writers ».
If a project is more elaborate, we would usually shift gears and use some more comprehensive, "high level" scripting language. In the 1990s, the leader for this was Perl, and it is still widely used. However, since 2012 or so, it appears the smart money is on Python. One advantage of Python is that it includes an interactive shell "interpreter" where the user can test out commands. A Silicon Valley recruiter told us in 2015 that a young programmer with a good knowledge of R and Python can "certainly make a living" these days. However, among the GRAs that have worked in the CRMDA, we notice a tendency to use the statistical programming packages as scripting languages. The R statistical framework has now absorbed almost all functionality that we rely on in Perl or Python. In fashion, they are saying "pink is the new black" ( or at least they were). In our lab, we say "R is the new Perl."