BASH (terminal use)

[Steady_Bear]

Many people underestimate the power of BASH. So, I thought I'd write a tutorial on getting something more from the bash shell to help bring more folks over to the dark side.
The scripting/programing capabilities aren't as powerful as Perl or similar languages, but often it can do the job just as well, and sometimes better.

Some quick ones for those not in the know:
Mouses have some limited functionality in bash. You can select some text with the left mouse button and paste with the middle button. Makes life easy.
shift-page up/down will allow you to scroll through previous lines of output text.
up/down arrow will allow you to recall previous commands (try 'history' ).
left/right arrows allows you to move through the command you are typing to edit mistakes etc.
hitting 'TAB' will either autocomplete the command or filename you are typing, or if there are multiple completions possible it will appear to do nothing - hitting it again will provide a list of possible completions.

That should conclude the more common shortcuts.

Stacking commands - ie typing a load of commands that you want to run one after the other, or all at once.

ALL TOGETHER NOW
'startx' will start the X server and give you pretty graphics. If you go back to the terminal that X was launched from (ctrl-alt-F1) you'll see it is still busy (ctrl-alt-F7 should get back to X).
'startx&' will start X, and instantly give you your terminal back for other use.
'startx& updatedb' will start X and at the same time start updating the system's database of files (use 'locate filename' to quickly find a file BTW, provided it is installed).

You'll notice when running 'startx&' there is a line printed with two separate numbers, these are 1) job id 2) process id. They aren't important right now.
'startx& updatedb' will result in the terminal being kept busy with updatedb running.
'startx& updatedb&' does exactly what you'd expect.

ORDERLY QUEUE PLEASE
'startx;' is exactly the same as 'startx'. The advantage comes in by allowing you to provide a first come, first serve list of commands.
'startx; updatedb' would start X up, do nothing whilst X is running, then start running updatedb as soon as you kill X. A better way of testing this is with the following:
'sleep 3; echo "Hello World" ' - sleep just tells the PC to sleep idly for a given number of seconds. echo just prints what you say to the terminal. Compare this to
'sleep 3& echo "Hello World" '

ACT ONLY ON SUCCESS
Sometimes you want the second command to run ONLY if the first command finished without error. This is possible with the double ampersand.
'mkdir ~/Pi_Rocks && echo "Directory Created" ' will create a directory called Pi_Rocks in the user's home directory (~ is shorthand for /home/username). Once the directory has been successfully created it prints 'Directory Created' to the terminal. Run it again and see what happens. You can't create a directory that already exists remember.

You aren't just limited to two commands in the list. Stack up as many as you need.


The above is a quick introduction to bash, and is far from complete. It provides some shortcuts to help speed up your day to day use on the command line and shows that there is potential to do lots more than people would think coming over from the good ol' days of DOS.
Later I'll cover how to bounce data about and between programs. Handling error outputs from programs (redirect to special files), and how to make sure that logging out doesn't cause the program you started running to die. All those commands people suggest to fix machines should be explained here.
Sometime after that we'll actually start looking at the programming under bash basics.

There is lots to learn.

[Steady_Bear]

Doing more with your data.
Most stuff in the command line environment either expects the user to input data or the software outputs some data to the command line. Or both. There is lots to be done with this data that can make life that little more simple.

MOVING THROUGH THE PIPES:
One of the more powerful features of bash is being able to have one program directly feed its output to another program. This is done with the | symbol.
'history' will output a list of commands that have been used in the past.
'grep' allows you to display only output that matches certain criteria.

'history | grep "echo" '
This command will output all of your previous commands straight to the grep program and not to the terminal. grep then starts looking for any lines fed to it that contain the word echo. If the line has a match then it will be printed to the terminal.

grep is case sensitive (unless you use -i) and can be very useful for searching for little bits of information from either large outputs or files. An example you have likely already seen is either:
'dmesg | grep XXXXX', to search for any system messages pertaining to a device XXXXX
'cat /var/log/Xorg.0.log | grep "(WW)" ' to look for any warnings output by X on start up.
'cat /var/log/Xorg.0.log | grep "(WW)" | grep "fonts" ' to look for any warnings concerning fonts.

grep can also allow multi-term search with 'grep -e "(WW)" -e "(EE)" ' as an example.
cat just dumps the content of a file onto the terminal.

Simples!


THE SCENIC ROUTE
MORE THAN
Something you may also have seen is:
'dmesg > error.log' this simple takes the output from dmesg and stores it in a file - good for easy uploading to forums...
Each time the command is run, the file is completely overwritten. Data can be appended to a file, so you don't lose the old stuff, with the following:
'dmesg >> error.log'

LESS THAN
The '>' operator is very similar to '|', but is good for getting stuff INTO a file from a program. What if we have a file that we'd like to process with grep? We could use the above example of:
'cat /var/log/Xorg.0.log | grep "(WW)" ', or we could use an operator very similar to '>'
'grep "(WW)" < /var/log/Xorg.0.log' does pretty much the same.

ie, it tells the program to take its requirements for user input straight from that file and to leave you alone.

GOING OUT IN 3, 2, 1...
What if you have lots of output, but only want to see program errors listed? How about have the standard output go to one file and errors to another? That's simple too:
'ls * NOT_HERE' will list all files in the current directory, and will also make a special case of trying to list the file 'NOT_HERE'. Obviously you'll see two outputs:
the list, and the error.
'ls * NOT_HERE 1> /dev/null ' consigns the standard output (STDOUT) to the blackhole - /dev/null is where the naughty bytes are sent. Your output will read:
"ls: cannot access NOT_HERE: No such file or directory"

'ls * NOT_HERE 2> /dev/null ' consigns the standard error (STDERR) to oblivion.

'ls * NOT_HERE 1> the_list 2> the_errors' will place the list of files in "the_list" and all errors in "the_errors". 'cat' the files to see.

There are fancy tricks to be done with redirects, but that's for another Google, erm, day.


Taking media as an example we can form nice and useful scripts. Say I wanted to make one big playlist file based on all of my playlists (stored in /music/playlists/ )
'cat /music/playlists/* > all_my_music.m3u' this prints all the playlists in the directory and dumps them in the new file. Having a large music collection, this has one potential problem:
Every m3u file starts with "#EXTM3U". So this tag keeps popping up whenever a new playlist is added and appears an awful lot of times.
'cat /music/playlists/* | grep -v "#EXTM3U" > all_my_music.m3u' solves this (almost, it's now missing the opening line of "#EXTM3U", and I'm fussy). grep -v means find everything that does NOT match the expression.

'echo "#EXTM3U" > all_my_music.m3u; cat /music/playlists/* | grep -v "#EXTM3U" >> all_my_music.m3u' puts together a part from the first lesson, and a good amount from this one.
print #EXTM3U to the .m3u file (after deleting its contents), then append the processed output to the new file - note the >>.

This is obviously getting a little long. It can get longer - sed is another lovely little tool. It allows the changing of strings of characters, and could easily feature in the above to fix all your playlists that have the MS Windows style directory path (\) to the *nix style path (/)....
'echo "#EXTM3U" > all_my_music.m3u; cat /music/playlists/* | sed 's#\\#\/#g' | grep -v "#EXTM3U" >> all_my_music.m3u'
the following command:
sed 's#\\#\/#g' takes all \ and turns them to /. Why all the extra # and \ characters?
because sed 's/\\/\//g' can get hard to follow... Simples right?


So, we have now seen how to do some very simple and basic stuff in bash. And, I'm NOT being sarcastic. I've barely scratched the surface of what bash can do and how complex and convoluted the commands can look. The next session while look at the more program related flow control (WHILE, IF/ELSE, FOR) etc. Woohoo! Simple programming at last.


Don't use sed -i until you really know what you're doing with it, or if you have a backup copy of the file you're editing.

ctrl-c, then blindly typing 'reset' and return will fix the following. ctrl-l (eL) just clears the terminal without fixing funky characters....
'cat /dev/random'

[dhardingham]

Brilliant post. Takes me back to the old days, when I was writing shell scripts every day and listening to 80's music. Those were the good old days

Of course, we were stuck with the Borne shell back then. Only real geeks played with other shells.

[williamhbell]

Hi,

A lot of us still use BASH. To help the local PhD students,

http://wbell.web.cern.ch/wbell/BashIntro/

was written. It scratches the surface on some issues, but might prove interesting to some. Personally, I end up chaining BASH, Perl and Python together. Here are some more eccentric examples,

run_number=$(echo $in_ds | tr '.' '\n' | grep -vP "[a-zA-Z]+")

Try using in_ds="something.12345.anotherThing987654" as input. Here is a typical example using Perl

ls -l * | perl -ne '@_=split(/\s+/, $_); print "$_[4] $_[8]\n";'

EOF is very useful too. For example, one can use it to embed Python in BASH

python << EOF
import math,string,ROOT
from ROOT import TFile,TH1F,MakeNullPointer
files = [ "$elMatched", "$elNotMatched", "$muMatched", "$muNotMatched" ]
names = [ "elMatched", "elNotMatched", "muMatched", "muNotMatched" ]
hList = []
for i in xrange(len(files)):
h = TH1F(names,";Isolation [GeV];Events",62,-0.250,30.750)
f = open(files, 'r')
ibin = 1 # underflow, bin 1,... , overflow
for line in f:
line = string.strip(line)
frags = string.split(line, ' ')
h.SetBinContent(ibin,float(frags[1]))
h.SetBinError(ibin,math.sqrt(float(frags[1])))
ibin = ibin + 1
f.close()
hList += [h]

f = TFile.Open("iso.root","RECREATE")
for h in hList:
f.WriteTObject(h)
f.Close()
EOF

In this case the BASH variables are substituted before the python script runs.

Regards,

Will

[andyl]

EOF is very useful too. For example, one can use it to embed Python in BASH

Yes - but the spaces got munched in that example so it won't run. Definitely a case for using the code button.

The powerful pipelining and processing ability of unix is only part of the story. When you use sortable and parse-able filenames (for example embedding dates in the filename in the format yyyyMMdd) and use wherever possible plain text files, you can do some very powerful things indeed.

[SN]

And its that spaces thing that makes python seem such a retrograde step to me - the inventor of python needs to be shot for such a faux pas because the rest of it isn't too bad? Sigh. . .

As an aside to bash and trad unix tools, I've yet to find a data processing problem i can't crack using awk - the grand-daddy of all filter/search/sort/transform tools.

[Steady_Bear]

Sorry for having been away a while and not continuing the tutorial on the more programming oriented side of Bash. Health (well, lack of) is getting in the way.

I'll get back to finishing bits in the fullness of time. Until then William's guide above has lots of information (more detailed than I intend to provide), so if you are eager to get on learning head over to the link above and have a read. If you're more patient and want to read my tutorial, then go read his anyway

[Joe Schmoe]

And its that spaces thing that makes python seem such a retrograde step to me - the inventor of python needs to be shot for such a faux pas because the rest of it isn't too bad? Sigh. . .

Totally agree. It seems obvious to anyone with any real world (read: End User Support, aka, "picking up the pieces") experience. It certainly seems to me that any time a Python program passes through any medium that screws up spacing (and there are many, both mechanical and social), the program is lost. Just as a starter, you can't post Python programs to most online forums (i.e., the common web-based ones, such as this one) since they routinely strip out spacing from the beginnings of lines. So, it seems clear to me that Python programs should always be uploaded as attachments (probably zipped first as well).

But yet the afficionados don't see any problem with it. I"m sure we'll be hearing from them by and by about how great it is and how it isn't really a problem.

As an aside to bash and trad unix tools, I've yet to find a data processing problem i can't crack using awk - the grand-daddy of all filter/search/sort/transform tools.

Totally agree with this, too. I use AWK every day.

[Adeianos]

Bash scripting is wonderful for automation. An example of something I wrote today to handle playlist generation for an icecast server that has a few people uploading a number of files, while taking turns between uploaders during the playlist, and adding an announcement MP3 of who selected the next song (added comments here to explain what everything does):
#!/bin/bash #hasbang
touch /tmp/faillist #creates the temp file where information will be stored
echo "" > /home/icecast/playlist # clears out the current playlist
count="1" #Set count to 1
echo "Beginning Songlist Generation" #user information
while true #for as long as true evaluates to true, do the following:
do
for uploader in `/bin/ls -A /home/icecast/media/` # take each directory in /home/icecast/media, and put it in the variable $uploader in turn.
do
song=`/bin/ls -A /home/icecast/media/$uploader/ | sed -n "$count"'p'` #set $song variable. the sed statment pulls only the line number specified in $count from the output
if [[ -f /home/icecast/media/$uploader/$song ]] # checks to determine if the target file is an existing regular file
then
echo "/home/icecast/$uploader.mp3" >> /home/icecast/playlist #add announcement MP3 to playlist
echo "/home/icecast/media/$uploader/$song" >> /home/icecast/playlist #add chosen song to playlist
else #if the song doesn't exist
failcheck=`grep $uploader /tmp/faillist | wc -l` #sets failcheck to be the number of times the user appears in failist
if [[ $failcheck == "1" ]] # if the user appears in faillist
then
continue #then skip to the next uploader
else
echo $uploader >> /tmp/faillist #else, since the file doesn't exist, the user doesn't have any more songs, so add them to faillist so only the other uploaders will be added
continue
fi
fi
done
let count=count+1 #we're done with this round of playlist additions, increase the count and continue
echo "Number of rotations="$count #show user number of song rotations so far
failcount=`cat /tmp/faillist | wc -l` #count how many users are in faillist
ulcount=`/bin/ls -A /home/icecast/media | wc -l` # count how many users there are total
if [[ $failcount == $ulcount ]] # if the number of failed users has reached the number of total users..
then
break #then finish out the original while loop
fi
done
rm -fv /tmp/faillist #remove the temp file as it's no longer needed