Doing more with your data.
Most stuff in the command line environment either expects the user to input data or the software outputs some data to the command line. Or both. There is lots to be done with this data that can make life that little more simple.
MOVING THROUGH THE PIPES:
One of the more powerful features of bash is being able to have one program directly feed its output to another program. This is done with the | symbol.
'history' will output a list of commands that have been used in the past.
'grep' allows you to display only output that matches certain criteria.
'history | grep "echo" '
This command will output all of your previous commands straight to the grep program and not to the terminal. grep then starts looking for any lines fed to it that contain the word echo. If the line has a match then it will be printed to the terminal.
grep is case sensitive (unless you use -i) and can be very useful for searching for little bits of information from either large outputs or files. An example you have likely already seen is either:
'dmesg | grep XXXXX', to search for any system messages pertaining to a device XXXXX
'cat /var/log/Xorg.0.log | grep "(WW)" ' to look for any warnings output by X on start up.
'cat /var/log/Xorg.0.log | grep "(WW)" | grep "fonts" ' to look for any warnings concerning fonts.
grep can also allow multi-term search with 'grep -e "(WW)" -e "(EE)" ' as an example.
cat just dumps the content of a file onto the terminal.
THE SCENIC ROUTE
Something you may also have seen is:
'dmesg > error.log' this simple takes the output from dmesg and stores it in a file - good for easy uploading to forums...
Each time the command is run, the file is completely overwritten. Data can be appended to a file, so you don't lose the old stuff, with the following:
'dmesg >> error.log'
The '>' operator is very similar to '|', but is good for getting stuff INTO a file from a program. What if we have a file that we'd like to process with grep? We could use the above example of:
'cat /var/log/Xorg.0.log | grep "(WW)" ', or we could use an operator very similar to '>'
'grep "(WW)" < /var/log/Xorg.0.log' does pretty much the same.
ie, it tells the program to take its requirements for user input straight from that file and to leave you alone.
GOING OUT IN 3, 2, 1...
What if you have lots of output, but only want to see program errors listed? How about have the standard output go to one file and errors to another? That's simple too:
'ls * NOT_HERE' will list all files in the current directory, and will also make a special case of trying to list the file 'NOT_HERE'. Obviously you'll see two outputs:
the list, and the error.
'ls * NOT_HERE 1> /dev/null ' consigns the standard output (STDOUT) to the blackhole - /dev/null is where the naughty bytes are sent. Your output will read:
"ls: cannot access NOT_HERE: No such file or directory"
'ls * NOT_HERE 2> /dev/null ' consigns the standard error (STDERR) to oblivion.
'ls * NOT_HERE 1> the_list 2> the_errors' will place the list of files in "the_list" and all errors in "the_errors". 'cat' the files to see.
There are fancy tricks to be done with redirects, but that's for another Google, erm, day.
Taking media as an example we can form nice and useful scripts. Say I wanted to make one big playlist file based on all of my playlists (stored in /music/playlists/ )
'cat /music/playlists/* > all_my_music.m3u' this prints all the playlists in the directory and dumps them in the new file. Having a large music collection, this has one potential problem:
Every m3u file starts with "#EXTM3U". So this tag keeps popping up whenever a new playlist is added and appears an awful lot of times.
'cat /music/playlists/* | grep -v "#EXTM3U" > all_my_music.m3u' solves this (almost, it's now missing the opening line of "#EXTM3U", and I'm fussy). grep -v means find everything that does NOT match the expression.
'echo "#EXTM3U" > all_my_music.m3u; cat /music/playlists/* | grep -v "#EXTM3U" >> all_my_music.m3u' puts together a part from the first lesson, and a good amount from this one.
print #EXTM3U to the .m3u file (after deleting its contents), then append the processed output to the new file - note the >>.
This is obviously getting a little long. It can get longer - sed is another lovely little tool. It allows the changing of strings of characters, and could easily feature in the above to fix all your playlists that have the MS Windows style directory path (\) to the *nix style path (/)....
'echo "#EXTM3U" > all_my_music.m3u; cat /music/playlists/* | sed 's#\\#\/#g' | grep -v "#EXTM3U" >> all_my_music.m3u'
the following command:
sed 's#\\#\/#g' takes all \ and turns them to /. Why all the extra # and \ characters?
because sed 's/\\/\//g' can get hard to follow...
So, we have now seen how to do some very simple and basic stuff in bash. And, I'm NOT being sarcastic. I've barely scratched the surface of what bash can do and how complex and convoluted the commands can look. The next session while look at the more program related flow control (WHILE, IF/ELSE, FOR) etc. Woohoo! Simple programming at last.
Don't use sed -i until you really know what you're doing with it, or if you have a backup copy of the file you're editing.
ctrl-c, then blindly typing 'reset' and return will fix the following. ctrl-l (eL) just clears the terminal without fixing funky characters....