Linux Tips

Kevin Cole
Gallaudet Research Institute
kjcole@gri.gallaudet.edu
Copyright © 2000

[This document is designed for a browser that can handle color and the two special characters sometimes referred to as French quotes « (not to be confused with <<) & » (not to be confused with >>). However, I hope the document will also be somewhat readable from browsers which do not have these features, such as lynx.]

The following tips are ones that I've become tired of looking up in the man and info pages, then whittling down to their barest essences.


  1. Redirecting Standard Error to Standard Out
  2. Making a patch file
  3. Copying directory trees
  4. Getting landscape output
  5. Verifying Red Hat packages
  6. Security: Watching the watchers...
  7. Stripping comments:
  8. Verifying all RPM's
  9. Viewing post-installation RPM scripts
  10. What are we listening to?
  11. Finding duplicate files with identical contents
  12. Listing DNS stuff
  13. Who's been sleeping in *MY* bed?
  14. Clip-clip. Taking care of really LONG lines
  15. Find and delete
  16. Formatting and using a floppy
  17. Mounting an NSF device
  18. Searching for files and manipulating them
  19. Starting a remote X windows program on a local screen
  20. Making a boot floppy
  21. Switching parallel from printer to ZIP
  22. More fun with RPM's
  23. Burning CD's
  24. Turning off NetQUE broadcasts
  25. Checking out from CVS
  26. Streaming with Icecast and Darwin
  27. Pretty-printing code as web pages
  28. md5sum
  29. Slow Hand
  30. Synchronizing with rsync
  31. Exploring binary RPM's without installing them
  32. Renaming all files in a directory
  33. "rpm -qil" in Debian
  34. Handling NULL's in Postgresql
  35. Using BitTorrent with RedHat
  36. Decoding base-64 encoded text

1. Redirecting Standard Error to Standard Out

The proper way to redirect stderr is to first decide where stdout is going, and THEN redirect the stderr to stdout. So, for example:

«yada-yada» | «pager» 2>&1

takes the output of «yada-yada» and pipes it into «pager», then tells the system to send stderr to wherever stdout is going. (A pager is a program that allows a user to scroll through long documents. I use one called most, but older, commonly used ones include more and less.

Back to the top


2. Making a patch file

Assume we have a Red Hat source RPM...

  1. rpm -Uvh «package».src.rpm
  2. cd /usr/src/redhat/SPECS
  3. rpm -bp «package».spec
  4. cd /usr/src/redhat/BUILD
  5. mv «package» «package».orig
  6. cd /usr/src/redhat/SPECS
  7. rpm -bp «package».spec
  8. cd /usr/src/redhat/BUILD
  9. (edit to your heart's content)
  10. diff -Naur «package».orig «package» > ../SOURCES/«package».patch
  11. rm -rf «package».orig
  12. cd /usr/src/redhat/SPECS
  13. emacs «package».spec
    ...
    Source: ...
    Patch: «package».patch
    ...
    %prep
    %setup
    %patch -p 1
  14. rpm -ba «package».spec

The idea is to create two directories with identical contents, then modify one of them. Create a diff file of the changes and save it. I THINK I got all the basic steps in there... However, it may be necessary to create the tarball too, in which case you need something like:

tar czvf «archive».tar.gz «directory_to_archive»

Back to the top


3. Copying directory trees

Often, it becomes necessary to copy entire directory trees from one directory to another. The method I saw somewhere was:

cd «olddir» ; tar cf - . | (cd «newdir» ; tar xpf -)

This creates a tarball that never actually becomes a file. The tarball is piped directly to a little script subroutine which changes to the new directory, and untars the tarball on the fly.

According to JP Abgrall, there's an optimized way to do this:

tar cf - -C «olddir» . | tar xpf - -C «newdir»

Back to the top


4. Getting landscape output

mpage -1lvH «filename» | lpr

The man page suggests that there's a way to pass pr switches to mpage (using -p instead of -H), but I've been unable to pass the line-numbering switch -n together with pr into mpage. (What I want is the headings a la -H together with line numbering.)

A fancier way, requiring a bit more study, is to use enscript. In fact, enscript is neat for LOTS of stuff -- customizable layout, "Page X of Y", line numbers, etc.

Back to the top


5. Verifying Red Hat packages

You can verify each package with the following command:

rpm --checksig

If you only wish to verify that each package has not been corrupted or tampered with, examine only the md5sum with the following command:

rpm --checksig --nopgp

Back to the top


6. Security: Watching the watchers...

The netstat command is a handy tool for seeing who's poking around at any given moment.

netstat -v | most

Back to the top


7. Stripping comments:

Assuming you have a script that uses the number sign (a.k.a. pound symbol, hash mark) as a comment character, and you wish to examine only those lines containing "active" commands and options, the following will produce such a listing:

grep -v "^\#" «scriptfile» | grep -v "^[[:space:]]*$"

What's happening: The file is first stripped of lines beginning with #. Then, that result is stripped of any lines which have 0 or more whitespace characters (and nothing else) between the start and end of the line.

Back to the top


8. Verifying all RPM's

Here's a small script that constructs a list of all package names sans version numbers, then feeds the list to rpm with the verify option. It also echoes the package name. NOTE THE QUOTES: Some are quotes, NONE are apostrophes, and some are backquotes (a.k.a. accents grave).

for i in `rpm -qa --queryformat "%{NAME}\n" | sort`
do
  echo $i":" >>verify.log 2>&1
  rpm --verify $i >>verify.log 2>&1
done

The stuff enclosed in backquotes gets run as a script within a script, and the output of that is fed to the outer script. (See man eval and other stuff about evaluating.)

Back to the top


9. Viewing post-installation RPM scripts

Occasionally, after the files are dropped onto the system, hither and yon, RPM will run a script embedded in the package file. It's nice to see what the squirrels are doing under the hood.

rpm -q --scripts «package-name»

Back to the top


10. What are we listening to?

lsof shows which processes are listening on a given port. (Actually it stands for "list open files", which shows what files are currently open.) lsof -i will list which ports are open on the machine.

lsof -i

Back to the top


11. Finding duplicate files with identical contents

There's probably a better way, but this worked for me:

diff -qrs «directory1» «directory2» 2>&1 | grep "identical$" > «unedited-shell-script.sh»

Then edit the file «unedited-shell-script.sh» to your heart's content.

Back to the top


12. Listing DNS stuff

Lots of different ways to do this, but I like:

nslookup
> ls -d gallaudet.edu

Back to the top


13. Who's been sleeping in *MY* bed?

Here's a more informative way to use the last command:

last -adf «wtmp_file»

Back to the top


14. Clip-clip. Taking care of really LONG lines

Lots of times, we only need to see the beginning of lines in a file to determine something. (For example, a subroutine that takes a single string argument may have a really long string literal.) To see just the first 100 characters on a line use the cut command. Like so:

cut -b -100 «FY2000.sql» | land

(land is an alias I've set up to print a file in landscape orientation using enscript command -- a very nice printing program.)


15. Find and delete

Sometimes it's nice to do something (like delete) a bunch of files based on a searchable criteria, e.g. portion of the filename, size, date, etc. Here's how:

find . -empty -exec rm -v {} \;

This is a specific example that searches for empty files and directories from the current working directory down, and then it deletes them. The important parts are the {} which gets replaced with whatever has been found, and the \;, an escaped semicolon indicating the end of the command to be "exec'ed". (And the -v is the ubiquitous "verbose" option, to tell you what's happening.)

A better approach, I'm told, is:

find . -empty | xargs rm -v

Back to the top


16. Formatting and using a floppy

Without mounting anything, just pop a floppy in the drive and...

fdformat /dev/fd0H1440
mke2fs /dev/fd0H1440
mount /dev/fd0 /mnt/floppy
...
umount /mnt/floppy

Back to the top


17. Mounting an NSF device

mount «server»:«remote_directory» /«local_mount_point» -t nfs -o ro

Back to the top


18. Searching for files and manipulating them

To find files in or beneath the current directory, of type "file", of size 800 KB or greater, and then pipe the results through the ls command:

find . -type f -size +800k -exec ls -l {} \;

As mentioned in an earlier tip, it's better with xargs. The command above could be improved as:

find . -type f -size +800k | xargs ls -l

To find all files that match the pattern *.o, print the full filespec (%p), the last-access time (%a) and the last-modified time (%t), and prompt for deletion:

find . -name "*.o" -printf "%p\nA: %a\nM: %t\n" -exec rm -i {} \;

To find files whose data has changed since midnight:

find . -daystart -mtime 0

(The -mtime can be replaced with the -ctime to show files whose status has changed.)

Back to the top


19. Starting a remote X windows program on a local screen

xon «remote_host» -access -user «remote_username» «remote_program_path»

Back to the top


20. Making a boot floppy

dd bs=8192 if=/vmlinuz of=/dev/fd0

That allowed me to recover from a machine that someone infected with a boot sector virus. I still had to rebuild the kernel.

Back to the top


21. Switching parallel from printer to ZIP

modprobe -r lp
modprobe ppa
mount /dev/sdc4 /mnt/ZIP

Back to the top


22. More fun with RPM's

That --queryformat be some powerful voodoo. Convert the datetime tags to human readable form via the system command

convdate -c `rpm -q --queryformat "%{INSTALLTIME}\n" «package name» `

Back to the top


23. Burning CD's

Unfortunately, with no CD-burner on any of the Linux boxes, you have to resort to M$ to do the actual burn. But just copying the files and trying to burn things didn't seem to work. So, on a Linux box, create a disk image, then move the image to a M$ machine. Like so:

mkisofs -vrTJV "«Volume Label»" -o «image filename».iso «root directory»/
mount -t iso9660 -o ro,loop=/dev/loop0 «image filename».iso /mnt/cdrom

The first line makes an ISO file system and write it to a file. The .iso just makes it easier for the Windoze software to recognize it as a CD image. The command line options used are: -v verbose, -r Rockridge extensions, -T make TRANS.TBL files, -J Joliett extensions, -V Volume label, -o output file.

The second line tests the image, by mounting it as though it were a real device.

Fancying the image creation up a bit, the following puts an abstract on the CD and hides the TRANS.TBL from systems that can handle long file names

mkisofs -vrTJV "«Volume Label»" \
     -abstract "
«Short description»" -hide-joliet-trans-tbl \
     -o
«image filename».iso «root directory»/

And a variation with some Macintosh options thrown in...

mkisofs -vrJV "«Volume Label»" -hfs -magic «magic file» -probe \
     -o
«image filename».iso «root directory»/

The magic file helps mkisofs determine which CREATOR and TYPE to use so that a Macintosh knows how to open the files. It appears the magic file is only needed if the system cannot already determine what the file is by examining the first few bytes. (I used /dev/null for the magic file.)

If you DO have a burner on your box, you can add the command:

cdrecord -v -speed=«##» dev=«#,#,#» -data «image filename»

In my case the speed is 10 and the dev is 2,1,0. This burns the image file created by the mkisofs command onto your CD.

Back to the top


24. Turning off NetQUE broadcasts

NetQue boxes attached to dumb printers keep sending RWHO packets (UDP/513) all over campus. This is annoying. To turn it off:

  1. Telnet into the NetQue
  2. Type SU at the prompt.
  3. It will display a Password> prompt. Type a Control-H (ASCII backspace) and then SYSTEM and hit enter. (SYSTEM is the default password.)
  4. If you get to the prompt, type DEFINE SERVER ANNOUNCEMENT DISABLE
  5. Type SYNC
  6. Type LO
  7. Power-cycle the printer server for it to take affect.

Back to the top


25. Checking out from CVS

I don't yet understand what I've done, but apparently, I got it right. The following pulled the latest version of Boa Constructor

cvs -z3 -d:pserver:anonymous@cvs.Boa-Constructor.sourceforge.net:/cvsroot/boa-constuctor co boa

Back to the top


26. Streaming with Icecast and Darwin

Icecast streams MP3 and Ogg Vorbis, Darwin is Apple's QuickTime streamer. Again, I'm not certain of all the details, but I've got them both going.

icecast -b
liveice -F ~/liveice.test 2> /dev/null
/usr/local/sbin/streamingadminserver.pl

The first line starts Icecast listening. The second sends a stream to Icecast for rebroadcast. The third starts the Darwin server. Be sure to check the configuration files in /etc/icecast and ~/liveice.test.

Back to the top


27. Pretty-printing code as web pages

My favorite program lister enscript, can generate color-coded web pages, as well as color-coded Postscript. Separate colors are used for comments, keyowrds, functions, and quoted strings. To generate a page, complete with a table of contents, the magic is:

enscript -E -C -G -j -Whtml --color --toc -p«output».html «program1» [«program2» «program3» ...]

Back to the top


28. md5sum

MD5 checksums are frequently distributed with files to be downloaded, in an effort to insure data integrity. The program md5sum for Linux is fairly easy to find, and may already be on your system.

To check a file, download the corresponding MD5SUM file (possibly named «filename».md5) to the same directory where the files to be checked live. Then issue the command:

md5sum -c «filename».md5

To create an MD5 checksum file for others to use against your files, issue the command:

md5sum «filenames» > «filename».md5

I donwnloaded a Microsoft DOS/Windows version of md5sum from http://etree.org/md5com.html. The web page suggests the directory in which to save the program. It needs to be run from the DOS command prompt.

Back to the top


29. Slow Hand

The problem: During a rescue operation, I needed to copy a HUGE file. Unfortunately, while booted up in Red Hat's rescue mode, memory management seems to have some problems. Every attempt to copy this would go for a while then run out of memory and force a reboot.

Solution: I hypothesized that if I could slow the machine down, I might give it time to recover its memory. (I know I forget things when I think too fast.)

So, how?

  1. Break the file into chunks and copy a chunk at a time, with delays between chunks.
  2. The file (a bzipped tarball) was 689459745 bytes long.
  3. dd (a file copying program) writes nulls when it hasn't got any data. So, I couldn't write more data than was actually in the original file. Otherwise I'd end up with nulls at the end.
  4. That means, the blocksize times the number of blocks had to exactly match the file size.
  5. I found a web page with a factoring calculator, and learned that the prime factors of 689459745 are 3, 5, 13, 19, 379 and 491.
  6. Armed with that info, I opted for 379 blocks of 1819155 bytes each.
  7. Finally, I wrote a little script

#!/bin/sh
# SLOW DOWN! Copy slooooowly, and provide a running
# progress report comparing the file sizes of the
# two files in question. When done, compute the MD5
# checksum for each file, for comparison.
#
# NOTE: A 1-second delay wasn't enough. A 10-second
# delay was, but it was also probably overkill.
 
cd /mnt/sysimage/usr/share
rm /mnt/jaz/home.tbz
touch /mnt/jaz/home.tbz
ls -al home.tar.bz2
ls -al /mnt/jaz/home.tbz
sleep 1
 
for ((blk = 0; blk < 379; blk++))
do
  dd if=home.tar.bz2 of=/mnt/jaz/home.tbz \
     bs=1819155 count=1 \
     seek=$blk  skip=$blk
  ls -al home.tar.bz2
  ls -al /mnt/jaz/home.tbz
  sleep 10
done
md5sum home.tar.bz2 /mnt/jaz/home.tbz

UPDATE: Apparently, I misread or misunderstood the dd command. It doesn't pad its output with NULL's unless you explicitly ask it to by using the conv=sync option. So any reasonable block size should have worked above...

Back to the top


30. Synchronizing with rsync

To copy big directory trees across the entire universe, and maintain protections, user and group ID's etc, use rsync. It appears to have a few kinks--like it's slow as molassas on my machine, I think I caused a kernel panic the first time I used it, and now that it's finished a copy it appears to be hanging, but it gets the job done.

Important tip: It seems to do better at pushing files out to the remote machine, rather than pulling from the remote.

rsync -avz --rsh=ssh «local_directory_tree» «remote_machine»:«remote_destination»

Back to the top


31. Exploring binary RPM's without installing them

Sometimes it's nice to see what's in an RPM file without actually installing it. If you have the source RPM (.src) then it's easy. Just make the binary. But if you don't have it and don't want to bother getting it, you can extract the CPIO "heart" of the RPM and explore that. (cpio = copy in and out.)

rpm2cpio «package».rpm > «package».cpio
cpio -it --verbose < «package».cpio | most
cpio -id --verbose < «package».cpio

The first line pulls out the CPIO from the RPM file. The second gives a verbose listing of the contents of the file, and the third actually does the extraction, forcing the creation of directories that aren't already present.

Back to the top


32. Renaming all files in a directory

Sometimes you want to rename all the files in a directory, and the new names will in some way be based on the old names. Here's one way to tackle the problem (not necessarily the best way):

  1. ls | grep -v "[on]names" > onames
  2. ls | grep -v "[on]names" > nnames
  3. (edit nnames and change the old filenames to their new filenames.)
  4. (edit onames and insert "mv " at the start of each line.)
  5. paste onames nnames > script.sh
  6. bash script.sh
  7. rm onames nnames script.sh

The first two lines make identical files containing all the filenames in the directory, sans the two files being created. The "paste" command in step 5 puts them together, line by line, so you end up with several lines of "mv old-name new-name", which you just push through your shell.

If all you want to do is lowercase the names, this will do the trick:

for i in *; do mv $i `echo $i | tr [A-Z] [a-z]`; done

(Watch out for those backticks.)

Back to the top


33. "rpm -qil" in Debian

To query a Debian package and obtain both a package description and a list of files within the package, use the command:

(dpkg -p «package» ; dpkg -L «package» ) | «pager»

To just get a list of ALL packages (installed and uninstalled, use:

dpkg -l '*'

Back to the top


34. Handling NULL's in Postgresql

The COALESCE function is your friend. It allows you to substitute a string for a NULL value. The example below shows how to combine several fields together into a single string.

SELECT	   COALESCE(name,'')	|| '-' ||
	   COALESCE(version,'')	|| '-' ||
	   COALESCE(release,'') || '\n\t' ||
	   COALESCE(summary,'')	|| '\n\t' ||
	   COALESCE(url,'')
  FROM	   gri
  WHERE	   name ~* '.*devel.*' and release ~* '.*ximian.*'
  ORDER BY name;

That produces output like this:

 ...
sane-backends-devel-1.0.8-1.ximian.1
       The SANE (a universal scanner interface) development toolkit.
       http://www.mostang.com/sane/
SDL-devel-1.2.4-1.ximian.3
       Files needed to develop Simple DirectMedia Layer applications.
       http://www.libsdl.org/
xmms-devel-1.2.7-1.ximian.4
       XMMS - Static libraries and header files.
       http://www.xmms.org/
(28 rows)

Back to the top


35. Using BitTorrent with RedHat

According to http://rhl.redhat.com/, RPMS for Red Hat Linux 7.3 through 9 of BitTorrent are available from:

http://torrent.dulug.duke.edu/btrpms/

Usage is simple:

btdownloadcurses.py --url «http://URL.torrent»

Allow incoming TCP 6881 - 6889 to join the torrent swarm.

Back to the top


Decoding base-64 encoded text

If you end up with a file that is base-64 encoded, fetch a copy of uudecode and add a line to the top of the base-64 encoded file (if it isn't there already) that looks like:

begin-base64 664 «ASCII-file»

Then issue the command:

uudecode -m «b64-file»

This should read in b64-file and create ASCII-file.

Back to the top