Linux Tips

[This document is designed for a browser that can handle color and the two special characters sometimes referred to as French quotes « (not to be confused with <<) & » (not to be confused with >>). However, I hope the document will also be somewhat readable from browsers which do not have these features, such as lynx.]

The following tips are ones that I've become tired of looking up in the man and info pages, then whittling down to their barest essences.

• Boldfaced red teletype text indicates text that should be typed exactly as shown.
• «Italicized text within French quotes» indicates variable text -- often a filename.
• Boldface black text refers to exact text that is already in a file (or possibly a prompt provided by the computer).

1. Redirecting Standard Error to Standard Out
2. Making a patch file
3. Copying directory trees
4. Getting landscape output
5. Verifying Red Hat packages
6. Security: Watching the watchers...
7. Stripping comments:
8. Verifying all RPM's
9. Viewing post-installation RPM scripts
10. What are we listening to?
11. Finding duplicate files with identical contents
12. Listing DNS stuff
13. Who's been sleeping in *MY* bed?
14. Clip-clip. Taking care of really LONG lines
15. Find and delete
16. Formatting and using a floppy
17. Mounting an NSF device
18. Searching for files and manipulating them
19. Starting a remote X windows program on a local screen
20. Making a boot floppy
21. Switching parallel from printer to ZIP
22. More fun with RPM's
23. Burning CD's
24. Turning off NetQUE broadcasts
25. Checking out from CVS
26. Streaming with Icecast and Darwin
27. Pretty-printing code as web pages
28. md5sum
29. Slow Hand
30. Synchronizing with rsync
31. Exploring binary RPM's without installing them
32. Renaming all files in a directory
33. "rpm -qil" in Debian
34. Handling NULL's in Postgresql
35. Using BitTorrent with RedHat
36. Decoding base-64 encoded text

1. Redirecting Standard Error to Standard Out

The proper way to redirect stderr is to first decide where stdout is going, and THEN redirect the stderr to stdout. So, for example:

«yada-yada» | «pager» 2>&1

takes the output of «yada-yada» and pipes it into «pager», then tells the system to send stderr to wherever stdout is going. (A pager is a program that allows a user to scroll through long documents. I use one called most, but older, commonly used ones include more and less.

Back to the top

2. Making a patch file

Assume we have a Red Hat source RPM...

1. rpm -Uvh «package».src.rpm
2. cd /usr/src/redhat/SPECS
3. rpm -bp «package».spec
4. cd /usr/src/redhat/BUILD
5. mv «package» «package».orig
6. cd /usr/src/redhat/SPECS
7. rpm -bp «package».spec
8. cd /usr/src/redhat/BUILD
9. (edit to your heart's content)
10. diff -Naur «package».orig «package» > ../SOURCES/«package».patch
11. rm -rf «package».orig
12. cd /usr/src/redhat/SPECS
13. emacs «package».spec
    ...
    Source: ...
    Patch: «package».patch
    ...
    %prep
    %setup
    %patch -p 1
14. rpm -ba «package».spec

The idea is to create two directories with identical contents, then modify one of them. Create a diff file of the changes and save it. I THINK I got all the basic steps in there... However, it may be necessary to create the tarball too, in which case you need something like:

tar czvf «archive».tar.gz «directory_to_archive»

Back to the top

3. Copying directory trees

Often, it becomes necessary to copy entire directory trees from one directory to another. The method I saw somewhere was:

cd «olddir» ; tar cf - . | (cd «newdir» ; tar xpf -)

This creates a tarball that never actually becomes a file. The tarball is piped directly to a little script subroutine which changes to the new directory, and untars the tarball on the fly.

According to JP Abgrall, there's an optimized way to do this:

tar cf - -C «olddir» . | tar xpf - -C «newdir»

Back to the top

4. Getting landscape output

mpage -1lvH «filename» | lpr

The man page suggests that there's a way to pass pr switches to mpage (using -p instead of -H), but I've been unable to pass the line-numbering switch -n together with pr into mpage. (What I want is the headings a la -H together with line numbering.)

A fancier way, requiring a bit more study, is to use enscript. In fact, enscript is neat for LOTS of stuff -- customizable layout, "Page X of Y", line numbers, etc.

Back to the top

5. Verifying Red Hat packages

You can verify each package with the following command:

rpm --checksig

If you only wish to verify that each package has not been corrupted or tampered with, examine only the md5sum with the following command:

rpm --checksig --nopgp

Back to the top

6. Security: Watching the watchers...

The netstat command is a handy tool for seeing who's poking around at any given moment.

netstat -v | most

Back to the top

7. Stripping comments:

Assuming you have a script that uses the number sign (a.k.a. pound symbol, hash mark) as a comment character, and you wish to examine only those lines containing "active" commands and options, the following will produce such a listing:

grep -v "^\#" «scriptfile» | grep -v "^[[:space:]]*$"

What's happening: The file is first stripped of lines beginning with #. Then, that result is stripped of any lines which have 0 or more whitespace characters (and nothing else) between the start and end of the line.

Back to the top

8. Verifying all RPM's

Here's a small script that constructs a list of all package names sans version numbers, then feeds the list to rpm with the verify option. It also echoes the package name. NOTE THE QUOTES: Some are quotes, NONE are apostrophes, and some are backquotes (a.k.a. accents grave).

for i in `rpm -qa --queryformat "%{NAME}\n" | sort`
do
  echo $i":" >>verify.log 2>&1
  rpm --verify $i >>verify.log 2>&1
done

The stuff enclosed in backquotes gets run as a script within a script, and the output of that is fed to the outer script. (See man eval and other stuff about evaluating.)

Back to the top

9. Viewing post-installation RPM scripts

Occasionally, after the files are dropped onto the system, hither and yon, RPM will run a script embedded in the package file. It's nice to see what the squirrels are doing under the hood.

rpm -q --scripts «package-name»

Back to the top

10. What are we listening to?

lsof shows which processes are listening on a given port. (Actually it stands for "list open files", which shows what files are currently open.) lsof -i will list which ports are open on the machine.

lsof -i

Back to the top

11. Finding duplicate files with identical contents

There's probably a better way, but this worked for me:

diff -qrs «directory1» «directory2» 2>&1 | grep "identical$" > «unedited-shell-script.sh»

Then edit the file «unedited-shell-script.sh» to your heart's content.

Back to the top

12. Listing DNS stuff

Lots of different ways to do this, but I like:

nslookup
> ls -d gallaudet.edu

Back to the top

13. Who's been sleeping in *MY* bed?

Here's a more informative way to use the last command:

last -adf «wtmp_file»

Back to the top

14. Clip-clip. Taking care of really LONG lines

Lots of times, we only need to see the beginning of lines in a file to determine something. (For example, a subroutine that takes a single string argument may have a really long string literal.) To see just the first 100 characters on a line use the cut command. Like so:

cut -b -100 «FY2000.sql» | land

(land is an alias I've set up to print a file in landscape orientation using enscript command -- a very nice printing program.)

15. Find and delete

Sometimes it's nice to do something (like delete) a bunch of files based on a searchable criteria, e.g. portion of the filename, size, date, etc. Here's how:

find . -empty -exec rm -v {} \;

This is a specific example that searches for empty files and directories from the current working directory down, and then it deletes them. The important parts are the {} which gets replaced with whatever has been found, and the \;, an escaped semicolon indicating the end of the command to be "exec'ed". (And the -v is the ubiquitous "verbose" option, to tell you what's happening.)

A better approach, I'm told, is:

find . -empty | xargs rm -v

Back to the top

16. Formatting and using a floppy

Without mounting anything, just pop a floppy in the drive and...

fdformat /dev/fd0H1440
mke2fs /dev/fd0H1440
mount /dev/fd0 /mnt/floppy ...
umount /mnt/floppy

Back to the top

17. Mounting an NSF device

mount «server»:«remote_directory» /«local_mount_point» -t nfs -o ro

Back to the top

18. Searching for files and manipulating them

To find files in or beneath the current directory, of type "file", of size 800 KB or greater, and then pipe the results through the ls command:

find . -type f -size +800k -exec ls -l {} \;

As mentioned in an earlier tip, it's better with xargs. The command above could be improved as:

find . -type f -size +800k | xargs ls -l

To find all files that match the pattern *.o, print the full filespec (%p), the last-access time (%a) and the last-modified time (%t), and prompt for deletion:

find . -name "*.o" -printf "%p\nA: %a\nM: %t\n" -exec rm -i {} \;

To find files whose data has changed since midnight:

find . -daystart -mtime 0

(The -mtime can be replaced with the -ctime to show files whose status has changed.)

Back to the top

19. Starting a remote X windows program on a local screen

xon «remote_host» -access -user «remote_username» «remote_program_path»

Back to the top

20. Making a boot floppy

dd bs=8192 if=/vmlinuz of=/dev/fd0

That allowed me to recover from a machine that someone infected with a boot sector virus. I still had to rebuild the kernel.

Back to the top

21. Switching parallel from printer to ZIP

modprobe -r lp
modprobe ppa
mount /dev/sdc4 /mnt/ZIP

Back to the top

22. More fun with RPM's

That --queryformat be some powerful voodoo. Convert the datetime tags to human readable form via the system command

convdate -c `rpm -q --queryformat "%{INSTALLTIME}\n" «package name» `

Back to the top

23. Burning CD's

Unfortunately, with no CD-burner on any of the Linux boxes, you have to resort to M$ to do the actual burn. But just copying the files and trying to burn things didn't seem to work. So, on a Linux box, create a disk image, then move the image to a M$ machine. Like so:

mkisofs -vrTJV "«Volume Label»" -o «image filename».iso «root directory»/
mount -t iso9660 -o ro,loop=/dev/loop0 «image filename».iso /mnt/cdrom

The first line makes an ISO file system and write it to a file. The .iso just makes it easier for the Windoze software to recognize it as a CD image. The command line options used are: -v verbose, -r Rockridge extensions, -T make TRANS.TBL files, -J Joliett extensions, -V Volume label, -o output file.

The second line tests the image, by mounting it as though it were a real device.

Fancying the image creation up a bit, the following puts an abstract on the CD and hides the TRANS.TBL from systems that can handle long file names

mkisofs -vrTJV "«Volume Label»" \
     -abstract "«Short description»" -hide-joliet-trans-tbl \
     -o «image filename».iso «root directory»/

And a variation with some Macintosh options thrown in...

mkisofs -vrJV "«Volume Label»" -hfs -magic «magic file» -probe \
     -o «image filename».iso «root directory»/

The magic file helps mkisofs determine which CREATOR and TYPE to use so that a Macintosh knows how to open the files. It appears the magic file is only needed if the system cannot already determine what the file is by examining the first few bytes. (I used /dev/null for the magic file.)

If you DO have a burner on your box, you can add the command:

cdrecord -v -speed=«##» dev=«#,#,#» -data «image filename»

In my case the speed is 10 and the dev is 2,1,0. This burns the image file created by the mkisofs command onto your CD.

Back to the top

24. Turning off NetQUE broadcasts

NetQue boxes attached to dumb printers keep sending RWHO packets (UDP/513) all over campus. This is annoying. To turn it off:

1. Telnet into the NetQue
2. Type SU at the prompt.
3. It will display a Password> prompt. Type a Control-H (ASCII backspace) and then SYSTEM and hit enter. (SYSTEM is the default password.)
4. If you get to the prompt, type DEFINE SERVER ANNOUNCEMENT DISABLE
5. Type SYNC
6. Type LO
7. Power-cycle the printer server for it to take affect.

Back to the top

25. Checking out from CVS

I don't yet understand what I've done, but apparently, I got it right. The following pulled the latest version of Boa Constructor

cvs -z3 -d:pserver:anonymous@cvs.Boa-Constructor.sourceforge.net:/cvsroot/boa-constuctor co boa

Back to the top

26. Streaming with Icecast and Darwin

Icecast streams MP3 and Ogg Vorbis, Darwin is Apple's QuickTime streamer. Again, I'm not certain of all the details, but I've got them both going.

icecast -b
liveice -F ~/liveice.test 2> /dev/null
/usr/local/sbin/streamingadminserver.pl

The first line starts Icecast listening. The second sends a stream to Icecast for rebroadcast. The third starts the Darwin server. Be sure to check the configuration files in /etc/icecast and ~/liveice.test.

Back to the top

27. Pretty-printing code as web pages

My favorite program lister enscript, can generate color-coded web pages, as well as color-coded Postscript. Separate colors are used for comments, keyowrds, functions, and quoted strings. To generate a page, complete with a table of contents, the magic is:

enscript -E -C -G -j -Whtml --color --toc -p«output».html «program1» [«program2» «program3» ...]

Back to the top

28. md5sum

MD5 checksums are frequently distributed with files to be downloaded, in an effort to insure data integrity. The program md5sum for Linux is fairly easy to find, and may already be on your system.

To check a file, download the corresponding MD5SUM file (possibly named «filename».md5) to the same directory where the files to be checked live. Then issue the command:

md5sum -c «filename».md5

To create an MD5 checksum file for others to use against your files, issue the command:

md5sum «filenames» > «filename».md5

I donwnloaded a Microsoft DOS/Windows version of md5sum from http://etree.org/md5com.html. The web page suggests the directory in which to save the program. It needs to be run from the DOS command prompt.

Back to the top

29. Slow Hand

The problem: During a rescue operation, I needed to copy a HUGE file. Unfortunately, while booted up in Red Hat's rescue mode, memory management seems to have some problems. Every attempt to copy this would go for a while then run out of memory and force a reboot.

Solution: I hypothesized that if I could slow the machine down, I might give it time to recover its memory. (I know I forget things when I think too fast.)

So, how?

1. Break the file into chunks and copy a chunk at a time, with delays between chunks.
2. The file (a bzipped tarball) was 689459745 bytes long.
3. dd (a file copying program) writes nulls when it hasn't got any data. So, I couldn't write more data than was actually in the original file. Otherwise I'd end up with nulls at the end.
4. That means, the blocksize times the number of blocks had to exactly match the file size.
5. I found a web page with a factoring calculator, and learned that the prime factors of 689459745 are 3, 5, 13, 19, 379 and 491.
6. Armed with that info, I opted for 379 blocks of 1819155 bytes each.
7. Finally, I wrote a little script

#!/bin/sh
# SLOW DOWN! Copy slooooowly, and provide a running
# progress report comparing the file sizes of the
# two files in question. When done, compute the MD5
# checksum for each file, for comparison.
#
# NOTE: A 1-second delay wasn't enough. A 10-second
# delay was, but it was also probably overkill.
 
cd /mnt/sysimage/usr/share
rm /mnt/jaz/home.tbz
touch /mnt/jaz/home.tbz
ls -al home.tar.bz2
ls -al /mnt/jaz/home.tbz
sleep 1
 
for ((blk = 0; blk < 379; blk++))
do
  dd if=home.tar.bz2 of=/mnt/jaz/home.tbz \
     bs=1819155 count=1 \
     seek=$blk  skip=$blk
  ls -al home.tar.bz2
  ls -al /mnt/jaz/home.tbz
  sleep 10
done
md5sum home.tar.bz2 /mnt/jaz/home.tbz

UPDATE: Apparently, I misread or misunderstood the dd command. It doesn't pad its output with NULL's unless you explicitly ask it to by using the conv=sync option. So any reasonable block size should have worked above...

Back to the top

30. Synchronizing with rsync

To copy big directory trees across the entire universe, and maintain protections, user and group ID's etc, use rsync. It appears to have a few kinks--like it's slow as molassas on my machine, I think I caused a kernel panic the first time I used it, and now that it's finished a copy it appears to be hanging, but it gets the job done.

Important tip: It seems to do better at pushing files out to the remote machine, rather than pulling from the remote.

rsync -avz --rsh=ssh «local_directory_tree» «remote_machine»:«remote_destination»

Back to the top

31. Exploring binary RPM's without installing them

Sometimes it's nice to see what's in an RPM file without actually installing it. If you have the source RPM (.src) then it's easy. Just make the binary. But if you don't have it and don't want to bother getting it, you can extract the CPIO "heart" of the RPM and explore that. (cpio = copy in and out.)

rpm2cpio «package».rpm > «package».cpio
cpio -it --verbose < «package».cpio | most
cpio -id --verbose < «package».cpio

The first line pulls out the CPIO from the RPM file. The second gives a verbose listing of the contents of the file, and the third actually does the extraction, forcing the creation of directories that aren't already present.

Back to the top

32. Renaming all files in a directory

Sometimes you want to rename all the files in a directory, and the new names will in some way be based on the old names. Here's one way to tackle the problem (not necessarily the best way):

1. ls | grep -v "[on]names" > onames
2. ls | grep -v "[on]names" > nnames
3. (edit nnames and change the old filenames to their new filenames.)
4. (edit onames and insert "mv " at the start of each line.)
5. paste onames nnames > script.sh
6. bash script.sh
7. rm onames nnames script.sh

The first two lines make identical files containing all the filenames in the directory, sans the two files being created. The "paste" command in step 5 puts them together, line by line, so you end up with several lines of "mv old-name new-name", which you just push through your shell.

If all you want to do is lowercase the names, this will do the trick:

for i in *; do mv $i `echo $i | tr [A-Z] [a-z]`; done

(Watch out for those backticks.)

Back to the top

33. "rpm -qil" in Debian

To query a Debian package and obtain both a package description and a list of files within the package, use the command:

(dpkg -p «package» ; dpkg -L «package» ) | «pager»

To just get a list of ALL packages (installed and uninstalled, use:

dpkg -l '*'

Back to the top

34. Handling NULL's in Postgresql

The COALESCE function is your friend. It allows you to substitute a string for a NULL value. The example below shows how to combine several fields together into a single string.

SELECT	   COALESCE(name,'')	|| '-' ||
	   COALESCE(version,'')	|| '-' ||
	   COALESCE(release,'') || '\n\t' ||
	   COALESCE(summary,'')	|| '\n\t' ||
	   COALESCE(url,'')
  FROM	   gri
  WHERE	   name ~* '.*devel.*' and release ~* '.*ximian.*'
  ORDER BY name;

That produces output like this:

 ...
sane-backends-devel-1.0.8-1.ximian.1
       The SANE (a universal scanner interface) development toolkit.
       http://www.mostang.com/sane/
SDL-devel-1.2.4-1.ximian.3
       Files needed to develop Simple DirectMedia Layer applications.
       http://www.libsdl.org/
xmms-devel-1.2.7-1.ximian.4
       XMMS - Static libraries and header files.
       http://www.xmms.org/
(28 rows)

Back to the top

35. Using BitTorrent with RedHat

According to http://rhl.redhat.com/, RPMS for Red Hat Linux 7.3 through 9 of BitTorrent are available from:

http://torrent.dulug.duke.edu/btrpms/

Usage is simple:

btdownloadcurses.py --url «http://URL.torrent»

Allow incoming TCP 6881 - 6889 to join the torrent swarm.

Back to the top

Decoding base-64 encoded text

If you end up with a file that is base-64 encoded, fetch a copy of uudecode and add a line to the top of the base-64 encoded file (if it isn't there already) that looks like:

begin-base64 664 «ASCII-file»

Then issue the command:

uudecode -m «b64-file»

This should read in b64-file and create ASCII-file.

Back to the top