Netwinder Rescue HOWTO for Dummies

So, you've trashed your machine, you're in a hurry, and you don't understand, can't understand or don't want to understand Stany's Netwinder Rescue HOWTO? Well, oh, brave but foolish one, we've got just the thing...

This document is a history of what I typed in to get my own machine working. It contains an explicit step-by-step version of Stany's Netwinder Rescue HOWTO, combined with the documentation from Mike Montour's Minimal NFS Rescue System. The reader is STRONGLY advised to make another attempt at understanding what they're actually doing. Many thanks to both Stany and Mike, as well as Terry Boult and Rob Harley who pointed the way for me.

Below is an explanation of my understanding of what I did, followed by a Walk-Thru, and a more specific example.

"Cut the crap, forget the explanation, the fate of all humanity depends on getting my NetWinder running by YESTERDAY!"

Explanation

Let's say some hypothetical dummy skims the mailing list, doesn't read very carefully, ignores dire warnings about how to upgrade, and gets into a situation where:

• the firmware has been successfully upped Nettrom 2.0 or better,
• the hard disk and file systems are still mostly intact,
• nothing else seems right.

[Any resemblance between said dummy and the author are purely coincidental.]

What to do? Well, this Nettrom stuff is marvelous! Those of you new to Linux and Netwinders but fairly familiar with Intel Pentium-style beasties, should think of Nettrom like your BIOS/CMOS with delusions of grandeur. You're going to go into the equivalent of "Setup" (like pressing <F1> or <F2> during the bootup) and change the way the 'winder boots. Some BIOS's let you choose which devices to search for a bootable image, i.e. boot from CD, boot from A: then C:, etc. Nettrom goes at least one step further and lets you boot from an image residing on another computer on your network. In addition, it allows you to set the IP address of the 'winder, so that when the 'winder asks for a boot image from a server, the server has some idea of who's asking, and can decide to permit or deny access. And, finally, the Nettrom also allows you to specify where the root file system lives. Together, the IP address, location of the boot image, and the location of the root file system let you do a diskless boot the Netwinder.

You need to have a few things to start:

• a correctly flashed Nettrom 2.0 or later version of the firmware. (Look carefully at the blue text that appears briefly when the machine is booting. The first line should tell you what version your Nettrom is. If your Nettrom is older than 2.0, you're not stuck, but this document won't help you much. See Stany's Rescue HOWTO.)
• an IP address for your dead Netwinder (henceforth referred to as the dead CLIENT). This can either be explicitly burned into the firmware, or served up by a DHCP server on your network.
• a bootable kernel somewhere that the 'winder can find it.
• a Trivial FTP daemon program (tftpd) on the machine with the bootable kernel image. This machine will be referred to as the TFTP SERVER. (See "man tftp" and "man tftpd".)
• a minimal root system with some handy tools for tinkering with your messed up files on the dead client. In particular, (if I've understood correctly) the ability to mount file systems and export via NFS are the really critical pieces.
• a Network File System (NFS) daemon (nfsd) on the machine with the minimal root file system, now known as the NFS SERVER. (See "man nfsd" and "man exports" for starters.)
• a "HOSPITAL MACHINE", where you'll be performing surgery on the dead client. In my example that happens to be the same as my NFS server, but it doesn't have to be. (This machine must be able to mount NFS exports.)
• Godlike access (aka, root, superuser, etc.) to all the machines involved.

General nutshell outline of the steps:

1. Install (if necessary) and enable tftpd on the TFTP SERVER. Then, drop a bootable kernel where tftpd expects to find files. (Re)start tftpd.
2. Install (if necessary) and enable NFS on the NFS SERVER, place the minimal root system, and export the minimal root system's path to the dead CLIENT via the /etc/exports file. (Re)start nfsd.
3. Boot the dead client, and interrupt the boot to get into the Nettrom. Tell the dead client's Nettrom what it's IP address is (or where to find it), where to find the bootable kernel, and where to find the minimal root system.
4. Continue the boot. With any luck, you'll have some sort of operating system appear out of the ether, and you'll be in single user mode, with a nice little shell prompt.
5. Using this operating system, mount your dead CLIENT's root partition, and export it a la NFS to a machine where you can perform open-heart surgery. Mike's minimal rescue system has a shell script (I assume) that does the dirty work here for you.
6. Get on the machine where you've exported the CLIENT's root file system to, mount it via NFS, and slice, dice, cut, paste, zap with electrodes, til Frankenstein rises.
7. Dismount the NFS export, and stop/disable whatever services you've started on your SERVER(s) that you don't want running permanently.
8. Boot the dead CLIENT, interrupt and put all the Nettrom settings back to normal (i.e. tell it to boot locally again by setting all your boot and root stuff back to the client machine partitions, files, etc.)
9. Continue the boot and pray!

The events below are true. The IP addresses have been changed to protect the innocent. This involved three machines, a dead NetWinder client (134.231.999.0), an NFS server (134.231.999.1), and a TFTP server (134.231.999.2). Brown text means change to a different machine. Bold computer text means you type exactly what you see, except that you substitute your values when you see <red text between angle brackets>.

• On the TFTP SERVER (tftp.gallaudet.edu):
  1. mkdir /tftpboot
     cd /tftpboot
  2. ncftp www.netwinder.org
     cd /pub/ccc/kernel
     get vmlinux-<date>.gz
     exit
     gunzip vmlinux-<date>
  3. Edit /etc/inetd.conf
     Change the line beginning "#tftp ..." removing the "#", thus enabling tftp.
  4. /etc/rc.d/init.d/inet stop
     /etc/rc.d/init.d/inet start

• On the NFS SERVER (nfs.gallaudet.edu):
  1. cd <temp dir>
     ncftp www.netwinder.org
     cd /users/m/mmontour
     get <netwinder nfs rescue>.tar.gz
     exit
     tar xzvf <netwinder nfs rescue>.tar.gz
     mv <netwinder nfs rescue> rescue
  2. Edit /etc/exports
     Add "<temp dir>/rescue <dead client IP #>(no_root_squash)"
  3. /etc/rc.d/init.d/nfs stop
     /etc/rc.d/init.d/nfs start

• On the DEAD CLIENT (dead.gallaudet.edu):
  1. Hold the RESET button down til the system reboots. During the boot, you'll see a 5-second countdown, during which you should press the <SPACE> bar to interrupt the boot and get into the Nettrom.
  2. setenv netconfig_eth0 flash
     setenv eth0_ip <dead client IP #>/<subnet mask>
     setenv kernconfig tftp
     setenv kerntftpserver <tftp server IP #>
     setenv kerntftpfile vmlinux-<date>
     setenv rootconfig nfs
     setenv rootpath <nfs server IP #>:<temp dir>/rescue
     save-all
     boot
  3. With any luck you'll see the 'winder download a kernel, and crank itself up into single user mode. You should also see a disclaimer, and instructions from Mike.
  4. startnfs <hospital IP #> /dev/hda2

• On the HOSPITAL MACHINE (nfs.gallaudet.edu):
  1. mount -t nfs <dead client IP #>:/mnt/localdisk /mnt/dead
     cd /mnt/dead
  2. Fix whatever you need to fix. Everything under /mnt/dead is your dead client's root filesystem. In my case, the fix consisted of copying insmod.aout from the working server to /mnt/dead/lib/modules/misc and editing /mnt/dead/etc/rc.d/rc.sysinit to tell it to use /lib/modules/misc/insmod when loading fpem.o.
  3. umount /mnt/dead
  4. WAIT! It takes a bit of time for the disk to sync after being unmounted.

• On the DEAD CLIENT (dead.gallaudet.edu):
  1. Press <ENTER> to reboot.
  2. Interrupt the boot during the countdown to get into Nettrom.
  3. setenv netconfig_eth0 disk
     setenv eth0_ip 0.0.0.0/0
     setenv kernconfig fs
     setenv kerntftpserver 0.0.0.0
     setenv kerntftpfile vmlinux.rev4
     setenv kerndev /dev/hda2
     setenv kernfile /boot/vmlinux
     setenv rootconfig disk
     setenv rootdev /dev/hda2
     setenv rootpath undefined
     save-all
     boot
  4. Find someone, scream at them "It's ALIVE! It's ALIVE!" and laugh maniacally.

Note: Many of the steps above are not strictly necessary. Some are redundant, and some take the long way around to do things. (For example instead of permanently setting the environment variables in the Nettrom, the save-all can be skipped. Also, the boot command takes several parameters, some of which can replace the setting of the environment variables. I don't think there's any real reason to use two servers either, provided you've got the space. (tftpd and nfsd can happily coexist.)

Here's a more specific example:

• On the TFTP SERVER (tftp.gallaudet.edu):
  1. mkdir /tftpboot
     cd /tftpboot
  2. ncftp www.netwinder.org
     cd /pub/ccc/kernel
     get vmlinux-981015.gz
     exit
     gunzip vmlinux-981015
  3. vi /etc/inetd.conf
     Uncomment the line beginning "#tftp ..." (enabling) tftp.
  4. /etc/rc.d/init.d/inet stop
     /etc/rc.d/init.d/inet start

• On the NFS SERVER (nfs.gallaudet.edu):
  1. cd /mnt/hda4
     ncftp www.netwinder.org
     cd /users/m/mmontour
     get netwinder_nfs_rescue.tar.gz
     exit
     tar tzvf netwinder_nfs_rescue.tar.gz
     mv netwinder_nfs_rescue rescue
  2. vi /etc/exports
     Add "/mnt/hda4/rescue 134.231.999.0(no_root_squash)"
  3. /etc/rc.d/init.d/nfs stop
     /etc/rc.d/init.d/nfs start

• On the DEAD CLIENT (dead.gallaudet.edu):
  1. Hold the RESET button down til the system reboots, and interrupt the boot process, to get into the Nettrom.
  2. setenv netconfig_eth0 flash
     setenv eth0_ip 134.231.999.0/16
     setenv kernconfig tftp
     setenv kerntftpserver 134.231.999.2
     setenv kerntftpfile vmlinux-981015
     setenv kerndev /dev/hda2
     setenv rootconfig nfs
     setenv rootdev /dev/hda2
     setenv rootpath 134.231.999.1:/mnt/hda4/rescue
     setenv kernfile /boot/vmlinux
     save-all
     boot
  3. With any luck you'll see the 'winder download a kernel, and crank itself up into single user mode. You should also see a disclaimer, and instructions from Mike.
  4. startnfs 134.231.999.1 /dev/hda2

• On the HOSPITAL MACHINE (nfs.gallaudet.edu):
  1. mount -t nfs 134.231.999.0:/mnt/localdisk /mnt/dead
     cd /mnt/dead
  2. Fix whatever you need to fix. Everything under /mnt/dead is your dead client's root filesystem. In my case, the fix consisted of copying insmod.aout from the working server to /mnt/dead/lib/modules/misc and editing /mnt/dead/etc/rc.d/rc.sysinit to tell it to use /lib/modules/misc/insmod when loading fpem.o.
  3. umount /mnt/dead
  4. WAIT! It takes a bit of time for the disk to sync after being unmounted.

• On the DEAD CLIENT (dead.gallaudet.edu):
  1. Press <ENTER> to reboot.
  2. Interrupt the boot during the countdown to get into the Nettrom.
  3. setenv netconfig_eth0 disk
     setenv eth0_ip 0.0.0.0/0
     setenv kernconfig fs
     setenv kerntftpserver 0.0.0.0
     setenv kerntftpfile vmlinux.rev4
     setenv kerndev /dev/hda2
     setenv kernfile /boot/vmlinux
     setenv rootconfig disk
     setenv rootdev /dev/hda2
     setenv rootpath undefined
     save-all
     boot
  4. Find someone, scream at them "It's ALIVE! It's ALIVE!" and laugh maniacally.

Disclaimer: Hey, what do I know? It worked for me... You get what you pay for.