Any plans to get rid of nfs+overlayfs for netboot?

[Thomas Neumann]

Hello

This posting is one half rant to let some steam off and one half
documentation/request.


Okay, I'm not a newbie admin. I think I know my way around quite a lot of
linux distributions. That somehow only adds insult to injury, because I'm
struggling for about several hours to get a working fai 4.x installation.
Most of the problems stem from nfs & aufs.

So - Fai 3.x on lucid is running fine and I set out to install a nice and
shiny Dell Blade. Which does not go as expected since it has multiple
network interfaces which believe themselves to have a link (which is true)
and therefor a connection (which is not.) Sadly the initrd decides it
likes eth3 best which definitely does not have a link and does not like to
shuffle around during reboots.

I know about the fai+initramfs+multipleNIC boot problems and therefor
decide to give fai 4.0.1 a spin. How hard can it be?

Very.

Prepare a nice and shiny new Debian Sid box to join the bleeding edge,
install fai-nfsroot, check /etc/fai and let fai-make-nfsroot do it's
magic. Configure nfs exports, check tftp-server and be done with it.
Right?

Wrong.

There was a small hickup, because instead of dracut-network dracut was
installed. Saw the patch by Thomas. Fixed this easily. Rebuild initrd.
Copy initrd.

- non nfs/aufs related quirks -

a) For some reason the kernel (3.2.0-2-amd64) totally refuses to dhcp any
network interface on my VMware virtual machines which I normally use for
testing. (NIC is configured as E1000, nothing esoteric like vmxnet or
flexible) I am now forced to test on the hardware blade.

b) The dracut-initrd recognizes each of the interfaces (which is good) but
takes nearly 200 seconds until the last one has been identified (not so
good...) [the blade has 4 network interfaces]

- nfs/aufs related quirks -

c) aufs does not work. Tells me something is not right (/etc/fstab,
/sysconfig/etc/fstab is missing) and gives up with a a kernel panic.

Okay I say, let's try the lowlevel approach - export /srv/fai/nfsroot as
readwrite and mount that without any overlay.

d) dracut-initrd mounts the root volume, but prefers to mount it as NFSv4.
Which results in such nice things like every file belonging to
4294967294:4294967294

d-1) I can not force a fall back to NFSv3, since dracut-initrd fails with
'protocol not supported' if I try to specify 'vers=3' or 'nfsvers=3'. I
really don't know why, because if I do a chroot into /srv/fai/nfsroot I
can mount NFSv3 to my heart's content. Kernel module is okay, mount.nfs
binary is the same in initramfs and in nfsroot. chroot: yes, netboot: no.
I'm stumped.

d-2) Okay, let's set up NFSv4 idmapping properly instead! (According to
http://serverfault.com/questions/98741/files-mounted-over-nfsv4-are-owned-by-4294967294-uids-and-gids-match)
Still no cookie. /etc and friends still not belong to root:root

Resume: Install process fails, because /var/log/dmesg refuses to belong to
root:root.


This small adventure all started out on monday, when it was raining
outside and I just couldn't be bothered to take a small trip to our
datacenter and install a server locally. By now I could have taken that
trip a couple of times and still have some time left to drink a nice hot
chocolate or something like that. I'm quite a bit miffed on how hard this
multi-NIC adventure turned out. The only thing that keeps me going is the
awesomeness of FAI. Really.



At the moment this whole setup seems pretty fragile to me. In my naive
thinking it should be possible to compress the required functionality into
a (somewhat largish) initrd and skip out on nfs-root completely. But it's
probably not that easy, or else it would be reality already...

It sounds very tempting though, because
 - since there's no nfs-root, we don't need an overlayfs
 - since there's no nfs-root, we don't need to worry about nfs at all
 - since there's no nfs-root, it doesn't matter in what way/order or if at
all the initrd decides to enumerate the network interfaces.

Apart from not being able to mount the root-fs dracut is actually doing a
pretty decent job. Even at handling multiple NICS (except if they're
VMware/E1000).


bye
thomas