FAI install with huge JBODs present - disk/by-id?

[Steffen Grunewald]

On Tue, 2024-12-17 at 10:07:56 +0100, Steffen Grunewald wrote:
> Good morning,
> 
> after a very promising start, I experienced a set-back yesterday with an
> otherwise proven-to-work FAI setup, and need an alternative now.
> 
[...]
> 
> After setting up multipath and storage zpools, I can no longer boot into
> the system as long as at least one JBOD is connected - grub 2.06 gets an
> "error: out of memory" and even refuses such elementary tasks as
> "ls (...)/".
> 
[...]
> 
> Is there a regular way via disk_config, or do I have to invent a 17-sided
> wheel myself?


To summarize the approach that worked in the end, and just in time:


(1) in a class/* script, write a new disklist to ${LOGDIR}/additional.var
    that doesn't have any /dev/ prefixes

	disklist="sdbj sddu"

    e.g. by checking /dev/disk/by-* for the right disk model (sysinfo was
    very helpful to find a pattern that sorts the disk the right way, i.e.
    by-path may be a better way that by-id, if you need to know which slot)

(2) since a ZFS root pool didn't work, I went for mdraid instead, with the
    disk_config referring to "disk1" and "disk2",

disk_config disk1 disklabel:gpt fstabkey:label align-at:1M
primary -                   512         -       -
primary /boot/efi           512         vfat    rw      createopts="-n EFI1"
primary -                   30G         -       -
primary -                 -100%         -       -

disk_config disk2 disklabel:gpt fstabkey:label align-at:1M
primary -                   512         -       -
primary /boot/efi2          512         vfat    rw      createopts="-n EFI2"
primary -                   30G         -       -
primary -                 -100%         -       -

disk_config raid fstabkey:label
raid1   /boot   disk1.1,disk2.1         ext4    rw      createopts="-L BOOT"
raid1   swap    disk1.3,disk2.3         swap    sw      createopts="-L SWAP"
raid1   /       disk1.4,disk2.4         ext4    rw,noatime,errors=remount-ro    createopts="-m1 -L ROOT"

    note that mkfs.vfat needs "-n" to set the label while all others use "-L"


Labels turned out to be essential since the 247 disks detected by the OS are
sorted in any imaginable order (and more). The "p=xxx" scheme unfortunately
doen't apply to "raid" lines.
There could be improvement, as the /boot partition likely isn't necessary,
but /boot/efi is (and the second EFI partition still needs to be synced by
hand).


Thanks to all who provided ideas and insight.

Enjoy the time "between the years", whatever you celebrate.

Thanks again,
 Steffen


-- 
Steffen Grunewald, Cluster Administrator
Max Planck Institute for Gravitational Physics (Albert Einstein Institute)
Am Mühlenberg 1 * D-14476 Potsdam-Golm * Germany
~~~
Fon: +49-331-567 7274
Mail: steffen.grunewald(at)aei.mpg.de
~~~