Disk config with GRUB/EFI, RAID + LVM
Rémy Dernat
remy.d1 at gmail.com
Tue Dec 5 13:51:30 CET 2023
BTW, I am wondering what would be the corresponding BTRFS config.
Does something like this work as expected or not (mixing / and /tmp this time) :
```
disk_config disk1 disklabel:gpt bootable:1 fstabkey:uuid
primary - 2G - -
primary - 70% - -
primary - 0- - -
disk_config disk2 sameas:disk1
disk_config btrfs fstabkey:uuid
btrfs raid1 /boot/efi disk1.1,disk2.1
auto,ssd,lazytime,rw,nosuid,nodev,exec,subvol=@boot
btrfs raid1 / disk1.4,disk2.4
subvol=@root,defaults,noatime,compress=zstd
btrfs raid1 /home disk1.3,disk2.3
subvol=@home,defaults,noatime,compress=zstd
```
??
Le mar. 5 déc. 2023 à 13:15, Rémy Dernat <remy.d1 at gmail.com> a écrit :
>
> Hi,
>
> After digging some time around this issue, I am finally able to boot
> the RAID1 partition with EFI boot enabled on it (using this example:
> https://fai-project.org/download/misc/disk_config/lvm-on-luks-on-raid).
>
> Here is my final disk_config file:
>
> ```# example of new config file for setup-storage
> #
> # <type> <mountpoint> <size> <fs type> <mount options> <misc options>
>
> disk_config disk1 disklabel:gpt bootable:1 fstabkey:uuid
> primary - 2G - -
> primary - 0- - -
>
> disk_config disk2 sameas:disk1
>
> disk_config raid
> raid1 /boot/efi disk1.1,disk2.1 vfat rw,noatime
> mdcreateopts="--verbose --metadata=0.90"
> raid1 - disk1.2,disk2.2 - -
>
> disk_config lvm
> vg vg_system md1
> vg_system-swap swap 16G swap sw
> vg_system-root / 60G-200G ext4 rw,noatime
> vg_system-tmp /tmp 20G-50G ext2 rw,noatime,nosuid,nodev
> vg_system-home /home 30%- ext4 rw,noatime,nosuid,nodev
>
> ```
>
> Best regards,
> Rémy
>
> Le ven. 1 déc. 2023 à 12:11, Rémy Dernat <remy.d1 at gmail.com> a écrit :
> >
> > Hi Andrew,
> >
> > Thanks for suggestions.
> >
> > Now, my file "disk_config/GRUB_EFI_RAID1" is [*]:
> >
> > ```
> > # example of new config file for setup-storage
> > #
> > # <type> <mountpoint> <size> <fs type> <mount options> <misc options>
> >
> > disk_config disk1 disklabel:gpt fstabkey:uuid bootable:1
> > primary - 2G - -
> > primary /boot/efi 1G vfat rw createopts="-F32"
> > primary - 2G - -
> > primary - 0- - -
> >
> > #disk_config disk2 sameas:disk1
> > disk_config disk2 disklabel:gpt fstabkey:uuid
> > primary - 2G - -
> > primary /boot/efi2 1G vfat rw createopts="-F32"
> > primary - 2G - -
> > primary - 0- - -
> >
> > disk_config raid fstabkey:uuid
> > raid1 /boot disk1.1,disk2.1 ext4 rw,noatime mdcreateopts="--verbose
> > --metadata=0.90"
> > raid1 swap disk1.3,disk2.3 swap sw
> > raid1 - disk1.4,disk2.4 - -
> >
> > disk_config lvm fstabkey:uuid
> > vg vg_system md2
> > vg_system-root / 60G-200G ext4 rw,noatime
> > vg_system-tmp /tmp 20G-50G ext2 rw,noatime,nosuid,nodev
> > ##vg_system-scratch /scratch 30%- ext2 rw,noatime
> > vg_system-home /home 30%- ext4 rw,noatime,nosuid,nodev
> > ```
> >
> > => Even if the installation succeeded, grub does not seem to work.
> >
> > Anyway, I am wondering how you set this in the disk_config file (syntax ?):
> >
> > > I like to put a 100MB partition at the end of disks in a RAID pair to allow
> > > for slightly differing size of a replacement disk. Although currently it
> > > seems that any new disk we get is many many GB bigger than the old failed
> > > one.
> >
> > Thanks again,
> >
> > Best regards,
> > Rémy
> >
> >
> >
> > [*] there was an error for /boot partition which was not formatted the
> > right way (vfat does not allow symbolic links). vfat should be used
> > for /boot/EFI only.
> >
> > Le ven. 1 déc. 2023 à 05:43, Andrew Ruthven <andrew at etc.gen.nz> a écrit :
> > >
> > > Nice, good work.
> > >
> > > A couple of suggestions...
> > >
> > > I'd put swap on RAID1, if you have a disk go offline while the system is
> > > running, you've lost anything that had been swapped out to the swap on the
> > > failed disk.
> > >
> > > I like to put a 100MB partition at the end of disks in a RAID pair to allow
> > > for slightly differing size of a replacement disk. Although currently it
> > > seems that any new disk we get is many many GB bigger than the old failed
> > > one.
> > >
> > > I've found that some servers allow the EFI partition to be RAID1, the
> > > current servers we're using support that. So no need to copy files around.
> > >
> > > Cheers,
> > > Andrew
> > >
> > > On Thu, 2023-11-30 at 17:20 +0100, Rémy Dernat wrote:
> > >
> > > > Ok, FYI, I think I have a working configuration with :
> > > >
> > > > ```
> > > > # example of new config file for setup-storage
> > > > #
> > > > # <type> <mountpoint> <size> <fs type> <mount options> <misc
> > > > options>
> > > >
> > > > disk_config disk1 disklabel:gpt fstabkey:uuid bootable:1
> > > > primary - 2G - -
> > > > primary /boot/efi 1G vfat rw createopts="-F32"
> > > > primary swap 2G swap sw,pri=1
> > > > primary - 0- - -
> > > >
> > > > #disk_config disk2 sameas:disk1
> > > > disk_config disk2 disklabel:gpt fstabkey:uuid
> > > > primary - 2G - -
> > > > primary /boot/efi2 1G vfat rw createopts="-F32"
> > > > primary swap 2G swap sw,pri=1
> > > > primary - 0- - -
> > > >
> > > > disk_config raid fstabkey:uuid
> > > > #raid1 /boot disk1.1,disk2.1 - - mdcreateopts="--verbose
> > > > --metadata=1.0" createopts="-L boot"
> > > > raid1 /boot disk1.1,disk2.1 vfat rw,noatime mdcreateopts="--verbose
> > > > --metadata=0.90"
> > > > raid1 - disk1.4,disk2.4 - -
> > > >
> > > > disk_config lvm fstabkey:uuid
> > > > vg vg_system md1
> > > > vg_system-root / 60G-200G ext4 rw,noatime
> > > > vg_system-tmp /tmp 20G-50G ext2 rw,noatime,nosuid,nodev
> > > > ##vg_system-scratch /scratch 30%- ext2 rw,noatime
> > > > vg_system-home /home 30%- ext4 rw,noatime,nosuid,nodev
> > > > ```
> > > >
> > > > My previous message was crystal clear. md0 did not have any formatted
> > > > filesystem on it. So I added `vfat` option on /boot.
> > > >
> > > > Best regards,
> > > > Rémy
> > > >
> > > >
> > > > Le jeu. 30 nov. 2023 à 16:53, Rémy Dernat
> > > <[remy.d1 at gmail.com](mailto:remy.d1 at gmail.com)> a écrit :
> > > >
> > > > >
> > > > > What is strange here, is that md1 is found, as well as md0, but
> > > > > contrary to md0, md1 have an UUID ...
> > > > >
> > > > > However, UUID seems to be retrieve by blkid from /dev/disk/by-id,
> > > > > where both md array have their own mdadm UUID, but only md1 have an
> > > > > UUID from LVM (with pvdisplay / pvs ) ...
> > > > >
> > > > >
> > > > > Le jeu. 30 nov. 2023 à 14:52, Rémy Dernat
> > > <[remy.d1 at gmail.com](mailto:remy.d1 at gmail.com)> a écrit :
> > > > >
> > > > > >
> > > > > > Hi Andrew,
> > > > > >
> > > > > > Indeed, this removed the syntax issue. However, I have a new error
> > > now:
> > > > > >
> > > > > > ```
> > > > > > /sbin/blkid -c /dev/null -s UUID -o value /dev/md0 had exit code 2
> > > > > > Command had non-zero exit code
> > > > > > ```
> > > > > > Indeed, blkid is not able to found md0 UUID.
> > > > > >
> > > > > > I tried to modify the disk_config file to avoid this error, but I am
> > > > > > still stuck with it. Here is what look like my disk_config file now:
> > > > > >
> > > > > > ```
> > > > > > # example of new config file for setup-storage
> > > > > > #
> > > > > > # <type> <mountpoint> <size> <fs type> <mount options> <misc
> > > options>
> > > > > >
> > > > > > disk_config disk1
> > > > > > primary - 2G - -
> > > > > > primary /boot/efi 1G vfat rw createopts="-F32"
> > > > > > primary swap 2G swap sw,pri=1
> > > > > > primary - 0- - -
> > > > > >
> > > > > > #disk_config disk2 sameas:disk1
> > > > > > disk_config disk2 disklabel:gpt fstabkey:uuid
> > > > > > primary - 2G - -
> > > > > > primary /boot/efi2 1G vfat rw createopts="-F32"
> > > > > > primary swap 2G swap sw,pri=1
> > > > > > primary - 0- - -
> > > > > >
> > > > > > disk_config raid fstabkey:uuid
> > > > > > raid1 /boot disk1.1,disk2.1 - - mdcreateopts="--verbose
> > > > > > --metadata=0.90" createopts="-L boot"
> > > > > > raid1 - disk1.4,disk2.4 - -
> > > > > >
> > > > > > disk_config lvm fstabkey:uuid
> > > > > > vg vg_system md1
> > > > > > vg_system-root / 60G-200G ext4 rw,noatime
> > > > > > vg_system-tmp /tmp 20G-50G ext2 rw,noatime,nosuid,nodev
> > > > > > ##vg_system-scratch /scratch 30%- ext2 rw,noatime
> > > > > > vg_system-home /home 30%- ext4 rw,noatime,nosuid,nodev
> > > > > > ```
> > > > > >
> > > > > > I also tried to change the raid metadata to 1.0, but that did not
> > > > > > change anything. I wrote back ` fstabkey:uuid` behind disk_config raid
> > > > > > because it does not fail with syntax error.
> > > > > >
> > > > > > Thanks for help
> > > > > >
> > > > > > Le jeu. 30 nov. 2023 à 11:55, Andrew Ruthven
> > > <[andrew at etc.gen.nz](mailto:andrew at etc.gen.nz)> a écrit :
> > > > > >
> > > > > > >
> > > > > > > Hey,
> > > > > > >
> > > > > > > On Thu, 2023-11-30 at 11:14 +0100, Rémy Dernat wrote:
> > > > > > >
> > > > > > > > Hi awesome Fai guys,
> > > > > > > >
> > > > > > > > I have a weird syntax issue with the following disk_config:
> > > > > > > >
> > > > > > >
> > > > > > > [snip]
> > > > > > >
> > > > > > > > disk_config raid disklabel:gpt fstabkey:uuid bootable:1
> > > > > > >
> > > > > > >
> > > > > > > This line is your problem, remove everything after the word raid. So
> > > it
> > > > > > > should just be:
> > > > > > >
> > > > > > > disk_config raid
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Andrew
> > > > > > > --
> > > > > > > Andrew Ruthven, Wellington, New Zealand
> > > > > > > [andrew at etc.gen.nz](mailto:andrew at etc.gen.nz) |
> > > > > > > Catalyst Cloud: | This space intentionally left blank
> > > > > > > [https://catalystcloud.nz](https://catalystcloud.nz) |
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > ```
> > > --
> > > Andrew Ruthven, Wellington, New Zealand
> > > andrew at etc.gen.nz |
> > > Catalyst Cloud: | This space intentionally left blank
> > > https://catalystcloud.nz |
> > >
> > >
> > > ```
> > >
> > >
More information about the linux-fai
mailing list