Issues with Older Backported Kernel Versions 6.5 and 6.7 in FAI Server

[Samadhan Fuke]

We are using FAI tools to install Debian10 Buster and Debian12 Bookworm
clients with backported kernel versions.

We use the latest FAI CD ISO 6.2
<https://fai-project.org/fai-cd/faicd64-small_6.2.2.iso> for Debian 12
Bookworm clients.
We installed Debian12 with dedicated separate kernel versions using
backported kernels 6.5 and 6.7
linux-image-6.5.0-0.deb12.4-amd64 linux-image-6.7.12+bpo-amd64
Everything was working fine until last week.

As you know, backported kernels 6.5 and 6.7 have been removed from the
official Debian repositories and replaced with the new backported kernel
6.9, which is in testing. Debian -- Details of package
linux-image-6.9.7+bpo-amd64 in bookworm-backports
<https://packages.debian.org/bookworm-backports/kernel/linux-image-6.9.7+bpo-amd64>

To manage these backported features in FAI, we created shell scripts. Since
the backported kernels are no longer available on the remote site, we used
the Debian Snapshot snapshot.debian.org to retrieve old kernel packages
based on dates and version numbers. I have verified this manually and was
able to get the required older kernel versions on the FAI server.


*Here is what I have tried to achieve this:*
1. As backported kernel versions are removed from the remote, we cannot get
kernels 6.5 and 6.7, only the latest kernel 6.9 is able to get.
root at faiserver:~# apt-cache search linux-image-6.5
root at faiserver:~# apt-cache search linux-image-6.7

root at faiserver:~# apt-cache search linux-image-6.9
linux-headers-6.9.7+bpo-amd64 - Header files for Linux 6.9.7+bpo-amd64
linux-headers-6.9.7+bpo-cloud-amd64 - Header files for Linux
6.9.7+bpo-cloud-amd64
linux-headers-6.9.7+bpo-rt-amd64 - Header files for Linux 6.9.7+bpo-rt-amd64
linux-image-6.9.7+bpo-amd64-dbg - Debug symbols for
linux-image-6.9.7+bpo-amd64
linux-image-6.9.7+bpo-amd64-unsigned - Linux 6.9 for 64-bit PCs
linux-image-6.9.7+bpo-cloud-amd64-dbg - Debug symbols for
linux-image-6.9.7+bpo-cloud-amd64
linux-image-6.9.7+bpo-cloud-amd64-unsigned - Linux 6.9 for x86-64 cloud
linux-image-6.9.7+bpo-rt-amd64-dbg - Debug symbols for
linux-image-6.9.7+bpo-rt-amd64
linux-image-6.9.7+bpo-rt-amd64-unsigned - Linux 6.9 for 64-bit PCs,
PREEMPT_RT
linux-image-6.9.7+bpo-amd64 - Linux 6.9 for 64-bit PCs (signed)
linux-image-6.9.7+bpo-cloud-amd64 - Linux 6.9 for x86-64 cloud (signed)
linux-image-6.9.7+bpo-rt-amd64 - Linux 6.9 for 64-bit PCs (signed),
PREEMPT_RT

2. Update the FAI server sources file for the Debian mirror.


*Default sources list:*
root at faiserver:~# cat /etc/apt/sources.list
# These lines should work for many sites

deb http://deb.debian.org/debian bookworm main contrib non-free
non-free-firmware
deb http://deb.debian.org/debian-security bookworm-security main contrib
non-free non-free-firmware
#deb http://deb.debian.org/debian bookworm-backports main contrib non-free
non-free-firmware

# repository that may contain newer fai packages for bookworm
deb http://fai-project.org/download bookworm koeln

*Updated sources file:*

root at faiserver:~/Debian/Manufacturing/fai/scripts# cat /etc/apt/sources.list
# These lines should work for many sites
deb http://deb.debian.org/debian bookworm main contrib non-free
non-free-firmware
deb http://deb.debian.org/debian-security bookworm-security main contrib
non-free non-free-firmware
deb http://deb.debian.org/debian bookworm-backports main contrib non-free
non-free-firmware
# repository that may contain newer fai packages for bookworm
deb http://fai-project.org/download bookworm koeln

# Debian Snapshot
deb [check-valid-until=no]
https://snapshot.debian.org/archive/debian/20240516T145256Z/
bookworm-backports main


3. Create a `99proxy` file under `/etc/apt/apt.conf.d/`:
Ref
https://snapshot.debian.org/#:~:text=To%20access%20snapshots%20of,update%20for%20this%20purpose
.

cat /etc/apt/apt.conf.d/99proxy
Acquire::https::snapshot.debian.org::Proxy "DIRECT";

4. Execute `apt-get update` to update the package lists.

5. Now I am able to get the available kernel versions from the remote site:
root at faiserver:~/Debian/Manufacturing/fai/scripts# apt-cache search
linux-image-6.5
linux-headers-6.5.0-0.deb12.4-amd64 - Header files for Linux
6.5.0-0.deb12.4-amd64
linux-headers-6.5.0-0.deb12.4-cloud-amd64 - Header files for Linux
6.5.0-0.deb12.4-cloud-amd64
linux-image-6.5.0-0.deb12.4-amd64-dbg - Debug symbols for
linux-image-6.5.0-0.deb12.4-amd64
linux-image-6.5.0-0.deb12.4-amd64-unsigned - Linux 6.5 for 64-bit PCs
linux-image-6.5.0-0.deb12.4-cloud-amd64-dbg - Debug symbols for
linux-image-6.5.0-0.deb12.4-cloud-amd64
linux-image-6.5.0-0.deb12.4-cloud-amd64-unsigned - Linux 6.5 for x86-64
cloud
linux-image-6.5.0-0.deb12.1-amd64 - Linux 6.5 for 64-bit PCs (signed)
linux-image-6.5.0-0.deb12.1-cloud-amd64 - Linux 6.5 for x86-64 cloud
(signed)
linux-image-6.5.0-0.deb12.1-rt-amd64 - Linux 6.5 for 64-bit PCs, PREEMPT_RT
(signed)
linux-image-6.5.0-0.deb12.4-amd64 - Linux 6.5 for 64-bit PCs (signed)
linux-image-6.5.0-0.deb12.4-cloud-amd64 - Linux 6.5 for x86-64 cloud
(signed)

root at faiserver:~/Debian/Manufacturing/fai/scripts# apt-cache search
linux-image-6.7
linux-headers-6.7.12+bpo-amd64 - Header files for Linux 6.7.12+bpo-amd64
linux-headers-6.7.12+bpo-cloud-amd64 - Header files for Linux
6.7.12+bpo-cloud-amd64
linux-headers-6.7.12+bpo-rt-amd64 - Header files for Linux
6.7.12+bpo-rt-amd64
linux-image-6.7.12+bpo-amd64-dbg - Debug symbols for
linux-image-6.7.12+bpo-amd64
linux-image-6.7.12+bpo-amd64-unsigned - Linux 6.7 for 64-bit PCs
linux-image-6.7.12+bpo-cloud-amd64-dbg - Debug symbols for
linux-image-6.7.12+bpo-cloud-amd64
linux-image-6.7.12+bpo-cloud-amd64-unsigned - Linux 6.7 for x86-64 cloud
linux-image-6.7.12+bpo-rt-amd64-dbg - Debug symbols for
linux-image-6.7.12+bpo-rt-amd64
linux-image-6.7.12+bpo-rt-amd64-unsigned - Linux 6.7 for 64-bit PCs,
PREEMPT_RT

6. We have a custom shell script to prepare the FAI server setup for client
installations.

Sample script to configure NFS server:
# Configure NFS Server

configure_nfs()
{
    /etc/init.d/nfs-kernel-server stop

    sed -i "s/linux-image-amd64/#/g" /etc/fai/NFSROOT
    sed -i '/AMD64/a\linux-image-6.7.12+bpo-amd64-unsigned' /etc/fai/NFSROOT

    run sed -i "s/NFSROOT_ETC_HOSTS/#NFSROOT_ETC_HOSTS/g"
/etc/fai/nfsroot.conf

    run sed --in-place '/bookworm-backports/d' /etc/fai/apt/sources.list
"echo 'deb http://deb.debian.org/debian bookworm-backports main contrib
non-free' | tee -a /etc/fai/apt/sources.list
}

Updated `/etc/fai/NFSROOT` file:
root at faiserver:~# cat /etc/fai/NFSROOT
# package list for creating the NFSROOT

PACKAGES install-norec FULL
nfs-common fai-nfsroot rpcbind
rsync
lshw procinfo
dump reiserfsprogs xfsprogs xfsdump btrfs-progs
hwinfo hdparm smartmontools nvme-cli
zile
numactl
udns-utils
netcat-traditional nmap
pxelinux syslinux-common
ca-certificates # for get-config-dir-https and apt-transport-https
usbutils pciutils
ssh
screen tmux
netselect
mdadm
cryptsetup
#git # git consumes a lot of disk space on the FAI CD

PACKAGES install-norec
# dracut replaces live-boot and initramfs-tools
dracut live-boot- initramfs-tools-
dracut-config-generic
dracut-network
curl lftp
less
ntpsec-ntpdate rdate
dosfstools
lvm2
psmisc
uuid-runtime
dialog
console-common kbd
xz-utils pigz zstd
gpg

# some network cards needs firmware
PACKAGES install-norec NONFREE
firmware-bnx2 firmware-bnx2x firmware-realtek
firmware-cavium firmware-myricom firmware-netronome
firmware-misc-nonfree

# you should not edit the lines below
# architecture dependent list of packages that are installed

PACKAGES install-norec I386
grub-pc
efibootmgr
linux-image-686-pae

PACKAGES install-norec AMD64
linux-image-6.7.12+bpo-amd64-unsigned
grub-pc
grub-efi-amd64-bin
efibootmgr
##/bookworm-backports # if you want to use a newer kernel

PACKAGES install-norec ARM64
grub-efi-arm64
efibootmgr
linux-image-arm64

PACKAGES install-norec ARMHF
grub-efi-arm
efibootmgr
linux-image-armmp

I've hardcoded the package name for `linux-image-6.7.12+bpo-amd64-unsigned`
in the `NFSROOT` file.
but it is still failing with the error: `E: Unable to locate package
linux-image-6.7.12+bpo-amd64-unsigned` but it is still available on
snapshot.
Classes are set to NFSROOT FULL AMD64 DEBIAN DEBIAN_12
Adding additional packages to /srv/fai/nfsroot:
nfs-common fai-nfsroot rpcbind rsync lshw procinfo dump reiserfsprogs
xfsprogs xfsdump btrfs-progs hwinfo hdparm smartmontools nvme-cli zile
numactl udns-utils netcat-traditional nmap pxelinux syslinux-common
ca-certificates usbutils pciutils ssh screen tmux netselect mdadm
cryptsetup dracut live-boot- initramfs-tools- dracut-config-generic
dracut-network curl lftp less ntpsec-ntpdate rdate dosfstools lvm2 psmisc
uuid-runtime dialog console-common kbd xz-utils pigz zstd gpg
linux-image-6.7.12+bpo-amd64-unsigned grub-pc grub-efi-amd64-bin efibootmgr
install_packages: reading config files from directory /etc/fai
install_packages: read config file NFSROOT
install_packages: executing chroot /srv/fai/nfsroot apt-get clean
install_packages: executing chroot /srv/fai/nfsroot apt-get -y -o
Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confnew
--allow-change-held-packages --fix-missing install --no-install-recommends
nfs-common fai-nfsroot rpcbind rsync lshw procinfo dump reiserfsprogs
xfsprogs xfsdump btrfs-progs hwinfo hdparm smartmontools nvme-cli zile
numactl udns-utils netcat-traditional nmap pxelinux syslinux-common
ca-certificates usbutils pciutils ssh screen tmux netselect mdadm
cryptsetup dracut live-boot- initramfs-tools- dracut-config-generic
dracut-network curl lftp less ntpsec-ntpdate rdate dosfstools lvm2 psmisc
uuid-runtime dialog console-common kbd xz-utils pigz zstd gpg
linux-image-6.7.12+bpo-amd64-unsigned grub-pc grub-efi-amd64-bin efibootmgr
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package linux-image-6.7.12+bpo-amd64-unsigned
E: Couldn't find any package by glob 'linux-image-6.7.12+bpo-amd64-unsigned'
E: Couldn't find any package by regex
'linux-image-6.7.12+bpo-amd64-unsigned'
ERROR: 25600 25600
ERROR: chroot /srv/fai/nfsroot apt-get -y -o
Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confnew
--allow-change-held-packages --fix-missing install --no-install-recommends
 nfs-common fai-nfsroot rpcbind rsync lshw procinfo dump reiserfsprogs
xfsprogs xfsdump btrfs-progs hwinfo hdparm smartmontools nvme-cli zile
numactl udns-utils netcat-traditional nmap pxelinux syslinux-common
ca-certificates usbutils pciutils ssh screen tmux netselect mdadm
cryptsetup dracut live-boot- initramfs-tools- dracut-config-generic
dracut-network curl lftp less ntpsec-ntpdate rdate dosfstools lvm2 psmisc
uuid-runtime dialog console-common kbd xz-utils pigz zstd gpg
linux-image-6.7.12+bpo-amd64-unsigned grub-pc grub-efi-amd64-bin efibootmgr
return code 100
install_packages: executing chroot /srv/fai/nfsroot apt-get clean
install_packages: executing chroot /srv/fai/nfsroot dpkg --configure
--pending
install_packages: executing chroot /srv/fai/nfsroot dpkg -C
install_packages: executing chroot /srv/fai/nfsroot apt-get clean
1 errors during executing of install_packages
ERROR: install_packages had exit code: 3


However, if I try it manually, it is found:
root at faiserver:~/Debian/Manufacturing/fai/scripts# apt-get -y -o
Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confnew
--allow-change-held-packages --fix-missing install --no-install-recommends
linux-image-6.7.12+bpo-amd64-unsigned


Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Suggested packages:
  linux-doc-6.7 debian-kernel-handbook
Recommended packages:
  firmware-linux-free apparmor
The following NEW packages will be installed:
  linux-image-6.7.12+bpo-amd64-unsigned
0 upgraded, 1 newly installed, 0 to remove and 56 not upgraded.
Need to get 99.8 MB of archives.
After this operation, 103 MB of additional disk space will be used.
Get:1 https://snapshot.debian.org/archive/debian/20240516T145256Z
bookworm-backports/main amd64 linux-image-6.7.12+bpo-amd64-unsigned amd64
6.7.12-1~bpo12+1 [99.8 MB]
Fetched 99.8 MB in 17s (5,744 kB/s)
Selecting previously unselected package
linux-image-6.7.12+bpo-amd64-unsigned.
(Reading database ... 25696 files and directories currently installed.)
Preparing to unpack
.../linux-image-6.7.12+bpo-amd64-unsigned_6.7.12-1~bpo12+1_amd64.deb ...
Unpacking linux-image-6.7.12+bpo-amd64-unsigned (6.7.12-1~bpo12+1) ...
Setting up linux-image-6.7.12+bpo-amd64-unsigned (6.7.12-1~bpo12+1) ...
I: /vmlinuz is now a symlink to boot/vmlinuz-6.7.12+bpo-amd64
I: /initrd.img is now a symlink to boot/initrd.img-6.7.12+bpo-amd64
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-6.7.12+bpo-amd64
I: The initramfs will attempt to resume from /dev/sda5
I: (UUID=663da36d-e9d7-4559-bbb4-a2d5141bd99c)
I: Set the RESUME variable to override this.
/etc/kernel/postinst.d/zz-update-grub:
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.7.12+bpo-amd64
Found initrd image: /boot/initrd.img-6.7.12+bpo-amd64
Found linux image: /boot/vmlinuz-6.1.0-18-amd64
Found initrd image: /boot/initrd.img-6.1.0-18-amd64
Found memtest86+x64 image: /boot/memtest86+x64.bin
done

I suspect there may be an issue with the sources list because my
`/etc/fai/apt/sources.list` might not be updated correctly. I am still
debugging this further. If anyone has a better suggestion for obtaining
older kernel versions for client installations or any other alternative
approach, please share it with me.

Thanks in advance.


Regards.
Samadhan
India
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.uni-koeln.de/pipermail/linux-fai/attachments/20240729/bb8b54d0/attachment-0001.html>