<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">We are using FAI tools to install Debian10 Buster and Debian12 Bookworm clients with backported kernel versions.<br><br>We use the latest <a href="https://fai-project.org/fai-cd/faicd64-small_6.2.2.iso">FAI CD ISO 6.2</a> for Debian 12 Bookworm clients.<div>We installed Debian12 with dedicated separate kernel versions using backported kernels 6.5 and 6.7<br><span style="font-family:Monaco,Menlo,Consolas,"Courier New",monospace;font-size:12px;font-variant-ligatures:none;background-color:rgba(232,232,232,0.04)"><font color="#000000">linux-image-6.5.0-0.deb12.4-amd64
linux-image-6.7.12+bpo-amd64
</font></span><br>Everything was working fine until last week.<br><br>As you know, backported kernels 6.5 and 6.7 have been removed from the official Debian repositories and replaced with the new backported kernel 6.9, which is in testing. <a href="https://packages.debian.org/bookworm-backports/kernel/linux-image-6.9.7+bpo-amd64">Debian -- Details of package linux-image-6.9.7+bpo-amd64 in bookworm-backports</a><br><br>To manage these backported features in FAI, we created shell scripts. Since the backported kernels are no longer available on the remote site, we used the Debian Snapshot <a href="https://snapshot.debian.org/">snapshot.debian.org</a> to retrieve old kernel packages based on dates and version numbers. I have verified this manually and was able to get the required older kernel versions on the FAI server.<br><br><b>Here is what I have tried to achieve this:<br></b><br>1. As backported kernel versions are removed from the remote, we cannot get kernels 6.5 and 6.7, only the latest kernel 6.9 is able to get.<br><div style="color:rgb(204,204,204);background-color:rgb(31,31,31);font-family:Consolas,"Courier New",monospace;font-size:14px;line-height:19px;white-space:pre"><div>root@faiserver:~# apt-cache search linux-image-6.5</div><div>root@faiserver:~# apt-cache search linux-image-6.7</div><div><br></div><div>root@faiserver:~# apt-cache search linux-image-6.9</div><div>linux-headers-6.9.7+bpo-amd64 - Header files for Linux 6.9.7+bpo-amd64</div><div>linux-headers-6.9.7+bpo-cloud-amd64 - Header files for Linux 6.9.7+bpo-cloud-amd64</div><div>linux-headers-6.9.7+bpo-rt-amd64 - Header files for Linux 6.9.7+bpo-rt-amd64</div><div>linux-image-6.9.7+bpo-amd64-dbg - Debug symbols for linux-image-6.9.7+bpo-amd64</div><div>linux-image-6.9.7+bpo-amd64-unsigned - Linux 6.9 for 64-bit PCs</div><div>linux-image-6.9.7+bpo-cloud-amd64-dbg - Debug symbols for linux-image-6.9.7+bpo-cloud-amd64</div><div>linux-image-6.9.7+bpo-cloud-amd64-unsigned - Linux 6.9 for x86-64 cloud</div><div>linux-image-6.9.7+bpo-rt-amd64-dbg - Debug symbols for linux-image-6.9.7+bpo-rt-amd64</div><div>linux-image-6.9.7+bpo-rt-amd64-unsigned - Linux 6.9 for 64-bit PCs, PREEMPT_RT</div><div>linux-image-6.9.7+bpo-amd64 - Linux 6.9 for 64-bit PCs (signed)</div><div>linux-image-6.9.7+bpo-cloud-amd64 - Linux 6.9 for x86-64 cloud (signed)</div><div>linux-image-6.9.7+bpo-rt-amd64 - Linux 6.9 for 64-bit PCs (signed), PREEMPT_RT</div></div></div><div><br>2. Update the FAI server sources file for the Debian mirror.<br><br><b>Default sources list:<br></b><br><font face="monospace">root@faiserver:~# cat /etc/apt/sources.list<br># These lines should work for many sites<br><br>deb <a href="http://deb.debian.org/debian">http://deb.debian.org/debian</a> bookworm main contrib non-free non-free-firmware<br>deb <a href="http://deb.debian.org/debian-security">http://deb.debian.org/debian-security</a> bookworm-security main contrib non-free non-free-firmware<br>#deb <a href="http://deb.debian.org/debian">http://deb.debian.org/debian</a> bookworm-backports main contrib non-free non-free-firmware<br><br># repository that may contain newer fai packages for bookworm<br>deb <a href="http://fai-project.org/download">http://fai-project.org/download</a> bookworm koeln</font><br><br><b>Updated sources file:</b><br><br>root@faiserver:~/Debian/Manufacturing/fai/scripts# cat /etc/apt/sources.list</div><div><font face="monospace"># These lines should work for many sites<br>deb <a href="http://deb.debian.org/debian">http://deb.debian.org/debian</a> bookworm main contrib non-free non-free-firmware<br>deb <a href="http://deb.debian.org/debian-security">http://deb.debian.org/debian-security</a> bookworm-security main contrib non-free non-free-firmware<br>deb <a href="http://deb.debian.org/debian">http://deb.debian.org/debian</a> bookworm-backports main contrib non-free non-free-firmware<br># repository that may contain newer fai packages for bookworm<br>deb <a href="http://fai-project.org/download">http://fai-project.org/download</a> bookworm koeln<br><br># Debian Snapshot<br>deb [check-valid-until=no] <a href="https://snapshot.debian.org/archive/debian/20240516T145256Z/">https://snapshot.debian.org/archive/debian/20240516T145256Z/</a> bookworm-backports main<br></font><br><br>3. Create a `99proxy` file under `/etc/apt/apt.conf.d/`: <br>Ref <a href="https://snapshot.debian.org/#:~:text=To%20access%20snapshots%20of,update%20for%20this%20purpose">https://snapshot.debian.org/#:~:text=To%20access%20snapshots%20of,update%20for%20this%20purpose</a>.</div><div><br><font face="monospace">cat /etc/apt/apt.conf.d/99proxy<br>Acquire::https::snapshot.debian.org::Proxy "DIRECT";<br><br></font></div><div>4. Execute `apt-get update` to update the package lists.<br><br>5. Now I am able to get the available kernel versions from the remote site:</div><div><div style="color:rgb(204,204,204);background-color:rgb(31,31,31);font-family:Consolas,"Courier New",monospace;font-size:14px;line-height:19px;white-space:pre"><div>root@faiserver:~/Debian/Manufacturing/fai/scripts# apt-cache search linux-image-6.5</div><div>linux-headers-6.5.0-0.deb12.4-amd64 - Header files for Linux 6.5.0-0.deb12.4-amd64</div><div>linux-headers-6.5.0-0.deb12.4-cloud-amd64 - Header files for Linux 6.5.0-0.deb12.4-cloud-amd64</div><div>linux-image-6.5.0-0.deb12.4-amd64-dbg - Debug symbols for linux-image-6.5.0-0.deb12.4-amd64</div><div>linux-image-6.5.0-0.deb12.4-amd64-unsigned - Linux 6.5 for 64-bit PCs</div><div>linux-image-6.5.0-0.deb12.4-cloud-amd64-dbg - Debug symbols for linux-image-6.5.0-0.deb12.4-cloud-amd64</div><div>linux-image-6.5.0-0.deb12.4-cloud-amd64-unsigned - Linux 6.5 for x86-64 cloud</div><div>linux-image-6.5.0-0.deb12.1-amd64 - Linux 6.5 for 64-bit PCs (signed)</div><div>linux-image-6.5.0-0.deb12.1-cloud-amd64 - Linux 6.5 for x86-64 cloud (signed)</div><div>linux-image-6.5.0-0.deb12.1-rt-amd64 - Linux 6.5 for 64-bit PCs, PREEMPT_RT (signed)</div><div>linux-image-6.5.0-0.deb12.4-amd64 - Linux 6.5 for 64-bit PCs (signed)</div><div>linux-image-6.5.0-0.deb12.4-cloud-amd64 - Linux 6.5 for x86-64 cloud (signed)</div><div><br></div><div>root@faiserver:~/Debian/Manufacturing/fai/scripts# apt-cache search linux-image-6.7</div><div>linux-headers-6.7.12+bpo-amd64 - Header files for Linux 6.7.12+bpo-amd64</div><div>linux-headers-6.7.12+bpo-cloud-amd64 - Header files for Linux 6.7.12+bpo-cloud-amd64</div><div>linux-headers-6.7.12+bpo-rt-amd64 - Header files for Linux 6.7.12+bpo-rt-amd64</div><div>linux-image-6.7.12+bpo-amd64-dbg - Debug symbols for linux-image-6.7.12+bpo-amd64</div><div>linux-image-6.7.12+bpo-amd64-unsigned - Linux 6.7 for 64-bit PCs</div><div>linux-image-6.7.12+bpo-cloud-amd64-dbg - Debug symbols for linux-image-6.7.12+bpo-cloud-amd64</div><div>linux-image-6.7.12+bpo-cloud-amd64-unsigned - Linux 6.7 for x86-64 cloud</div><div>linux-image-6.7.12+bpo-rt-amd64-dbg - Debug symbols for linux-image-6.7.12+bpo-rt-amd64</div><div>linux-image-6.7.12+bpo-rt-amd64-unsigned - Linux 6.7 for 64-bit PCs, PREEMPT_RT</div></div><br>6. We have a custom shell script to prepare the FAI server setup for client installations.<br><br>Sample script to configure NFS server:<br><div style="color:rgb(204,204,204);background-color:rgb(31,31,31);font-family:Consolas,"Courier New",monospace;font-size:14px;line-height:19px;white-space:pre"><div><span style="color:rgb(106,153,85)"># Configure NFS Server</span></div><div><span style="color:rgb(106,153,85)"><br></span></div><div><span style="color:rgb(220,220,170)">configure_nfs</span>()</div><div>{</div><div> <span style="color:rgb(220,220,170)">/etc/init.d/nfs-kernel-server</span> <span style="color:rgb(206,145,120)">stop</span></div><br><div> <span style="color:rgb(206,145,120)">sed</span> <span style="color:rgb(86,156,214)">-i</span> <span style="color:rgb(206,145,120)">"s/linux-image-amd64/#/g"</span> <span style="color:rgb(206,145,120)">/etc/fai/NFSROOT</span></div><div> s<span style="color:rgb(206,145,120)">ed</span> <span style="color:rgb(86,156,214)">-i</span> <span style="color:rgb(206,145,120)">'/AMD64/a\linux-image-6.7.12+bpo-amd64-unsigned' </span><span style="color:rgb(206,145,120)">/etc/fai/NFSROOT</span></div><div> </div><div> <span style="color:rgb(220,220,170)">run</span> <span style="color:rgb(206,145,120)">sed</span> <span style="color:rgb(86,156,214)">-i</span> <span style="color:rgb(206,145,120)">"s/NFSROOT_ETC_HOSTS/#NFSROOT_ETC_HOSTS/g"</span> <span style="color:rgb(206,145,120)">/etc/fai/nfsroot.conf</span></div><div><br></div><div> <span style="color:rgb(220,220,170)">run</span> <span style="color:rgb(206,145,120)">sed</span> <span style="color:rgb(86,156,214)">--in-place</span> <span style="color:rgb(206,145,120)">'/bookworm-backports/d'</span> <span style="color:rgb(206,145,120)">/etc/fai/apt/sources.list</span></div><div><span style="color:rgb(206,145,120)"> "echo 'deb <a href="http://deb.debian.org/debian">http://deb.debian.org/debian</a> bookworm-backports main contrib non-free' | tee -a </span><span style="color:rgb(206,145,120)">/etc/fai/apt/sources.list</span></div><div><span style="color:rgb(206,145,120)">}</span></div></div><br>Updated `/etc/fai/NFSROOT` file:<br>root@faiserver:~# cat /etc/fai/NFSROOT<br><div style="color:rgb(204,204,204);background-color:rgb(31,31,31);font-family:Consolas,"Courier New",monospace;font-size:14px;line-height:19px;white-space:pre"><div><span style="color:rgb(86,156,214);font-weight:bold"># package list for creating the NFSROOT</span></div><br><div>PACKAGES install-norec FULL</div><div>nfs-common fai-nfsroot rpcbind</div><div>rsync</div><div>lshw procinfo</div><div>dump reiserfsprogs xfsprogs xfsdump btrfs-progs</div><div>hwinfo hdparm smartmontools nvme-cli</div><div>zile</div><div>numactl</div><div>udns-utils</div><div>netcat-traditional nmap</div><div>pxelinux syslinux-common</div><div>ca-certificates # for get-config-dir-https and apt-transport-https</div><div>usbutils pciutils</div><div>ssh</div><div>screen tmux</div><div>netselect</div><div>mdadm</div><div>cryptsetup</div><div>#git # git consumes a lot of disk space on the FAI CD</div><br><div>PACKAGES install-norec</div><div><span style="color:rgb(86,156,214);font-weight:bold"># dracut replaces live-boot and initramfs-tools</span></div><div>dracut live-boot- initramfs-tools-</div><div>dracut-config-generic</div><div>dracut-network</div><div>curl lftp</div><div>less</div><div>ntpsec-ntpdate rdate</div><div>dosfstools</div><div>lvm2</div><div>psmisc</div><div>uuid-runtime</div><div>dialog</div><div>console-common kbd</div><div>xz-utils pigz zstd</div><div>gpg</div><br><div><span style="color:rgb(86,156,214);font-weight:bold"># some network cards needs firmware</span></div><div>PACKAGES install-norec NONFREE</div><div>firmware-bnx2 firmware-bnx2x firmware-realtek</div><div>firmware-cavium firmware-myricom firmware-netronome</div><div>firmware-misc-nonfree</div><br><div><span style="color:rgb(86,156,214);font-weight:bold"># you should not edit the lines below</span></div><div><span style="color:rgb(86,156,214);font-weight:bold"># architecture dependent list of packages that are installed</span></div><br><div>PACKAGES install-norec I386</div><div>grub-pc</div><div>efibootmgr</div><div>linux-image-686-pae</div><br><div>PACKAGES install-norec AMD64</div><div>linux-image-6.7.12+bpo-amd64-unsigned</div><div>grub-pc</div><div>grub-efi-amd64-bin</div><div>efibootmgr</div><div>##/bookworm-backports # if you want to use a newer kernel</div><br><div>PACKAGES install-norec ARM64</div><div>grub-efi-arm64</div><div>efibootmgr</div><div>linux-image-arm64</div><br><div>PACKAGES install-norec ARMHF</div><div>grub-efi-arm</div><div>efibootmgr</div><div>linux-image-armmp</div></div></div><div><br>I've hardcoded the package name for `linux-image-6.7.12+bpo-amd64-unsigned` in the `NFSROOT` file. </div><div>but it is still failing with the error: `E: Unable to locate package linux-image-6.7.12+bpo-amd64-unsigned` but it is still available on snapshot.</div><div><div style="color:rgb(204,204,204);background-color:rgb(31,31,31);font-family:Consolas,"Courier New",monospace;font-size:14px;line-height:19px;white-space:pre"><div>Classes are set to NFSROOT FULL AMD64 DEBIAN DEBIAN_12</div><div>Adding additional packages to /srv/fai/nfsroot:</div><div>nfs-common fai-nfsroot rpcbind rsync lshw procinfo dump reiserfsprogs xfsprogs xfsdump btrfs-progs hwinfo hdparm smartmontools nvme-cli zile numactl udns-utils netcat-traditional nmap pxelinux syslinux-common ca-certificates usbutils pciutils ssh screen tmux netselect mdadm cryptsetup dracut live-boot- initramfs-tools- dracut-config-generic dracut-network curl lftp less ntpsec-ntpdate rdate dosfstools lvm2 psmisc uuid-runtime dialog console-common kbd xz-utils pigz zstd gpg linux-image-6.7.12+bpo-amd64-unsigned grub-pc grub-efi-amd64-bin efibootmgr</div><div>install_packages: reading config files from directory /etc/fai</div><div>install_packages: read config file NFSROOT</div><div>install_packages: executing chroot /srv/fai/nfsroot apt-get clean</div><div>install_packages: executing chroot /srv/fai/nfsroot apt-get -y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confnew --allow-change-held-packages --fix-missing install --no-install-recommends nfs-common fai-nfsroot rpcbind rsync lshw procinfo dump reiserfsprogs xfsprogs xfsdump btrfs-progs hwinfo hdparm smartmontools nvme-cli zile numactl udns-utils netcat-traditional nmap pxelinux syslinux-common ca-certificates usbutils pciutils ssh screen tmux netselect mdadm cryptsetup dracut live-boot- initramfs-tools- dracut-config-generic dracut-network curl lftp less ntpsec-ntpdate rdate dosfstools lvm2 psmisc uuid-runtime dialog console-common kbd xz-utils pigz zstd gpg linux-image-6.7.12+bpo-amd64-unsigned grub-pc grub-efi-amd64-bin efibootmgr</div><div>Reading package lists...</div><div>Building dependency tree...</div><div>Reading state information...</div><div>E: Unable to locate package linux-image-6.7.12+bpo-amd64-unsigned</div><div>E: Couldn't find any package by glob 'linux-image-6.7.12+bpo-amd64-unsigned'</div><div>E: Couldn't find any package by regex 'linux-image-6.7.12+bpo-amd64-unsigned'</div><div>ERROR: 25600 25600</div><div>ERROR: chroot /srv/fai/nfsroot apt-get -y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confnew --allow-change-held-packages --fix-missing install --no-install-recommends nfs-common fai-nfsroot rpcbind rsync lshw procinfo dump reiserfsprogs xfsprogs xfsdump btrfs-progs hwinfo hdparm smartmontools nvme-cli zile numactl udns-utils netcat-traditional nmap pxelinux syslinux-common ca-certificates usbutils pciutils ssh screen tmux netselect mdadm cryptsetup dracut live-boot- initramfs-tools- dracut-config-generic dracut-network curl lftp less ntpsec-ntpdate rdate dosfstools lvm2 psmisc uuid-runtime dialog console-common kbd xz-utils pigz zstd gpg linux-image-6.7.12+bpo-amd64-unsigned grub-pc grub-efi-amd64-bin efibootmgr return code 100</div><div>install_packages: executing chroot /srv/fai/nfsroot apt-get clean</div><div>install_packages: executing chroot /srv/fai/nfsroot dpkg --configure --pending</div><div>install_packages: executing chroot /srv/fai/nfsroot dpkg -C</div><div>install_packages: executing chroot /srv/fai/nfsroot apt-get clean</div><div>1 errors during executing of install_packages</div><div>ERROR: install_packages had exit code: 3</div></div></div><div> <br><br>However, if I try it manually, it is found:<br><div style="color:rgb(204,204,204);background-color:rgb(31,31,31);font-family:Consolas,"Courier New",monospace;font-size:14px;line-height:19px;white-space:pre"><div>root@faiserver:~/Debian/Manufacturing/fai/scripts# apt-get -y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confnew</div><div>--allow-change-held-packages --fix-missing install --no-install-recommends linux-image-6.7.12+bpo-amd64-unsigned </div><div>Reading package lists... Done</div><div>Building dependency tree... Done</div><div>Reading state information... Done</div><div>Suggested packages:</div><div> linux-doc-6.7 debian-kernel-handbook</div><div>Recommended packages:</div><div> firmware-linux-free apparmor</div><div>The following NEW packages will be installed:</div><div> linux-image-6.7.12+bpo-amd64-unsigned</div><div>0 upgraded, 1 newly installed, 0 to remove and 56 not upgraded.</div><div>Need to get 99.8 MB of archives.</div><div>After this operation, 103 MB of additional disk space will be used.</div><div>Get:1 <a href="https://snapshot.debian.org/archive/debian/20240516T145256Z">https://snapshot.debian.org/archive/debian/20240516T145256Z</a> bookworm-backports/main amd64 linux-image-6.7.12+bpo-amd64-unsigned amd64 6.7.12-1~bpo12+1 [99.8 MB]</div><div>Fetched 99.8 MB in 17s (5,744 kB/s)</div><div>Selecting previously unselected package linux-image-6.7.12+bpo-amd64-unsigned.</div><div>(Reading database ... 25696 files and directories currently installed.)</div><div>Preparing to unpack .../linux-image-6.7.12+bpo-amd64-unsigned_6.7.12-1~bpo12+1_amd64.deb ...</div><div>Unpacking linux-image-6.7.12+bpo-amd64-unsigned (6.7.12-1~bpo12+1) ...</div><div>Setting up linux-image-6.7.12+bpo-amd64-unsigned (6.7.12-1~bpo12+1) ...</div><div>I: /vmlinuz is now a symlink to boot/vmlinuz-6.7.12+bpo-amd64</div><div>I: /initrd.img is now a symlink to boot/initrd.img-6.7.12+bpo-amd64</div><div>/etc/kernel/postinst.d/initramfs-tools:</div><div>update-initramfs: Generating /boot/initrd.img-6.7.12+bpo-amd64</div><div>I: The initramfs will attempt to resume from /dev/sda5</div><div>I: (UUID=663da36d-e9d7-4559-bbb4-a2d5141bd99c)</div><div>I: Set the RESUME variable to override this.</div><div>/etc/kernel/postinst.d/zz-update-grub:</div><div>Generating grub configuration file ...</div><div>Found linux image: /boot/vmlinuz-6.7.12+bpo-amd64</div><div>Found initrd image: /boot/initrd.img-6.7.12+bpo-amd64</div><div>Found linux image: /boot/vmlinuz-6.1.0-18-amd64</div><div>Found initrd image: /boot/initrd.img-6.1.0-18-amd64</div><div>Found memtest86+x64 image: /boot/memtest86+x64.bin</div><div>done</div></div><div><div><br></div><div>I suspect there may be an issue with the sources list because my `/etc/fai/apt/sources.list` might not be updated correctly. I am still debugging this further. If anyone has a better suggestion for obtaining older kernel versions for client installations or any other alternative approach, please share it with me.</div><div><br></div><div>Thanks in advance.</div><div><br><br>Regards.</div></div></div><div>Samadhan</div><div>India</div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>