fai-mirror for non-fai-system architectures (bug)

[C. V.]

Hi,

I am configuring a network with FAI since a couple of months. For this I am using Debian AMD64 as FAI server to be able to build also i386 NFSROOT. This all works more or less fine and I have my two NFSROOTs.

I also use fai-mirror to build a minimal Debian mirror. This way I am able to install all systems even without having internet. Generating the AMD64 mirror is not a problem using the standard command. The problem occurred when I tried to generate an i386 mirror.
I searched for help and found the wiki page

http://wiki.fai-project.org/index.php/FAI_How_to_build_a_i386_and_x86_64_compatible_FAI_server_on_Debian_Etch_x86_64

and tried it the described way. It worked, I thought at least. After I tried to install a i386 System, it failed because it did not find most or all of the packages. I found out, that the dist path of the mirror only had the binary-amd64 paths (sorry it that might not be totally true, because it was some time ago). Fact was that the System did not install completely because many packages were missing.
The way described in wiki page to generate the two mirrors is based on the ability of ''apt'' to "source" options:

APT {
  Architecture "i386";
}

I looked into the ''fai-mirror'' code and even though it should have worked, maybe the way of how ''fai-mirror'' executes apt(itude) prevents this usage.

Looking for the problem, I found it, I guess, in ''fai-mirror''. If the option ''-a'' is not given, the script uses the system architecture as default (and therefor probably ignores/overwrites the apt value).

Problem solved? No...

I used the ''-a'' with i386 and the mirror was build correctly, but still a problem remained: Some most packages installed but some were missing, eg. the kernel. I found that the kernel was there but apt was looking for a slightly different version of the available one. Apt wanted to install an older version but the newest was available on my mirror.
So, again digging into ''fai-mirror'' again...
I found the problem in the last lines of code.

fai-mirror original (last lines):

# create mirror directory structure
echo "Calling apt-move"
cat > $aptmovefile <<EOF   # generate apt-move.conf
APTSITES=*
LOCALDIR=$mirrordir
DIST=$debdist
FILECACHE=$archivedir
LISTSTATE=$aptcache/var/lib/apt/lists
DELETE=no
CONTENTS=no
PKGCOMP='none gzip'
EOF
# due to bug #441231 in apt-move, we have to specify the arhcitecture
[ -n "$arch" ] && echo "ARCH=$arch" >> $aptmovefile # append architecture
apt-move $qflag -c $aptmovefile update
# since Packages.gz from apt-move does not include packages from my
# repository, let's use apt-ftparchive for generiating correct index
# files
pfilegz=$(find $mirrordir/dists -name Packages.gz)
pfile=$(find $mirrordir/dists -name Packages)
pdist=$(cd $mirrordir/dists ; ls)
cd $mirrordir
# md5sums of apt-move are not valid, when we recreate Packages.gz using
# apt-ftparchive, but we can use the header of the Release file
grep -B99 MD5Sum:  $mirrordir/dists/$pdist/Release | grep -v MD5Sum: > $mirrordir/tmpfile
rm $mirrordir/dists/$pdist/Release
apt-ftparchive packages pool > $pfile
gzip -c $pfile > $pfilegz
apt-ftparchive release dists/$pdist >> tmpfile
mv tmpfile dists/$pdist/Release

echo "$0 finished."
echo -n "Mirror size and location: ";du -sh $mirrordir
cleandirs

You can see that ''find'' is called several times. That causes problems if ''find'' finds more than one match. And it does that in some cases because not all packages belong to i386. Specially the kernel of the AMD64 Systems is not included in the i386 binary-tree. But since it is included (as a standard) the ''package_config/DEFAULT'' ''fai-mirror'' downloads it anyway.
So, the execution of the following statements fail (without breaking the fai-mirror script) and prevent generating the new Release,Package,Package.gz files. This causes the described problem.

Now, finally after all this description, I present you my solution. It works, but it might be not perfect. The benefit should also be that you only need to build the mirror with the following command and get a combined i386 and AMD64 mirror (but I did not yet verify this due to time issues):

fai-mirror -v -a i386 /media/fai/debmirror

fai-mirror my version (last lines):

# create mirror directory structure
echo "Calling apt-move"
cat > $aptmovefile <<EOF   # generate apt-move.conf
APTSITES=*
LOCALDIR=$mirrordir
DIST=$debdist
FILECACHE=$archivedir
LISTSTATE=$aptcache/var/lib/apt/lists
DELETE=no
CONTENTS=no
PKGCOMP='none gzip'
EOF
# due to bug #441231 in apt-move, we have to specify the arhcitecture
[ -n "$arch" ] && echo "ARCH=$arch" >> $aptmovefile # append architecture
apt-move $qflag -c $aptmovefile update
# since Packages.gz from apt-move does not include packages from my
# repository, let's use apt-ftparchive for generiating correct index
# files
pfile=$(find $mirrordir/dists -name Packages)
pdist=$(cd $mirrordir/dists ; ls)
cd $mirrordir
for pdist_item in $pdist; do
  # md5sums of apt-move are not valid, when we recreate Packages.gz using
  # apt-ftparchive, but we can use the header of the Release file
  grep -B99 MD5Sum:  $mirrordir/dists/$pdist_item/Release | grep -v MD5Sum: > $mirrordir/tmpfile
  rm $mirrordir/dists/$pdist_item/Release
  
  for pfile_item in $pfile; do
    apt-ftparchive packages pool > $pfile_item
    gzip -c $pfile_item > ${pfile_item}.gz
  done
  
  apt-ftparchive release dists/$pdist_item >> tmpfile
  mv tmpfile dists/$pdist_item/Release
done

echo "$0 finished."
echo -n "Mirror size and location: ";du -sh $mirrordir
cleandirs



DIFF version: diff -Naur /root/fai-mirror*

--- /root/fai-mirror    2010-10-05 20:42:49.000000000 +0200
+++ /root/fai-mirror.corrected  2010-10-05 21:06:44.000000000 +0200
@@ -297,18 +297,23 @@
 # since Packages.gz from apt-move does not include packages from my
 # repository, let's use apt-ftparchive for generiating correct index
 # files
-pfilegz=$(find $mirrordir/dists -name Packages.gz)
 pfile=$(find $mirrordir/dists -name Packages)
 pdist=$(cd $mirrordir/dists ; ls)
 cd $mirrordir
-# md5sums of apt-move are not valid, when we recreate Packages.gz using
-# apt-ftparchive, but we can use the header of the Release file
-grep -B99 MD5Sum:  $mirrordir/dists/$pdist/Release | grep -v MD5Sum: > $mirrordir/tmpfile
-rm $mirrordir/dists/$pdist/Release
-apt-ftparchive packages pool > $pfile
-gzip -c $pfile > $pfilegz
-apt-ftparchive release dists/$pdist >> tmpfile
-mv tmpfile dists/$pdist/Release
+for pdist_item in $pdist; do
+  # md5sums of apt-move are not valid, when we recreate Packages.gz using
+  # apt-ftparchive, but we can use the header of the Release file
+  grep -B99 MD5Sum:  $mirrordir/dists/$pdist_item/Release | grep -v MD5Sum: > $mirrordir/tmpfile
+  rm $mirrordir/dists/$pdist_item/Release
+  
+  for pfile_item in $pfile; do
+    apt-ftparchive packages pool > $pfile_item
+    gzip -c $pfile_item > ${pfile_item}.gz
+  done
+  
+  apt-ftparchive release dists/$pdist_item >> tmpfile
+  mv tmpfile dists/$pdist_item/Release
+done
 
 echo "$0 finished."
 echo -n "Mirror size and location: ";du -sh $mirrordir


Ok, this was a long mail and I hereby want to put this to discussion.
I also did not change the wiki page since someone obviously did not have the problem.

Greets,

C.V.