Future of FAI (fwd)

[Niall Young]

It's a bit of a rant, comments welcome but I don't mean to clog up the
list if this isn't the best place for this:

Niall Young                                    Chime Communications Pty Ltd
niall at chime.net.au                            Level 6, 263 Adelaide Terrace
Ph: 08 9213 1330 / 0408 192 797               Perth, Western Australia 6000

     "there's a lot of movement in my trousers at the moment"
                                     -- Dennis Kristofich, Sep 2002

---------- Forwarded message ----------
Date: Thu, 21 Nov 2002 16:28:30 +0800 (WST)
From: Niall Young <niall at chime.net.au>
To: Thomas Lange <lange at informatik.uni-koeln.de>
Subject: Future of FAI

Hi Thomas,

I'm about to spend the next 2-3 weeks working on FAI.  I've got a basic
NFS/floppy install working and my goal is a bootable CD like Marc
Schaefer's customisation.  I really want to contrribute everything back
into FAI though, I have conditional approval to GPL everything I add -
this is for internal use at work.  I've got a lot of time and resources
for this as I want to make sure it's implemented and documented
properly.

I hope you don't mind me ranting, but I'm going to describe what I'm
about to start work on - you may have suggestions or alternatives on
how it should be implemented to maintain consistency.  I'd really
appreciate feedback if you have time.

Before that though, I've found it hard work to understand how the class
concept works and how to define new classes.  Obviously classes are arbitrarily
flexible, in particular being able to dynamically define new classes and run
any kind of script imaginable.  This is a lot of complexity for a newbie,
especially without much documentation available.  If I have time, I'll write up
some documentation for you going into more detail on how the class concept
works with lots of examples, a serious HOWTO would have helped out a
lot here.  Not everyone will be able to read through the source and figure it
out.

Secondly, I think it would be beneficial to group all of the class variables,
scripts, definitions etc. all in the one tree.  As FAI matures and missing
links like debconf seeding are included, people are going to want to swap and
share their classes.  At the moment it's a bit of a mess - it would be easier
to package them all up in a single directory per class, even read them in as
either a directory CLASS or an equivalent CLASS.tar.[gz|bz2] tarball which is
decompressed as required.  Eventually it'd be nice to see a collection
of popular classes all put into a fai-classes package for distribution.

Thirdly, as people come up with good ideas in their classes, more and
more of these common mechanisms (like files/packages/ - see below) could
be incorporated into the class definition itself and not in the scripts.
i.e. CLASS/dselect-upgrade.dpkg etc.  They can do whatever they want
already, but re-using common mechanisms and making useful stuff a
standardised 'feature' of each class will make it easier to come up with
new classes instead of writing a lot of logic in scripts/*

Ok: my aim is a bootable CD.  I've got specific requirements on top of
this as it will be used by several different departments, so it's got to
act in standalone mode with no network, and several variations of
network mode where package updates are retrieved or class metadata is
retrieved through a variety of different methods (MySQL query, wget
etc.).  Most of that's specific to our internal needs and I can put most
of this logic into our internal classes, the rest needs to be as generic
as possible so it's useful to the greatest number of users and to
increase the chance I can release non-internal stuff under the GPL.  Basically:

    -	Bootable CD

	    I need to leave the CD in the drive, so I've already got a
	    proof of concept for a Grub fallback, boot off the disk if
	    it's partitioned and the kernel can be found, otherwise boot
	    from CD and do a new installation and reboot.  This is mainly for our
	    environment, but I think others would also find it useful.

	    I'd like to re-use as much of the make-fai-* scripts as I
	    can, make-fai-bootfloppy as the boot image for the CD and
	    write a new make-fai-bootcd calling fai-setup, make-fai-nfsroot
	    etc.  I'd like to let people pick whether they want to build
	    a Grub or Lilo boot image, and if they want to pass a config
	    too or pick from a few basic types (the fallback one, or one
	    where the CD must be removed etc.)

    -	Packages on CD

	    I need to store a subset of a Debian mirror on the CD.
	    DEFAULT.var and DEFAULT/S01 are a good example, but it's
	    a bit inflexible if every class needs to define their
	    own list and mechanisms.  I'd like to add a real repository
	    to the NFSROOT/CDROM, I haven't looked at mkdebmirror since
	    early last year but I'd like to add logic so you can pass
	    a list of packages and have only the required and dependency
	    packages available, keep the disk usage minimal.  I think a
	    standard mechanism for all classes, say
	    CLASS/dselect-upgrade.dpkg which is passed to `apt-get
	    dselect-upgrade` would be useful instead of having to
	    separately list them in the .var, write your own block to
	    process them and potentially have a very large
	    files/packages/  If we can combine files/packages/ with the
	    Debian mirror (subset or whole mirror, user preference) then
	    it simplifies things a lot.  People can then find a class
	    they like, derive from it and then add a list of the extra
	    packages (and later on debconf data) that they want to add.
	    I'll probably look into doing `apt-get clean; apt-get
	    --download-only ...; apt-move ...` etc. to get the subset,
	    maybe a commandline option would get an entire mirror using
	    mkdebmirror/mirror or whatever works).

	    It also makes my life easier by putting everything into the
	    NFSROOT as a standard mechanism and basing the CDROM filesystem on
	    this.

    -	Generate CDROM image

	    I'm going to look in more detail, but I'm thinking that
	    adding a few variables and some more conditional logic into
	    rcS_fai/subroutines* would be a better approach instead of putting
	    all of this into a class.  Expanding the core install methods/media
	    should be pretty useful to everyone.  I want to re-use as
	    much of make-fai-nfsroot as I can, but perhaps it needs to
	    be modified and the other install scripts to detect if it's
	    an NFSROOT or CDROM or potentially other media...

Comments?  Any ways you'd prefer these were implemented in FAI?

    -	Live filesystem like Knoppix

	    This'll be a separate project in my spare time, but I was
	    really impressed with Knoppix.  I'm going to look into more
	    generic tools to build your own live CD filesystems, and
	    then integrate it with the bootable CD - you could have a
	    bootable CD that runs as CLASS in memory, but it also has
	    the FAI scripts etc. to install a copy of that class onto
	    the HD itself.  Kind of self-replicating like the Borg.  You
	    could have a master CD which is of CLASS "FAISERVER" which
	    has a library of other classes available and can be used to build
	    more clients via nfs (off the CD/memory or install it to the HD
	    first), or if it has a CDRW it can burn new CLASS CDs
	    to insert in each client.  With a proper master CD you could
	    replicate an entire server network with little or no network
	    connectivity.  If this could then be combined into the new
	    debian-installer then we could have this functionality on
	    each Official Debian CD - FAI is an install method and
	    classes can be collected off the CD like the current task-*
	    packages, off a floppy or whatever, or run as a live
	    filesystem using the class of your choosing from a menu etc.

	    I like the idea of having reliable, read-only media for
	    security.  As long as you've also got the tools to keep it
	    up to date and reproduce updated versions of the CD, on the
	    CD itself - you've got a fairly secure source of software.
	    Match all the md5sums and confirm they're all legitimate
	    packages, then burn a CD and you only have to worry about
	    physical access to it.

	    You could define useful liveCD classes like a GPG keysigning
	    server - everyone has their own on a miniCDR with their
	    keys, just insert floppies containing other people's public
	    keys, store it all in RAM and then write to your own floppy
	    at the end.  Or a liveCD FAISERVER class for disaster
	    recovery, if you lose everything you can bring up an FAI
	    server and rebuild the network within minutes.  Or
	    customised XFree86 liveCD classes for diskless workstations
	    with the option to install it to similar workstations with disks.

Niall Young                                    Chime Communications Pty Ltd
niall at chime.net.au                            Level 6, 263 Adelaide Terrace
Ph: 08 9213 1330 / 0408 192 797               Perth, Western Australia 6000

     "there's a lot of movement in my trousers at the moment"
                                     -- Dennis Kristofich, Sep 2002