automated testing for FAI development

[Henning Sprang]

Hi,

In the last weeks I am realizing more and more how hard FAI development
without automated testing is.

When testing installs, the process, depending an what part you work on
the process is

- build fai
- remove old fai
- install new fai
- maybe adjust config
- make nfsroot
- make base images (mutli-distribution)
- activate installation for testclient with fai-chboot
- restart testclient
- wait until FAI finishes or you see an error, or see it hanging
- if FAI finishes, log into the console
- check /tmp/fai/fai.log (mainly searching for "ERR" which indicates
software installation troubles which cause a lot harm mostly)
- check /tmp/target/boot if there's a kernel image
- check /tmp/target/boot/grub/menu.lst if there's an entry for the
kernel image
- reboot
- check if machine boots up
- check if specified services run and things work (often I only check if
X runs for a gnome demo install, but that's enough in most cases, real
problems appear on other places, mostly)


I already talked about that long before, but now the "pain" is big
enough, I plan to put some efforts in making installs automatically
testable before getting further on stuff like class directory
restructuring and further support for other Distributions.

The key to automated testing is using virtual Hardware, so also the
start and stop of Hardware can be checked by a script easily.
With qemu, I already have parts of the things described above, mostly
the build, package-install and run install part, not the test/checks
part. But, with an ssh master-key, the mentioned checks should be easily
possible.

I am also starting to play with xen these days. I am a bit unsure now if
I should finish the testing stuff I started with qemu to get stuff done,
or forget about that and concentrate on xen and forget about qemu. I
already know that xen is generally faster, but qemu doesn't have so many
hardware problems/dependencies. Xen doesn't run nice and easy on all
hardware, especially with unclean ACPI tables, while I saw none such
troubles with qemu. 

Does anybody else has experiences with testing installs and their
results automatically? How do you do it? Are there important checks I
forgot?

Henning