Testing 16 disk configurations takes how long?

[Thomas Neumann]

root at storage-dev:~/_testlab/fai-setup-storage# time ./run-tests
Found the following testcases:
[... output omitted ...]
Processing testcase [31_md] 02_raid1-wipe
Adding sdb to the list of required devices
Adding sdc to the list of required devices
 - preparing disk configuration
 - executing testcase
 - testcase passed verification
Completed 16 testcases, 1 tests did not match expected output.

real	0m7.939s
user	0m6.148s
sys	0m1.040s


Yep. 8 seconds.


I felt another need to scratch and this is my prototype -> 
https://github.com/ThomasNeumann/Public/tree/master/fai-setup-storage

It's a reimplementation of my initial attempt to create a test-harness for 
setup storage. (I'm mainly interested in FAI 4.x's setup-storage.) It's main 
advantage is in being able to test disk configurations without actually 
requiring a matching set of physical disks. It does this by replacing the 
/dev/sd#, /dev/hd# device links with loop devices mounted on top of sparse 
files. (Don't worry to much about the required black magic.) This makes it 
easily possible to test configurations with up to 8 256GB disks[1] on a system 
with 512MB of RAM and 2GB of disk space.

Sounds good. What's the itch?

Mainly it is to make sure setup-storage delivers the same result over and over 
again. Therefor the tool can automatically tell if a result has changed and 
warn you about it. And it does this _FAST_. The long term goal is to fix these 
stupid LVM-related bugs in setup-storage without introducing any new bugs or 
subtle changes. I'd like to be able to change the code and rerun 200+ tests to 
make sure everything is still as it should - in about 5 minutes or so...

But I'm too lazy to write these 200+ tests all by myself, so I turn to the 
mailing list. (again) Please send me all the disk_configs you care about. I 
want to see how my tool and setup-storage copes with these configs. (I request 
permission to add these configs to the git repository and therefore make them 
publicly available. If you don't want that, then please tell me.[2])

The procedure for adding a new testcase is quite simple:
* pick/create a directory in t/
* store the disk configuration as t/<directory>/<number>_<name>
* provide a <number>_<name>.preformat disk configuration if you want
  to start with a 'dirty' (partitioned) disk setup
* execute run-tests with the '--add-new' / '-a' option

Debugging a failed testcase is very easy too:
* execute run-tests with the '--show-diff' / '-d' option

I'd like to think it's pretty decent for a prototype ;)

bye
thomas


[1] 8 is the number of predefined loop devices, 256GB is the upper limit for 
files backed by a tmpfs volume. The size limit can definitely be raised to 
multi-terabyte values when using a 'real' filesystem instead of tmpfs to store 
the loop-mounted device files.

[2] AB, BK, TT, TU: I still have your configs and will run my tool against 
them. Since I don't know which one of these where to be kept private I will 
treat all of them confidential.