In the last few weeks I've been working together with Tim Flink and Kamil Paral from the Fedora QA team on bringing some installation testing expertise to Fedora and establishing an open source test lab to perform automated testing in. The infrastructure is already in relatively usable condition so I've decided to share this information with the community.
Beaker is Running Our Test Lab
Beaker is the software suite that powers the test lab infrastructure. It is quite complex, with many components and sometimes not very straight-forward to set up. Tim has been working on that with me giving it a try and reporting issues as they have been discovered and fixed.
In the process of working on this I've managed to create couple of patches against Beaker and friends. They are still pending release in a future version because of more urgent bug fixes which need to released first.
SNAKE is The Kickstart Template Server
SNAKE is a client/server Python framework used to support Anaconda installations. It supports plain text ks.cfg files, IIRC those were static templates, no variable substitution.
The other possibility is Python templates based on Pykickstart:
from pykickstart.constants import KS_SCRIPT_POST
from pykickstart.parser import Script
from installdefaults import InstallKs
def ks(**context):
'''Anaconda autopart'''
ks=InstallKs()
ks.packages.add(['@base'])
ks.clearpart(initAll=True)
ks.autopart(autopart=True)
script = '''
cp /tmp/ks.cfg /mnt/sysimage/root/ks.cfg || \
cp /run/install/ks.cfg /mnt/sysimage/root/ks.cfg
'''
post = Script(script, type=KS_SCRIPT_POST, inChroot=False)
ks.scripts.append(post)
return ks
At the moment SNAKE is essentially abandoned but feature complete. I'm thinking about adopting the project just in case we need to make some fixes. Will let you know more about this when it happens.
Open Source Test Suite
I have been working on opening up several test cases for what we (QE) call a tier #1 installation test suite. They can be found in git. The tests are base on beakerlib and the legacy RHTS framework which is now part of Beaker.
This effort has been coordinated with Kamil as part of a pilot project he's responsible for. I've been executing the same test suite against earlier Fedora 20 snapshots but using an internal environment. Now everything is going out in the open.
Executing The Tests
Well you can't do that - YET! There are command line client tools for Fedora but Beaker and SNAKE are not well suited for use outside a restricted network like LAN or VPN. There are issues with authentication most notably for SNAKE.
At the moment I have to ssh through two different systems to get proper access. However this is been worked on. I've read about a rewrite which will allow Beaker to utilize a custom authentication framework like FAS for example. Hopefully that will be implemented soon enough.
I will also like to see the test systems have direct access to the Internet for various reasons but this is not without its risks either. This is still to be decided.
If you are interested anyway see the kick-tests.sh file in the test suite for
examples and command line options.
Test Results
The first successfully completed test jobs are jobs 50 to 58. There's a failure in one of the test cases, namely SELinux related RHBZ #1027148.
From what I can tell the lab is now working as expected and we can start doing some testing against Fedora development snapshots.
Ping me or join #fedora-qa on irc.freenode.net if you'd like to join Fedora QA!
Comments !