While working on Kiwi TCMS I've had the opportunity to learn in-depth about how the standard test case classes in Django work. This is a quick post about creating initial data and order of execution!
Initial test data for TransactionTestCase or LiveServerTestCase
class LiveServerTestCase(TransactionTestCase), as the name suggests, provides a running
Django instance during testing. We use that for Kiwi's XML-RPC API tests, issuing
http requests against the live server instance and examining the responses!
For testing to work we also need some initial data. There are few key items
that need to be taken into account to accomplish that:
./manage.py flushwhich deletes all records from the database, including the ones created during initial migrations;
self.serialized_rollback- when set to True will serialize initial records from the database into a string and then load this back. Required if subsequent tests need to have access to the records created during migrations!
cls.setUpTestDatais an attribute of
class TestCase(TransactionTestCase)and hence can't be used to create records before any transaction based test case is executed.
self._fixture_setup()is where the serialized rollback happens, thus it can be used to create initial data for your tests!
In Kiwi TCMS all XML-RPC test classes have
serialized_rollback = True and
_fixture_setup() method instead of
setUpTestData() to create the
necessary records before testing!
NOTE: you can also use fixtures in the above scenario but I don't like using them and we've deleted all fixtures from Kiwi TCMS a long time ago so I didn't feel like going back to that!
Order of test execution
From Django's docs:
In order to guarantee that all TestCase code starts with a clean database, the Django test runner reorders tests in the following way:
- All TestCase subclasses are run first.
- Then, all other Django-based tests (test cases based on SimpleTestCase, including TransactionTestCase) are run with no particular ordering guaranteed nor enforced among them.
- Then any other unittest.TestCase tests (including doctests) that may alter the database without restoring it to its original state are run.
This is not of much concern most of the time but becomes important when you decide
to mix and match transaction and non-transaction based tests into one test suite.
As seen in Job #471.1
tcms/xmlrpc/tests/test_serializer.py tests errored out! If you execute these tests
standalone they all pass! The root cause is that these serializer tests are based on
test.TestCase class and are executed after a
The tests in
tcms/xmlrpc/tests/test_product.py will flush the database, removing all
records, including the ones from initial migrations. Then when
executed it will call its factories which in turn rely on initial records being available
and produces an error because these records have been deleted!
The reason for this is that pytest doesn't respect the order of execution for Django tests! As seen in the build log above tests are executed in the order in which they were discovered! My solution was not to use pytest (I don't need it for anything else)!
At the moment I'm dealing with strange errors/segmentation faults when running Kiwi's tests under Django 2.0. It looks like the http response has been closed before the client side tries to read it. Why this happens I have not been able to figure out yet. Expect another blog post when I do.
Thanks for reading and happy testing!