MTR testing for RonDB#
RonDB and NDB is used in extremely demanding applications. It has a great variety of features that are only used in failure modes, upgrade mode, meta data change modes. Thus a normal user will seldomly test the features in all manners possible. To ensure that we can guarantee that users don’t get problems with unexpected failure modes we have an advanced battery of tests that can be executed on each push, on a daily basis, on a weekly basis and at creation of a new release of the software.
After building a debug version of RonDB we can use these tests in an escalated manner, starting with simple tests, and moving on to more and more advanced test cases. We will show below how we can start with simple tests and move on to more complex testing using MTR to root out the bugs in your contribution.
In the above command we start at the top directory in our cloned RonDB tree. We have already built RonDB and we have made some changes to RonDB that we now want to test. The above command is a good simple test case that is a good test to start with. It runs a few simple statements to create RonDB tables and issue some insert, update, delete and select statements towards RonDB.
MTR stands for MySQL Test Run. It almost always access RonDB using a MySQL server. The ndb_basic test file is opened in the vi editor.
So any test in the ndb test suite has a file called test_name.test in this directory. The test run should always produce some output, this output is stored in a file that can be opened with the following command:
The ndb_basic test succeeds if the output from the test is the same as the text stored in the result file. Thus the test has to be repeatable to be a good test.
When you feel that you want to raise the test level you can run an entire test suite. I usually use the ndb test suite since this is the most challenging test suite. It contains almost three hundred test cases. This test suite is executed using the below command:
The force means that the test run will continue until ten tests have failed or until all tests have been executed.
To execute tests which can execute most tests require at least around 8 GByte of memory in your machine and even 16 GByte is a good idea. One can execute the tests a bit faster by adding a parallel flag as seen below:
Actually parallelism 6 for RonDB means that 3 tests will be executed in parallel. RonDB consumes more resources compared to running most MySQL test cases. However it is important to have sufficient resources to run the test cases. With a machine with 32 GByte of memory one can run with parallelism of 10, thus running 5 test cases in parallel. This assumes that the machine isn’t used for a lot of other things as well. If it is your machine which you use for web surfing, email, social media and so forth, then one need to lower the parallelism since it is important to have enough memory to run the tests.
Finally the below command executes all RonDB test suites with a parallelism of 8, thus executing 4 tests in parallel.
cd rondb_2104/debug_build/mysql-test
./mtr --suite=ndb,ndbcluster,ndb_opt,ndb_ddl,ndb_rpl,rpl_ndb,gcol_ndb,json --parallel=8 --force
The default cluster configuration is stored in the file opened with the below command:
Some tests use a different cluster configuration, in this case the configuration is found in test_name.cnf. Tests can include files called include_file_name.inc.
More on MTR#
MTR is the main test framework for testing MySQL. It runs mainly SQL queries against one or a few MySQL Servers and compares the result of running those queries against a result file. Each test run must generate exactly the same result each time, otherwise the test has failed. In addition no nodes are allowed to fail during the test run unless the test made them fail as part of the test.
Tests are organised in test suites. For the NDB storage engine there is a general test suite called ndb as mentioned above, this test suite covers all aspects of RonDB and is by far the largest test suite. There is ndb_rpl test suite that focus on testing RonDB global replication, there is another test suite rpl_ndb that tests a part of the MySQL replication tests using the NDB storage engine (not all MySQL replication tests are applicable to the NDB storage engine).
We have test suites that tests the pushdown of complex join queries in RonDB, this test suite is called ndb_opt. Another test suite focus on DDL for RonDB in ndb_ddl. There is a test suite for testing specifically that NDB storage engine interacts well with the binlog which is called ndb_binlog. There is a test suite called ndbcluster which contains a quick test to see if changes in the MySQL Server code has affected the NDB storage engine. There is a new test suite called gcol_ndb that tests NDB support for generated columns. In RonDB 22.01 there is also a new test called json_ndb to test RonDB using JSON columns.
The RonDB test suites contains a fair amount of test code, the total number of test lines (not counting the result lines) is about 107k lines in RonDB 22.01 of tests and this counts only the tests for the NDB storage engine, the test for the MySQL Server reaches around the 1 million lines mark.
MTR is a functional test framework and it tests mainly the NDB storage engine and parts of RonDB used to perform DDL and DML operations in SQL. It has very few tests that interact those things with restarts of various sorts. MTR is focused on functional testing. It is a very powerful tool for running functional tests. Given that SQL is a very expressive language it is fairly easy to generate complex functional tests that exercise very large parts of RonDB with fairly little effort in writing test cases.
MTR is the main tool for testing replication between RonDB clusters. It makes it very easy to setup very complex cluster configurations and verify that replication between clusters happens in the manner that they were designed to be.