Skip to content

Schema Transaction Protocol#

The Schema Transaction Protocol is a protocol that is used to handle schema changes (i.e. MySQL metadata) in a distributed manner.

Supported Operations#

The metadata changes RonDB supports are listed here. The current implementation of schema transactions is limited to supporting the subset of Data Definition Language (DDL) operations that MySQL supports:

  • CREATE

  • ALTER

  • DROP

  • RENAME

  • TRUNCATE

  • COMMENT

This also means that the protocol is limited to operations on a single table.

Cluster Limitations during Schema Transactions#

Whilst a schema transaction is running, the following limitations apply:

  • No other schema transactions can be run

  • Backups cannot be taken

  • Data nodes cannot be (re-)started

Thereby, one should take into account when to run schema transactions.

Transaction Steps#

Schema transactions are not fast, but they are very reliable and can handle arbitrarily complex schema changes. We use a lock to ensure that only a single schema transaction can be run at a time.

A schema transaction can be run via the NDB API or via a MySQL Server. Specifically creating tables is however recommended to run via the MySQL Server. The reason is that the MySQL Server stores the serialized table definition (SDI) inside RonDB together with the table. Other MySQL servers read it from RonDB and install the table in their own data dictionary, which is needed for the table to be accessible for them.

When using the MySQL server, it will initiate the protocol and communicate with the master data node. The master data node will then communicate with the other data nodes. Between the data nodes, it is the DBDICT block running the protocol.

A SQL statement to change the schema will be broken down into a set of smaller schema operations. For example, creating a table might consist of the operation to add a new table, a new hash map and a new set of indexes (hashed and ordered). It might also add foreign keys to the table.

Each operation is run via a set of 8 different phases. Each of these phases has a corresponding roll-back mechanism. Each transaction will define a commit point amongst its schema operations. If a node fails before the transaction has reached the commit point, every phase of all finished operations will roll-back. If a node (or even the cluster) fails after the commit point, the surviving / restarted nodes will continue with the transaction. This is called a roll forward.

Each data node records the progress of schema transactions in its local schema file, DBDICT/P0.SchemaLog. The file has one entry per schema object with its state (for example being added, committed or being dropped), its version and the id of the schema transaction that is changing it.

Recovery & Restoring Schema#

The definition of each table, index, foreign key and other metadata object is stored in a file of its own, for example DBDICT/T2/S1.TableList for the object with id 2 in version 1. Both kinds of files are written to the two directories D1 and D2. A data node starting up reads them in order to:

  1. restore its schema

  2. finish or abort a schema transaction

  3. ignore data in the REDO log and local checkpoints that has been dropped in the schema

If a transaction has passed its commit point, the schema file should suffice to finish the transaction locally on a restarting data node and only run the necessary remaining operations.

Example#

The most complex schema transaction we support is the operation to reorganize a table to make use of new node groups added since the table was created. This is done via the following statement:

ALTER TABLE ... REORGANIZE PARTITION;

This involves creating a set of new fragments on the new node groups. It also means that we have to copy over the data from the old table partitions to the new table partitions.