Skip to content

Changing Cluster Specs#

Adding & Removing Machines#

Adding and removing machines handles the following use cases:

  • Horizontal scaling

  • Vertical scaling (exchanging machines)

  • Handling VM failures

In this section, we will show how to scale out horizontally. However, all of these use cases follow the same procedure.

Assuming that we have started up our cluster, we now have 2 data nodes and 1 MySQL server running. Additionally, we have data nodes 2 and 4, and MySQL server 69 and 70 inactive. We will therefore be able to scale out to 4 data nodes and 2 MySQL servers using the management client and no rolling restart.

We also assume that we have new hosts ndbd3_hostname, ndbd4_hostname and mysqld2_hostname available.

We can therefore run the following commands to make our node slots available:

ndb_mgm -e "2 hostname ndbd3_hostname"
ndb_mgm -e "2 activate"

ndb_mgm -e "4 hostname ndbd4_hostname"
ndb_mgm -e "4 activate"

ndb_mgm -e "69 hostname mysqld2_hostname"
ndb_mgm -e "69 activate"

ndb_mgm -e "70 hostname mysqld2_hostname"
ndb_mgm -e "70 activate"

Once this is done, we can start up the new data nodes and MySQL server on their respective machines.

In the case of scaling vertically, one applies the same commands but simply moves the nodes to a more or less powerful machine. To have a homogenous cluster, one will also have to move the running nodes to the new machines. This requires them to be deactivated first:

ndb_mgm -e "1 deactivate"
ndb_mgm -e "1 hostname <new_host>"
ndb_mgm -e "1 activate"

ndb_mgm -e "3 deactivate"
ndb_mgm -e "3 hostname <new_host>"
ndb_mgm -e "3 activate"

Deactivating slots will automatically stop the nodes since they will be excluded from the cluster.

The only restriction for adding or exchanging machines is that data nodes need to be able to host the data they were storing previously. To see how much data a data node is storing, one can query the ndbinfo database.

Rolling Restarts and Software Up-/Downgrades#

Rolling restarts are cluster restarts whilst the cluster stays online. Rolling restarts are required when:

  • having reloaded the MGMd with a new cluster configuration

  • running a software up-/downgrade

The order of a rolling restart is the same order as starting up the cluster:

  1. ndb_mgmd: Management servers (MGMd)

  2. ndbmtd: Data nodes

  3. mysqld: MySQL servers (and API servers)

It is crucial during the rolling restart that there is always one data node per node group running. Otherwise, we will experience a cluster failure. This may also mean that we may have to temporarily start up data nodes on other machines to ensure that we have enough data nodes running. This can be done by using the management client to activate slots as described earlier.

Note that if a new activated slot is started on a different machine with the desired RonDB version and after the MGMd has been reloaded, it will not be required to restart anymore.

Whilst MySQL and API servers can theoretically be restarted in parallel, the application may require a certain amount of servers to be running at all times.

More details on supported software up-/downgrades can be found here.