Skip to content

Cluster Configuration#

Most of the following subsections preluding the actual configuration file are optional. Applying them can provide better performance, security or reliability, but are not necessary for a basic distributed cluster setup.

Inactive Slots - Preparing to Scale#

When creating a configuration file for a distributed cluster, one should take into account that every change to the configuration file will require a rolling restart of the cluster. However, the management client (MGM) also supports operations for scaling the cluster, which do not require a rolling restart. These are:

  • ndb_mgm -e "<node id> ACTIVATE"

  • ndb_mgm -e "<node id> DEACTIVATE"

  • ndb_mgm -e "<node id> HOSTNAME <new hostname>"

The particularly important action is the one to change hostnames since it means that we can easily place a node from one machine onto another. However, for this to work, the node must be inactive. Our configuration file will therefore contain several nodes with NodeActive=0.

The parameter NoOfReplicas will refer to both active and inactive nodes per node group. It is our maximum replication factor without requiring a rolling restart. It is usually recommended to set this to 3.

Lastly, every defined slot requires additional memory allocation in the data nodes - therefore one should only define as many inactive slots as one believes one may scale out to.

Node Slots per API Server#

Every API / MySQL node slot in the config.ini corresponds to one cluster connection. Each of these connections will have one socket per data node.

Any API server can make use of multiple node slots. Depending on the server’s architecture, this may enable the server to increase its throughput.

MySQL servers usually scale performance up to 4 slots and 32 CPU cores. Even though we therefore often recommend using 4 slots, a single slot per MySQLd is often sufficient.

Optimising for NVMe drives#

In the Local Quickstart Tutorial, we simply defined the DataDir parameter in the cluster configuration file (config.ini) for our data nodes. This can further be used with block storage - however, with NVMe drives, performance can be fully exploited by using a more granular set of data directories. This assumes that we can format different NVMe drives depending on their usage. The following shows how RonDB’s data nodes write to disk:

  • REDO logs & Local checkpoints: Sequential writes, large chunks

  • Disk column data: Random access, small chunks

  • UNDO logs: Sequential writes

To place these types of data into different directories, we can use the following parameters in the config.ini:

[ndbd default]

# PERSISTING IN-MEMORY DATA
# Local checkpoint data and REDO logs (cannot be separated)
FileSystemPath=<path>

# DISK COLUMN DATA (TABLESPACES)
# This overrides FileSystemPathDD (DD=disk data).
# This is where InitialTablespace is placed.
FileSystemPathDataFiles=<path>

# UNDO LOGS / LOG FILE GROUPS
# This overrides FileSystemPathDD (DD=disk data).
# It is where the files for InitialLogFileGroup are placed.
FileSystemPathUndoFiles=<path>

Note that this assumes that all data node VMs have identical drives - if not, the parameters can be defined individually per data node using the corresponding [ndbd] section.

As mentioned earlier, disk column data will be written into data files of a tablespace. Usually, one defines a single large data file using the InitialTablespace parameter. However, if one has multiple NVMe drives, one can create one data file per drive. This will then allow for parallel disk column reads.

The File#

To demo scaling data node replicas and MySQL servers later on, we will start with a configuration with only 1 active replica per data node and 1 active MySQL server (with 2 node slots). Note that we usually recommend using NoOfReplicas=3 and 4 connections per MySQL server - we condensed this here for brevity.

Further assumptions:

  • We have machines with hostnames mgmd1_hostname, mgmd2_hostname, ndbd1_hostname, ndbd2_hostname and mysqld1_hostname

  • Every machine contains an empty directory /usr/local/rondb_data, which is placed on an arbitrary storage drive.

  • No other larger programs running on our data nodes VMs; otherwise set TotalMemoryConfig and NumCPUs to avoid OOM errors and performance issues.

Given these requirements and assumptions our configuration file (config.ini) looks as follows:

[ndb_mgmd default]
DataDir=/usr/local/rondb_data
# Most commonly used:
PortNumber=1186

[ndb_mgmd]
NodeId=65
Hostname=mgmd1_hostname

[ndb_mgmd]
NodeId=66
Hostname=mgmd2_hostname

[ndbd default]
NoOfReplicas=2
DataDir=/usr/local/rondb_data
# Most commonly used, otherwise random:
ServerPort=11860

[ndbd]
NodeId=1
NodeGroup=0
Hostname=ndbd1_hostname

[ndbd]
NodeId=2
NodeGroup=0
NodeActive=0

[ndbd]
NodeId=3
NodeGroup=1
Hostname=ndbd2_hostname

[ndbd]
NodeId=4
NodeGroup=1
NodeActive=0

# 2 MySQLd slots per host (up to 4 recommended)
[mysqld]
NodeId=67
Hostname=mysqld1_hostname

[mysqld]
NodeId=68
Hostname=mysqld1_hostname

# Used for scaling later
[mysqld]
NodeId=69
NodeActive=0

[mysqld]
NodeId=70
NodeActive=0

# In case we want to use the NDB API later on
[api]
NodeId=71
NodeActive=0

[api]
NodeId=72
NodeActive=0