RonDB Tools#
There is a new GitHub tree. This tree is created by the RonDB team for use for experimentation with RonDB, running RonDB benchmarks, testing RonDB upgrades and system testing of RonDB. Its first use case was to setting up a cluster to perform more than 100M key lookups per second using RonDB.
The current set of tools uses Terraform
to setup a set of VMs in AWS
as directed by the Python
configuration file config.py
. Next it
deploys the RonDB software and the required libraries on the RonDB
nodes. It also deploys the benchmark tool Locust and the
valkey-benchmark
on the VMs used as benchmark clients.
The interactions with the tool goes through the script cluster_ctl
.
The intention is to continue developing this tool, now it installs on VMs, next phase will install RonDB using RonDB Helm charts. Currently it is only targeting cluster setups in AWS. This is likely to be followed by other clouds as well.
How to deploy a RonDB cluster#
Installations required to work with RonDB tool#
At first it is necessary to install the following tools, Terraform
,
AWS CLI
, python3
, rsync
and tmux
if you want to use a lot of
terminal windows.
Installing terraform is explained on this link.
On Mac OS X one best uses brew
to install the other tools using the
following command:
On Debian/Ubuntu the following commands can be used:
In order to run this tool you need an AWS API key to create cloud resources. Run the following command to configure the AWS access key id and the access key.
With this preparatory step you are now prepared to create a RonDB cluster. Creating a RonDB happens in 4 phases.
-
Edit the configuration in
config.py
-
Run ./cluster_ctl terraform
-
Run ./cluster_ctl install
-
Run ./cluster_ctl start
The terraform
, install
and start
can be combined into one command
./cluster_ctl deploy
.
After this command you have created the VMs, installed all the necessary binaries and libraries required to run the various VMs in the RonDB cluster and finally also started the RonDB cluster.
At this point we have also installed Locust software on the benchmark
clients as well as valkey-benchmark
. To run the Locust benchmark two
more steps are required.
-
Run ./cluster_ctl populate
-
Run ./cluster_ctl start_locust
These two steps creates a table, populates the table with data and
starts all the Locust workers in the benchmark clients. These two
commands can be combined into one command ./cluster_ctl bench_locust
.
The start command also started monitoring of the RonDB cluster. The
monitoring happens in a VM that runs Grafana
and this VM is supported
by another VM that runs Prometheus
. The start
command will return an
https
address from where you can see the RonDB dashboards.
The start_locust
command will return a https
address to a web page
from where you can start and stop the Locust benchmark, it will also
show the througput and latency of the operations it performs.
Command reference for the RonDB tool#
Some sub-commands take a NODES
argument. It is a comma-separated list
of node names and node types. If given, the command will operate only on
matching nodes, otherwise on all nodes. Available node types are:
-
ndb_mgmd
-
ndbmtd
-
mysqld
-
rdrs
-
prometheus
-
grafana
-
bench
terraform#
Configure, initialize and apply terraform as needed. Performs the following steps:
-
create or update
terraform.tfvars
-
call
terraform init
unless it is already initialized -
call
terraform apply
install#
Install necessary software and configuration files. Depending on the node type this may include RonDB, prometheus, grafana and locust.
start#
Start or restart RonDB services. This doesn’t include benchmark services.
Deploy#
This command is a convenience command equivalent to terraform
,
install
and start
.
stop#
Stop RonDB services. This doesn’t include benchmark services.
populate#
Creates and populates a table for use by the Locust benchmark.
The default for --cols
is 10. The default for --rows
is 100000. The
column types supported in --types
is INT
, VARCHAR
and
INT_AND_VARCHAR
. INT_AND_VARCHAR
is the default.
start_locust#
Start or restart locust services. You need to run populate
first.
--rows
must be set to the same value as in populate
. --workers
is
by default equal to the number of CPUs in the benchmark client VMs minus
one (one CPU is used by Locust for the manager part of the Locust
services, the remaining are one CPU per Locust worker.
bench_locust#
./cluster_ctl bench_locust [--cols COLUMNS] [--types COLUMN_TYPES] [--rows ROWS] [--workers WORKERS]
This is a convenience command that first calls populate
and next calls
start_locust
.
stop_locust#
Stops Locust
list#
Print node information, including node name, node type, public and private IP, NDB Node IDs, and running processes.
open_tmux#
Create a tmux session with ssh connections opened to all applicable nodes.
When attached to the tmux session, you can press Ctrl-b n
to go to the
next window, and Ctrl-b d
to detach.
If a session already exists, ignore the NODES
argument and reattach to
it.
To connect to a different set of nodes, use kill_tmux
first.
See also the ssh
command.
kill_tmux#
Kill the tmux session created by open_tmux
.
ssh#
Connect to a node using SSH. Run the list
command to discover the node
names available.
cleanup#
This command cleans up the entire RonDB cluster by terminating all the VMs created. After this command is executed all VMs and their data is lost.
Configuring a deployment#
Here is an example of the config.py
which is used to configure the
cluster setup.
# This config file is a python script. Only the config variable in the end
# matters, the rest are for convenience.
# For RonDB, we need the RonDB version and the glibc version.
latest_rondb = {
"glibc_version": "2.28",
"rondb_version": "24.10.7",
}
cluster_size = {
# The number of availability zones to use.
# Numbers larger than 1 means multi-AZ environment and 1 means single-AZ.
"num_azs": 2,
# For ndbmtd, mysqld, rdrs and bench nodes, we can specify the number of VMs
# to use.
"ndbmtd_count": 2,
"mysqld_count": 2,
"rdrs_count": 2,
"bench_count": 2,
# We also need to specify the number of data node replicas.
# Note that replicas * node groups = data nodes.
# Therefore ndbmtd_count must be divisible by rondb_replicas.
"rondb_replicas": 2,
# Disk sizes in GiB
"ndbmtd_disk_size": 200,
"mysqld_disk_size": 60,
"rdrs_disk_size": 60,
"prometheus_disk_size": 120,
"bench_disk_size": 60,
}
# CPU platform and node types. All nodes will use the same CPU platform, either
# arm64_v8 or x86_64.
arm_config = {
"cpu_platform": "arm64_v8",
"ndb_mgmd_instance_type": "c8g.large",
"ndbmtd_instance_type": "c8g.2xlarge",
"mysqld_instance_type": "c8g.large",
"rdrs_instance_type": "c8g.2xlarge",
"prometheus_instance_type": "c8g.medium",
"grafana_instance_type": "c8g.medium",
"bench_instance_type": "c8g.2xlarge",
}
x86_config = {
"cpu_platform": "x86_64",
"ndb_mgmd_instance_type": "t3.large",
"ndbmtd_instance_type": "t3.2xlarge",
"mysqld_instance_type": "t3.large",
"rdrs_instance_type": "t3.2xlarge",
"prometheus_instance_type": "t3.medium",
"grafana_instance_type": "t3.medium",
"bench_instance_type": "t3.2xlarge",
}
config= {
# AWS region
"region": "eu-north-1",
# RonDB version
**latest_rondb,
# Cluster size.
**cluster_size,
# Node type configs.
**arm_config,
#**x86_config,
}
The config
at the bottom of the file defines the parameter sections.
In this file we use the arm_config
, the region we use is the
eu-north-1
which is the swedish region in AWS. The latest_rondb
section is used to calculate the name of the RonDB binary tarball to
install. The cluster_size
section is used to define the number of VMs
of the various types, the number of Availability Zones to use, the disk
size to use in the VMs.
Instance configuration#
In this case we need to define the instances used by the various VM
types. However first we need to set the cpu_platform
to either
arm64_v8
or . This is used to make sure we get the correct RonDB
binary tarball as well.
Obviously we need to ensure that the VM types we choose exist in AWS and
that they actually use the CPU platform we configured. The instance
types are defined by AWS. So e.g. c8g.2xlarge
means a VM with 8 CPUs
of type Graviton 4 with 16 GB of memory and up to 15Gb/sec of network
bandwidth.
Cluster Size#
We make use of the following VM types:
-
ndb_mgmd, RonDB Management server
-
ndbmtd, RonDB Data node
-
mysqld, RonDB MySQL Server
-
rdrs, RonDB REST API Server (including Rondis)
-
prometheus, Prometheus time series database used by monitoring
-
grafana, Grafana, graphical UI to RonDB dashboards
-
bench, The benchmark clients
We can only have one Prometheus and one Grafana instance, thus they are not configurable and similarly we only support a single RonDB Management server. The other node types can be configured. We need at least one MySQL Server to be able to create tables, the number of RonDB REST API servers can be 0. Not having any benchmark clients would mean a cluster that cannot run any benchmarks, so probably should not use this tool.
The rondb_replicas
is the number of replicas of the data. In a
production cluster we would differ between this number and the number of
active data replicas. In this tool we are currently not setting things
up for flexibility, it is mainly set up for testing and experimentation.
By setting num_azs
to e.g. 2 we ensure that we will spread the VMs on
more than Availability Zone.
Network Load Balancer#
The tool will always set up network load balancer for the MySQL Servers, the RonDB REST API server and the Rondis server. It will use the AWS Network Load balancer. In most benchmark executions this will react very quickly and scale up. However when running Gbytes per second through it one sometimes need to run a second benchmark run for the network load balancer to scale up.
Experimenting with RonDB configurations#
If one wants to try with different configurations the /textttcluster_ctl tool will set defaults for MySQL configuration, for RonDB REST API configuration, for Rondis configuration and for RonDB cluster configuration.
If some of those things need to be changed one should do it before
running the install
command. Remember that the can be applied to a
subset of the node types if this change happens on a cluster already
created. Thus the install
command can be handled even after the
cluster is created to change the configurations of MySQL Servers, RonDB
REST API servers and Rondis servers.
The RonDB cluster configuration can be changed by logging in to the
ndb_mgmd
and go into the config_files
directory and edit the
config.ini
file. After this one need to run the start
command which
will first stop the cluster and next start it again from scratch.