Skip to content

RonDB and Docker#

There are numerous reasons why one may want to use Docker to run RonDB. At Hopsworks AB, we have decided to use docker-compose for quickly deploying RonDB clusters to run tests against and also to show reproduceable public demos. If you’re new to RonDB, playing around with a local docker-compose cluster is a great way to become accustomed to the database or even to develop an application against it.

Inspecting the Dockerfile is also a tangible way of showcasing the runtime dependencies that RonDB requires, so that developers can run RonDB on their host machines more easily as well. Production clusters on hopsworks.ai currently do not run with Docker, even though that may change in the future.

The official repository to run RonDB with Docker is here:

https://github.com/logicalclocks/rondb-docker

Note that there are currently no public Docker images available, even though this is in the process of changing. The instructions of building the Docker image locally are posted in the repository. Also note that it has currently been tested with RonDB 21.04, but newer versions will follow soon. The repository relies on pre-built RonDB binaries, which can be found on repo.hops.works/master.

The repository revolves around comfortability and quick local deployment, whilst mimicking the directory structure of a cluster deployed on hopsworks.ai. The latter allows the user to become accustomed to Hopsworks’ RonDB clusters before deploying one in the cloud. Using the main script build_run_docker.sh, it allows the user to create any size/shape RonDB cluster and takes care of dynamically creating the necessary files, most importantly

  • RonDB’s configuration file config.ini

  • The configuration file of the MySQL servers my.cnf

We assume a few hard-coded paramaters in these files, which can be edited under the templates to be found in the resources directory. It is important to note that these configurations are actively targeted towards deploying as many RonDB nodes onto a single machine as possible and thereby assign minimal resources to each. This has implications on the DataMemory of the data nodes, but also on the performance of these along with the MySQL servers. Lastly, the checkpointing frequency is also increased to accomodate for a smaller REDO log. This means that less disk storage is required, but larger loads also become more difficult to handle for the data nodes.

When creating a cluster with build_run_docker.sh, one will find the templated configuration files in the autogenerated_files directory. You can compare the templates and the outputs to obtain an understanding of which parameters have affected which fields.

Adjusting Docker Resources#

Another file that underlines the strategy of limiting resources per RonDB service is the docker.env file. This file can be edited to restrict the resources provided to the Docker containers. Each container will be running a single RonDB service: a management node (mgmd), a data node (ndbmtd), a MySQL server (mysqld) or an API node. Currently it is configured to give little resources to mgmds, the biggest proportion of memory to ndbmtds and the most computing power to the mysqlds. When changing these parameters, one however has to take into account that the Docker engine will have certain configurable CPU & memory limits set, which will override our parameters. To check whether the Docker containers are given the resources assigned to them in docker.env, run the following command on a running cluster:

docker stats

The data nodes will be most susceptible to limited resources. If Docker does not grant them their required memory, they will either not start up or die in the initialisation phase due to an OOM error.

Interacting with a running cluster#

There are many ways of interacting with a running RonDB cluster. A few of these are described in the benchmarking section below. Here, we will show the two most basic ones: using the management client and making queryies via the MySQL client. For both of these actions, we will open an interactive Docker terminal to a running container:

# Opening a terminal in a management container
docker exec -it mgmd_1 /bin/bash

# Opening a terminal in a MySQL container
docker exec -it mysqld_1 /bin/bash

After this you can either verify that the cluster is up and running in the mgmd shell or run a MySQL query via the MySQL shell:

# Check cluster status in the mgmd container
ndb_mgm -e show

# Query databases in the MySQL container
echo "show databases;" | mysql -uroot

Reading Logs#

When running the docker-compose RonDB cluster, the command will not run detached and so the stdout logs of all the RonDB nodes should be visible in the terminal. If one wants to view the logs of a single container, e.g. of the first data node, one can run:

docker logs ndbd_1

Independent of the stdout logs, all other RonDB-specific log and data directories are automatically mounted into Docker volumes. To list the created volumes of a running/killed cluster, run:

docker volume ls

The following directories are mounted into volumes:

  • the data & log directory of each mgmd

  • the data & log directory of each ndbmtd

  • the data directory of each mysqld

You can refer to the Docker documentation to view the contents of the volumes.

Lastly, for even quicker access, the logs of benchmarks are mounted into the host filesystem into the autogenerated_files directory.

Benchmarking and other API Node Applications#

The repository is set up so that the API nodes/containers can easily be used for benchmarking. View the output of build_run_docker.sh --help to see how to run benchmarks at startup and which benchmarks are supported. Once again, the environment is based on the environment deployed by clusters of managed Hopsworks, so the RonDB documentation on benchmarking can also be applied here. Note however, that not all RonDB tarballs compiled for ARM64 contain all of the benchmarking binaries. This is actively being worked on though.

Benchmarking is however only one of many use-cases for the API nodes. The Docker image has an entrypoint script that can start any program in the RonDB installation. It is precisely this script, which also executes the benchmarking script. If no benchmarking flag is specified for build_run_docker.sh, the API nodes will remain idle, such that one can enter the container via an interactive terminal and manually execute any RonDB program available under /srv/hops/mysql/bin. Such programs include NDB API, ClusterJ and NodeJS applications. One can also execute any other application using the NDB API, ClusterJ and using the MySQL client APIs in various languages. There is also NDB APIs available in NodeJS and Common Lisp.