Skip to content

Reporting issues in RonDB#

RonDB is maintained on GitHub. The RonDB tree is placed at https://github.com/logicalclocks/rondb. Any issues can be reported in the Issues section in this tree.

Write a description of how the issue is reproduced and some other notes about how the execution environment looks like.

Next we will describe how to put together a tarball of the important log files that are required to be able to find the solution to the issue. We will describe how to do this for Hopsworks user with some information how one can handle this for generic installations.

Preparing a crash tarball for a RonDB data node#

RonDB data nodes have very detailed log reports that assist in understanding how the crash came about. This includes node logs and trace files that show messages executed right before the crash (not message data, mostly the message header). RonDB also reports the last jumps made in the code, thus giving a very important addition to the stack trace, which is reported in the node log. The error log reports which thread crashed and there is one trace file for each executing block thread.

All those files are in the same directory. In Hopsworks this directory is /srv/hops/mysql-cluster/log. So to create a tarball of this simply use the command:

cd /srv/hops/mysql-cluster/log
tar cfz log.tar.gz log

Now upload this tarball to the GitHub issue and chances are good that we will be able to assist you in finding the problem. Often the fix requires upgrading RonDB and this is described in the chapter on upgrading RonDB.

Generally there are three file types:

  1. the node log, often called ndb_1_out.log, where 1 is the node id of the node

  2. the error log, often called ndb_1_error.log

  3. the trace files, usually named ndb_1_trace.log.3_t2; the first number being the node id, the second number is the number of the crash and t2 is the thread number 2. Thread 0 trace file is named ndb_1_trace.log.3

Regarding the error log, there can be several crashes reported inside it. We maintain a maximum number of error reports to avoid growing the file system in cases of repeated failures (RonDB clusters can often continue working even the case of repeated failures of one data node). Thus when we reached the maximum number of crash reports (configured and defaults to 25) the next error report will be 1 again and will overwrite the previous error report with number 1.

RonDB management server logs#

The placement of the log files for RonDB management server is the same as for the RonDB data node. There are two file types here of value. The first is often named ndb_65_cluster.log, where 65 is the node id of the management server. This is the cluster log (named cluster.log in Hopsworks). This log contains information about events in all the nodes of the cluster. There is also a node log of the management server in the same directory.

MySQL Server logs#

The MySQL server logs often contains much less information, but can still provide useful information about what happened in the cluster. In Hopsworks the MySQL error log is placed at the same place as other logs and is usually called mysql_67.log where 67 is a node id, but sometimes this isn’t the correct node id. A MySQL server uses often multiple node ids to scale to a larger amount of CPUs.