NDB management client ndb_mgm#
The NDB management client has a few useful commands. The management client is used to start and stop nodes, show cluster status, shutdown the cluster.
The commands to handle backups using the NDB management client are covered in the chapter Backup.
The management client is used to add new node groups to the cluster with
CREATE NODEGROUP, see Online changes of
Configuration. It can also drop a node group
with DROP NODEGROUP <id>. This only works when no table has data in
the node group, for example right after CREATE NODEGROUP and before any
table is created or reorganised. Otherwise the command fails because the
node group is in use.
It is possible to set the cluster in single user mode, this means that only one API node is allowed to be active.
The management client can be used to change the filters for what gets printed in the cluster log. The DUMP command provides a possibility to print lots of debug messages to the cluster log, most of those are undocumented since they require fairly deep understanding of the cluster code to be useful.
The HELP command lists all the commands supported and HELP RESTART will provide specific help on the RESTART command. ? is the same as HELP.
For someone that wants to understand more about NDB internals there are commands that log signals in the data nodes: LOG, TESTON and TESTOFF. HELP DEBUG lists them together with LOGLEVEL and DUMP, but HELP DEBUG exists only in debug builds; the commands themselves exist in all builds. In a special compile mode it is possible to inject errors in the data nodes for efficient testing of cluster.
There is a command to get rid of node ids that haven’t been freed through the command PURGE STALE SESSIONS.
All the management client commands are also available through the NDB Management API, this API is a C API.
The command client remembers the history of commands and these can be retrieved by using up arrow and down arrow.
SHOW command#
The simplest command is SHOW that displays the configured nodes and their current status.
When a node isn’t connected it lists the hosts that are allowed to connect. A node that is deactivated (NodeActive=0) is shown as (not connected, node is deactivated) instead.
When a node is connected it lists the RonDB version it runs. In addition it also lists node id and hostname connected from for connected nodes.
For NDB data nodes it also lists the node group each data node belongs to. The data node marked with * is the master data node.
Below is an example output from a small cluster started from MTR where there is one NDB management server, 2 data nodes and 14 API nodes. In this case there are 2 MySQL servers started with 4 cluster connections each, thus 8 API nodes are connected from 2 MySQL servers. The remaining 6 API nodes are available for connections from localhost.
The output shown comes from RonDB 24.10.0.
ndb_mgm> show
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=1 @127.0.0.1 (RonDB-24.10.0, Nodegroup: 0, *)
id=2 @127.0.0.1 (RonDB-24.10.0, Nodegroup: 0)
[ndb_mgmd(MGM)] 1 node(s)
id=65 @127.0.0.1 (RonDB-24.10.0)
[mysqld(API)] 14 node(s)
id=67 @127.0.0.1 (RonDB-24.10.0)
id=68 @127.0.0.1 (RonDB-24.10.0)
id=69 @127.0.0.1 (RonDB-24.10.0)
id=70 @127.0.0.1 (RonDB-24.10.0)
id=71 @127.0.0.1 (RonDB-24.10.0)
id=72 @127.0.0.1 (RonDB-24.10.0)
id=73 @127.0.0.1 (RonDB-24.10.0)
id=74 @127.0.0.1 (RonDB-24.10.0)
id=192 (not connected, accepting connect from localhost)
id=193 (not connected, accepting connect from localhost)
id=194 (not connected, accepting connect from localhost)
id=195 (not connected, accepting connect from localhost)
id=196 (not connected, accepting connect from localhost)
id=197 (not connected, accepting connect from localhost)
START/STOP commands#
The NDB management client cannot start processes; that needs a process manager or an agent on each host. It can however stop data nodes and management servers, and it can restart them. A data node normally runs with an angel process that starts the data node process. When the data node stops, the angel process can start it again, see Managing Data Nodes.
Thus the management client can be used to start and stop data nodes and management servers after the programs are started. This is heavily used in the NDB test programs where we can control start and stop of programs completely from the test program once all the cluster programs have started.
The management client have no capability to start and stop application programs and especially MySQL server programs.
All START/STOP/RESTART commands start with a node id. If ALL is used instead of a node id, these commands apply to all data nodes only. To also stop the management servers, use the SHUTDOWN command. The -f option has no effect with ALL STOP. Some other commands also accept ALL, and their sections describe which nodes it covers. For example, ALL SET with an API node parameter applies to all API nodes.
Several node ids can be given before STOP, START and RESTART, separated
by spaces, for example 2 3 RESTART.
For example to RESTART node 2, use the following command:
START command#
The START command can be applied on a data node that have been started with the option -n. This option means that the data node program will start, but the data node will not start until instructed to do so.
It is possible to reach this state through the RESTART command as well. This makes it possible to stop a node and at some later time restart it again.
STOP command#
The STOP command is used when it is desired to stop the program completely in a node.
This command will perform a graceful shutdown. This means that an attempt will be made to ensure that no interference with user transactions happen. Transactions have a few seconds to complete before the transactions are aborted to continue the node shutdown.
The command STOP -a means a more abrupt stop where we will not even ensure that committed transactions are safe on durable media. The use of this is mostly for testing purposes and not for production purposes.
Normally a STOP command will be aborted if the stop of the node means that the entire cluster is shut down. In order to shut it down in even in this case use the command STOP -f.
A stop of a large data node can take a long time. With -x (extended timeout) the management client waits up to 60 minutes for the stop to complete instead of 5 minutes.
RESTART command#
The restart command is very useful in situations where it is necessary to change the configuration or for some other reason one needs to restart a data node or a management server.
The command first does a STOP, this stop can be affected by the parameters -a, -f and -x similar to the STOP command. The START is normally immediate after the STOP. But using the flag -n it is possible to return without restarting until a START command is executed.
There is an option to restart using the -i flag. This flag means that an initial node restart will be performed. This flag should be used with great care since it will remove the entire file system of the data node. The node will get all its data from other live nodes in the cluster.
SHUTDOWN command#
The SHUTDOWN command shuts down all data nodes and management servers using a graceful shutdown.
REPORT command#
The REPORT command can report backup status, memory usage and print the current event log.
Here is an example of the output from the three different reports that can be printed. The event log is cut since it can be quite long.
The memory report shows how much of DataMemory is used, split into pages used for rows (Data usage) and pages used for hash indexes (Index usage). The pages are 32 kBytes in size.
ndb_mgm> 2 REPORT backupstatus
Node 2: Backup not started
ndb_mgm> 2 report MemoryUsage
Node 2: Data usage is 1%(10 32K pages of total 950)
Node 2: Index usage is 1%(10 32K pages of total 950)
ndb_mgm> 2 REPORT EventLog
2017-09-20 14:11:48 Node 2: CM_REGREF from Node 1 to our Node 2. ..
2017-09-20 14:11:51 Node 2: CM_REGREF from Node 2 to our Node 2. ..
......
2017-09-20 15:06:22 Node 2: Node Sent Heartbeat to node = 1
ndb_mgm>
The report type can be abbreviated and is not case sensitive, for
example 2 REPORT mem.
STATUS command#
The easiest way to explain the STATUS command is by showing an example. This example also gives examples of some of the other commands.
The example starts with the same cluster as the SHOW command displayed. Node 2 is one of the data nodes and it is up and running when the example starts. In this case the status is started and the version of the data node software is also provided.
Next we perform the RESTART command, as can be seen this command doesn’t return until the command is completed. We used RESTART -n to see how it looks when the node is not started. The status in this case is not started.
The next step is to start node 2 again. The command returns when the start have been initiated and we can check the status of the restart. In this case the status is starting with some additional information about the start phase it has reached.
When the start completes we will get a printout of this even though the management client has already returned.
ndb_mgm> 2 status
Node 2: started (RonDB-24.10.0)
ndb_mgm> 2 restart -n
Node 2: Node shutdown initiated
Node 2: Node shutdown completed, restarting, no start.
Node 2 is being restarted
ndb_mgm> 2 status
Node 2: not started (RonDB-24.10.0)
ndb_mgm> 2 start
Database node 2 is being started.
ndb_mgm> Node 2: Start initiated (RonDB version 24.10.0)
ndb_mgm> 2 status
Node 2: starting (Last completed phase 100) (RonDB-24.10.0)
ndb_mgm> Node 2: Started (RonDB version 24.10.0)
ndb_mgm>
PROMPT command#
In some cases it might be useful to have several windows with management clients and it might be necessary to change the command prompt. The PROMPT command changes the prompt.
ENTER SINGLE USER MODE command#
ENTER SINGLE USER MODE node_id is a command that can be useful if it is required to perform an offline ALTER TABLE change.
When single user mode is entered, only the API node that was given in the command is allowed access to the cluster. Normally changes in the cluster are performed using online operations. This command provides a safe way of performing bigger changes using ALTER TABLE that cannot be performed while updating the cluster. It leaves one MySQL Server with access to the cluster, this can be used for read while the ALTER TABLE is performed.
After completing the offline operation one goes back to normal mode by using the command EXIT SINGLE USER MODE.
CLUSTERLOG command#
The CLUSTERLOG command controls what log messages that will end up in the cluster log.
The commands CLUSTERLOG ON, CLUSTERLOG OFF and CLUSTERLOG TOGGLE are followed by one or more severities, or ALL, and turn reporting of those severities on, off, or toggle it. Without a severity they apply to the whole cluster log. CLUSTERLOG FILTER is the same as CLUSTERLOG TOGGLE. CLUSTERLOG INFO prints the severities that are enabled.
The severities are DEBUG, INFO, WARNING, ERROR, CRITICAL, ALERT.
For example, to also report DEBUG messages:
This command changes what is reported by the management server. By default all severities except DEBUG is reported.
In addition we have a number of log categories. These are STARTUP, SHUTDOWN, STATISTICS, CHECKPOINT, NODERESTART, CONNECTION, ERROR, INFO, CONGESTION, DEBUG, BACKUP, and SCHEMA. SCHEMA reports when schema objects are created, altered or dropped.
Each of those categories can have their own log level that can be set with a CLUSTERLOG command.
We provide an example where we set the log level of node restart and backup to 8.
ndb_mgm> 2 clusterlog noderestart=8 backup=8
Executing CLUSTERLOG NODERESTART=8 on node 2 OK!
Executing CLUSTERLOG BACKUP=8 on node 2 OK!
ndb_mgm>
NODELOG command#
The NODELOG command turns debug messages in the node log of a data node on or off. With ALL it applies to all data nodes.
SET command#
The SET command changes a configuration parameter on a running cluster, without restarting any node:
With a node id, the change applies to that node only. With ALL, it applies to all data nodes, or to all API nodes for an API node parameter. The command first updates the configuration stored in the management server, so that nodes that start later use the new value, and then applies the value to the running nodes. If a running node cannot be updated, the command reports it; the stored configuration is still changed, and the node uses the new value at its next start.
The following parameters can be set:
-
MaxDiskWriteSpeed: the maximum disk write speed of the data nodes for local checkpoints and backups, in bytes per second. The value can have the suffix K, M or G and must be between 1M and 1024G.
-
EnableProactiveDeadlockDetection: switches proactive deadlock detection on the data nodes on (1) or off (0); on/off and true/false are also accepted. See Concurrency Control.
-
CompiledInterpreter: whether the data nodes compile the interpreted programs pushed down to them, scan filters and aggregation, to machine code. The value is OFF (the default) or AUTO. ON is also accepted. It is reserved for a mode that forces compilation, and today it works like AUTO. The numbers 0, 1 and 2 can be used for OFF, AUTO and ON. AUTO and ON are only accepted on x86_64 and aarch64 data nodes. Programs compiled after the change use the new value; programs already compiled keep running. See RonSQL.
-
AdaptiveSendThreshold: an API node parameter that chooses between latency and throughput for the NDB API’s sends to the data nodes. A value N greater than 0 makes a cluster connection use adaptive send while N or more threads have waited for replies within the last 10 milliseconds: a small send can then wait up to 200 microseconds, so that it goes out together with others. This gives more throughput with many concurrent clients, at the cost of a higher latency tail. The default 16 engages only under high concurrency; 0 sends every request at once, the lowest latency at any load. The node id must be an API node id; the running API nodes are updated through the data nodes.
-
RdmaLogLevel: how much the RDMA transporter writes to the log, for clusters that use RDMA connections. 0 logs errors only, which also stops the periodic RDMA statistics and the connect and disconnect messages; on large clusters these can otherwise flood the log. 1 adds warnings, 2 (the default) adds information and RDMA statistics every 10 seconds, and 3 is debug output with statistics every second. Errors are always logged. The value is stored in all RDMA connection sections, or in those of the given node, and applied at once to the RDMA connections of the running data nodes; other nodes use it at their next start.
The SET command with MaxDiskWriteSpeed is available since RonDB 25.10.9, and with RdmaLogLevel since RonDB 26.02.11. The other parameters can be set since RonDB 26.10.0.
ACTIVATE and DEACTIVATE commands#
A node can be defined in the configuration with NodeActive=0. It is then part of the configuration of all nodes, but it is inactive: it cannot join the cluster, and no other node waits for it. This makes it possible to add a replica, a management server or an API node later without restarting the cluster, see Configurable number of replicas.
<id> ACTIVATE activates node <id>. It updates the stored
configuration and tells the running nodes that the node is now active.
The first time an activated data node or management server starts, it
must be started with --initial.
<id> DEACTIVATE deactivates node <id>. A running data node is
stopped first; the command fails if stopping it would stop the cluster.
A running data node or management server cannot be deactivated if it is
the last active one of its kind, and the command cannot deactivate the
management server it is connected to. A management server is stopped
after it has been deactivated. An API node, such as a MySQL Server, is
not stopped by the command.
Neither command accepts ALL.
HOSTNAME command#
<id> HOSTNAME <hostname> changes the hostname or IP address of node
<id>, in the stored configuration and in the running nodes. The node
must be deactivated. Together with ACTIVATE and DEACTIVATE, this moves a
node to another computer, or gives an inactive node the address of the
computer it will run on:
LOCATION_DOMAIN_ID command#
<id> LOCATION_DOMAIN_ID <domain id> sets the location domain id of
node <id>, in the stored configuration and in the running cluster. The
location domain id, the configuration parameter LocationDomainId (0 to
16), tells RonDB which availability zone a node runs in. RonDB then
prefers a transaction coordinator and replicas in the same zone, which
avoids network traffic between zones. The command can be used on a
running node, for example by Kubernetes when it knows which zone a
container was placed in. It does not accept ALL.
DATABASE QUOTA and USER commands#
The DATABASE QUOTA commands set rate limits and quotas on a database; the USER commands set rate limits on a user:
DATABASE QUOTA SET|ALTER <database> {--<property>=<value>}+
DATABASE QUOTA DROP|GET <database>
DATABASE QUOTA LIST
DATABASE QUOTA BACKUP|RESTORE <file>
USER SET|ALTER <user> {--<property>=<value>}+
USER DROP|GET <user>
USER LIST
USER BACKUP|RESTORE <file>
They are described in Rate Limits and Quotas.
DUMP command#
The DUMP command is available in all builds; HELP DEBUG, which describes it, only exists in debug builds. It can activate printouts to the cluster log, to the NDB data node log and it can even perform some intrusive changes of variables in the code to perform workarounds of difficult problems.
This command is not normally used by developers or DBAs, it is mostly intended for MySQL support that can use this to gather information about a special issue. The DUMP commands forms a signal sent to various blocks in the data nodes. The code in those blocks to handle those signals is the documentation of what can be done using the DUMP command.
EXIT/QUIT command#
The EXIT command exits from the NDB management client, as do the QUIT and BYE commands.
CONNECT command#
The CONNECT command can be used to reconnect to the management server or to connect to a different management server. The parameter to the CONNECT command is a connect string, thus host[:port].
This command can be useful to see if all management servers agree on the cluster status.
TLS INFO command#
The TLS INFO command reports whether the management client’s two connections to the management server, the interactive connection and the event listener connection, use TLS or cleartext. It then lists the TLS connections reported by the management server, with the session id, peer address, certificate name, serial number and expiry date of each, and ends with the management server’s statistics since its start:
-
Total accepted connections
-
Total connections upgraded to TLS
-
Current connections
-
Current connections using TLS
-
Authorization failures
The START TLS command upgrades the management client’s connection to TLS. The management client normally does this already when it connects. The command fails if the client has no TLS certificate or the connection already uses TLS.
PURGE STALE SESSIONS#
In the history of MySQL Cluster there was at times issues that a node id was kept allocated for longer than it was supposed to. As a workaround for those situations the PURGE STALE SESSIONS command was developed. This purges any sessions from node ids that are no longer used.
It should hopefully not be needed, but it still exists.