Skip to content

RDMA Transporter#

RonDB can use RDMA instead of TCP for the connections between data nodes, and optionally for the connections between API nodes and data nodes. RDMA works over InfiniBand and over RoCE (RDMA over Converged Ethernet). The data is sent with ibverbs reliable connection queue pairs; a TCP connection on the normal transporter port of the node is only used to set up the RDMA connection. The RDMA transporter is available in RonDB 26.02.11 and later releases of 26.02, and in RonDB 26.10.

Requirements#

  • RonDB built with the CMake option -DWITH``_``NDB``_``RDMA=ON, which needs the libibverbs development package (rdma-core-devel or libibverbs-dev). The release builds of RonDB do not yet include RDMA. The management servers and every node that has an RDMA connection need such binaries; for connections between API nodes and data nodes this includes the MySQL Servers and NDB API applications.

  • An RDMA adapter (HCA) with an active port on each host. Both ends of a connection must use the same link layer, InfiniBand or RoCE.

Configuration#

RDMA is used only for the node pairs that have an [RDMA] section in the cluster configuration. The management server never creates these sections; all other node pairs use TCP as usual. A node pair has either an RDMA connection or a TCP connection, never both, and there is no fallback from RDMA to TCP.

  • Connections between data nodes can always use RDMA.

  • Connections between API nodes and data nodes can use RDMA when AllowApiToDbRdma is set to true in the [SYSTEM] section.

  • Connections to management servers cannot use RDMA.

The order of the sections in config.ini matters: the node sections must come before the [RDMA] sections, the [SYSTEM] section must come before any [RDMA] section for an API node, and an [RDMA DEFAULT] section applies only to the [RDMA] sections after it. An example with two data nodes and one MySQL Server:

[SYSTEM]
Name=mycluster
AllowApiToDbRdma=true

[ndb_mgmd]
NodeId=65
HostName=mgm1

[ndbd]
NodeId=1
HostName=db1

[ndbd]
NodeId=2
HostName=db2

[mysqld]
NodeId=67
HostName=api1

[RDMA DEFAULT]
RdmaDevice=mlx5_0

[RDMA]
NodeId1=1
NodeId2=2

[RDMA]
NodeId1=67
NodeId2=1

[RDMA]
NodeId1=67
NodeId2=2

To change a node pair between TCP and RDMA, stop both nodes and start them with the new configuration.

Configuration parameters#

An [RDMA] section applies to both ends of the connection, so the values must suit both hosts.

  • RdmaDevice: the ibverbs name of the RDMA adapter, for example mlx5``_``0. Both hosts must have an adapter with this name. If it is empty, the default, the host must have exactly one RDMA adapter; otherwise the connection is refused.

  • RdmaPort: the port of the adapter, 1 to 4, by default 1. The port must be active.

  • RdmaGidIndex: on RoCE, the index in the GID table of the adapter, by default 0. It must point at a routable GID.

  • RdmaSendBufferMemory and RdmaRecvBufferMemory: the send and receive buffers of the connection in bytes, by default 4M each. Each buffer is divided into RdmaQueueDepth slots, and a slot must hold at least 32792 bytes, so each buffer must be at least RdmaQueueDepth times 32792 bytes; otherwise the connection fails when it is set up. RdmaSendBufferMemory also limits the data waiting to be sent on the connection.

  • RdmaQueueDepth: the number of messages that can be in flight in each direction, 16 to 4096, by default 64. It must also be within the limits of the adapter.

  • RdmaInlineThreshold: messages up to this size in bytes are sent inline, 0 to 4096, by default 256. The adapter can lower the limit.

  • RdmaCompletionPollBudget: the number of completions handled in each poll, 1 to 4096, by default 32.

  • RdmaSpintime: the time in microseconds an API node spins on the connection before it sleeps, 0 to 2000, by default 0. Data nodes do not use it; they spin according to SpinMethod.

  • RdmaRetryCount: the number of retries, 0 to 7, by default 7, before the connection fails.

  • RdmaRnrRetryCount: accepted but currently without effect; the connection always retries without limit when the receiver is not ready.

  • RdmaTrafficClass (0 to 255, RoCE only) and RdmaServiceLevel (0 to 15): mark the RDMA traffic for quality of service. They need a matching QoS configuration in the adapters and switches. Both default to 0.

  • RdmaLogLevel: how much the RDMA transporter logs, 0 to 3, by default 2. It can be changed online with the SET command of the management client, see NDB management client.

  • Checksum is true by default for RDMA connections, unlike TCP connections. OverloadLimit 0, the default, means 80% of RdmaSendBufferMemory.

Monitoring#

The ndbinfo table rdma``_``transporters has one row for each RDMA connection of each data node. Connections between data nodes in the same node group have one row for each of their queue pairs.

  • node``_``id, trp``_``id, remote``_``node``_``id: the data node, the transporter and the node at the other end.

  • reconnects: the number of attempts to set up the connection, including the first one and failed ones.

  • send``_``posted, send``_``ok, send``_``err: send requests posted, and completed with and without success.

  • recv``_``posted, recv``_``ok, recv``_``err: the same for receive requests.

  • send``_``credit``_``stalls: the number of times sending stopped because the other end had no free receive slots.

  • peer``_``credits: the free receive slots at the other end.

  • rnr, retry``_``exceeded, qp``_``fatal: error counts for receiver-not-ready, exhausted retries and other queue pair errors.

  • bytes``_``sent, bytes``_``received: bytes on the connection since it was last set up.

When a connection is up, the node log shows the lines QP reached RTS; data path live and handshake OK, with the link layer and MTU. With RdmaLogLevel 2 the node log also gets statistics for each connection every 10 seconds, and with 3 every second.

Failures#

An error on an RDMA connection closes the connection. For a connection between two data nodes this is handled as a failure of the other node, in the same way as missed heartbeats; when several queue pairs connect the same two data nodes, an error on one of them closes all of them. An API node connects again. A connection that cannot be set up is retried.

Limits#

  • RDMA connections are not encrypted; TLS covers TCP connections only.

  • Management servers cannot use RDMA.

  • A node pair is changed between TCP and RDMA only by stopping both nodes.