Skip to content

Rate Limits and Quotas#

In RonDB 24.10 we added a new feature that improves the use of RonDB in multi-tenant settings. As an example in Hopsworks the customers define projects, each project has defined a database to use in RonDB. These projects could be different departments from the same company, but could also be a cloud service where RonDB is used by thousands of concurrent users from different companies.

The Rate limits and Quotas are applied on databases in RonDB. Rate limits and quotas are defined using the RonDB management client.

The following management client command creates a limit on the rates used by the database test.

ndb_mgm> DATABASE QUOTA SET test --rate-per-sec=1000

The following items can be managed using the DATABASE QUOTA commands:

  1. --rate-per-sec: The microseconds of CPU usage per second allowed

  2. --in-memory-size: Memory usage in MBytes

  3. --on-disk-size: Disk column usage in GBytes

  4. --max-transaction-size: Max number of operations in a transaction

  5. --max-parallel-transactions: Max number of parallel transactions

Notable here is that each of those limits apply to one RonDB data node. If the RonDB cluster consists of one node group with 3 replicas this means that the actual rate available is 3x higher (for read and scan queries, update use CPUs in all nodes within one node group). However the memory and disk available is not higher since every node will store the same information since it is only one node group in the cluster. If instead we would have a cluster with 4 nodes in 2 node groups the database would have access to 2x the memory defined in the database quota.

If any of the limits isn’t specified or set to 0, it means that no rate limits or quotas are applied. However at least one of the parameters need to be set to a non-zero value.

Rate limits#

Rate limits means that we want to control how much CPU is used to access a certain database. This uses --rate-per-sec=value in the commands. The rates are provided in microseconds of CPU usage per second. Thus for example 1000 means that the database can use the CPU 1 milliseconds per CPU or by using the inverse we can say that the database has access to 0.001 CPUs.

So what happens if we use more than our rate limit? In this case we start slowing down the queries towards the database. If we have set the rate to 1000 and we used 1000 more than allowed we will delay every database request by one millisecond. If we continue to overuse our CPU rates to 2000 the delay will be increased to 2 milliseconds and so forth. If the user increases the rate very quickly we will also increase the delay to be able to react faster on the change.

This means that the rate limits gives the user the possibility to use small bursts of activity without being delayed. But if the user sustains a higher load, we will within less than a second start to slow the application down.

So can the number of CPUs be translated to queries per second. The simple answer is no, but we will give some indications when using Sysbench OLTP RW. This benchmark consists of 20 queries per transaction. 10 of those queries are simple primary key lookups, 4 of them are range scan queries that return 100 rows each, 4 of them are write operations, 1 insert, 1 delete and 2 updates. Finally there is a /textttBEGIN and a COMMIT query.

Using a rate of 100.000 the transactions per second is 330. Thus handling about 6600 SQL queries per second. Thus on average these SQL queries about 30 microseconds since we used 2 data nodes in the setup. Thus if the user decided that he wanted to handle 100 SQL queries per second, he would set the rate limit to about 1500.

If the rate usage is so high that the delay goes up to 30 milliseconds we start aborting any transaction reading data and at 60 milliseconds all transactions are aborted. In our testing we noted that even using 256 parallel threads with a rate limit of 100.000 we never reached this abort state. Thus if the database user is somewhat moderate in his usage and not attempting to use more than we have quota for, the application should only see delays and not see any aborts.

Our experiments showed that the rate limits are very exact and delivers a very stable throughput. If the application constantly exceeds the rate limit and gets slowed down the latency of his application will vary, but the average throughput per second will be very stable.

Memory usage limit#

For each database using quotas we will track the memory usage, if the database use too much memory RonDB will start to abort all transactions involving INSERT, WRITE and UPDATE. Only DELETE and SELECT queries are allowed when the memory quotas have been reached.

Disk space usage limit#

Tables allocate extents from tablespaces. When the database has reached the allowed limit of disk space usage, any query trying to allocate an extent will fail and get aborted.

Max transaction size#

We track the number of operations in a transaction (an operation is operating on a single row). If we limited the quota for this we will check that no transaction operates on more than the allowed rows in a single transaction.

Max parallel transactions#

We track the number of parallel transactions in a RonDB data node when quotas for this have been activated. If a new transaction is started beyond the limit, we will abort this transaction at the start of the transaction.

Rate limits and Quota commands#

The SET command#

The command below can be used set the rate per second to 10000 microseconds, the memory usage allowed to 128 MBytes and disk columns can use up to 2 GBytes of disk space on the database test. There are no specific limits on parallel transactions and transaction sizes.

Below is an example with its output.

ndb_mgm> DATABASE QUOTA SET test --rate-per-sec=10000 --in-memory-size=128 --on-disk-size=2
DATABASE QUOTA SET successfully executed

The ALTER command#

The ALTER command is used to change the rate limits and quotas either temporarily or permanently. The below will increase the above settings of the rate limits for the database test to have 50 milliseconds of CPU time per second or 0.02 CPUs. The memory usage and disk column usage quotas will be left untouched. At least one parameter need to be changed. Setting a parameter to 0 means that the parameter is no longer limited, thus we need not track it anymore.

Below is an example with its output.

ndb_mgm> DATABASE QUOTA SET test --rate-per-sec=50000
DATABASE QUOTA ALTER successfully executed

The DROP command#

The DROP command only needs the database name. No other parameter can be used. The rate limits and quotas will be lifted and the database can be used without limits again. The DROP command will have to wait until all query queues have been emptied before finishing.

Below is an example with its output.

ndb_mgm> DATABASE QUOTA DROP test
DATABASE QUOTA DROP successfully executed

The GET command#

The GET command displays the current setting of the given database. Below is an example of the command and its output.

ndb_mgm> DATABASE QUOTA GET test
Database Quotas for test
databaseId = 13
databaseVersion = 13
InMemorySize = 0 MByte
DiskSpaceSize = 0 GByte
RatePerSec = 1000
MaxTransactionSize = 0
MaxParallelTransactions = 0
MaxParallelComplexQueries = 0

MaxParallelComplexQueries is a parameter that can be set and can be retrieved. However for the moment it has no impact, we are currently not tracking complex queries. Complex queries are managed using the same delay mechanism as other queries for the moment.

The LIST command#

The LIST command will provide the current setting of all database quotas defined in the RonDB cluster.

Below is an example of the command and its output.

ndb_mgm> DATABASE QUOTA LIST
Database Quotas for test
  databaseId = 13
  databaseVersion = 13
  InMemorySize = 0 MByte
  DiskSpaceSize = 0 GByte
  RatePerSec = 2000
  MaxTransactionSize = 0
  MaxParallelTransactions = 0
  MaxParallelComplexQueries = 0

Database Quotas for test2
  databaseId = 14
  databaseVersion = 14
  InMemorySize = 0 MByte
  DiskSpaceSize = 0 GByte
  RatePerSec = 3000
  MaxTransactionSize = 0
  MaxParallelTransactions = 0
  MaxParallelComplexQueries = 0

The BACKUP command#

The BACKUP command essentially runs a list and produces the commands required to recreate the database quotas currently existing in the RonDB cluster.

Below is an example of the command and its output.

ndb_mgm> DATABASE QUOTA BACKUP filename.backup
Database Quota Backup command completed

Below is an example of the content of the backup file.

ndb_mgm> DATABASE QUOTA SET test --in-memory-size = 0M --on-disk-size = 0G --rate-per-sec = 2000 --max-transaction-size = 0 --max-parallel-transactions = 0 --max-parallel-complex-queries = 0
DATABASE QUOTA SET test2 --in-memory-size = 0M --on-disk-size = 0G --rate-per-sec = 3000 --max-transaction-size = 0 --max-parallel-transactions = 0 --max-parallel-complex-queries = 0

The RESTORE command#

The RESTORE command takes a backup file as input and restores the database quotas by running the DATABASE QUOTA SET command for each database saved in the backup file.

ndb_mgm> DATABASE QUOTA RESTORE filename.backup
Execute DATABASE QUOTA SET test --in-memory-size = 0M --on-disk-size = 0G --rate-per-sec = 2000 --max-transaction-size = 0 --max-parallel-transactions = 0 --max-parallel-complex-queries = 00
DATABASE QUOTA SET successfully executed
DATABASE QUOTA SET command succeeded for RESTORE
Execute DATABASE QUOTA SET test2 --in-memory-size = 0M --on-disk-size = 0G --rate-per-sec = 3000 --max-transaction-size = 0 --max-parallel-transactions = 0 --max-parallel-complex-queries = 00
DATABASE QUOTA SET successfully executed
DATABASE QUOTA SET command succeeded for RESTORE

Rate limits and Global Replication and Restarts#

Rate limits and quotas defined on one RonDB cluster is local to this cluster. The BACKUP command and RESTORE command can be used to transport the rate limits and quotas to a different RonDB cluster, but there is no automatic replication of these commands.

If a node restart or cluster restart is performed, the rate limits and quotas are not restored, rate limits and quotas are no not checked during the restart.