New features in RonDB 24.10#
New REST API Server#
A major development is the new REST API Server that is now implemented in C++. It conforms to the same protocol as the REST API Server in 22.10 which was implemented in Go. Performance has been significantly improved, much of this comes from the use of the simdjson library that is about 30x faster on JSON parsing compared to the Go implementation. The performance improvement varies between 25% up to 900%. See the documentation for some benchmark examples.
The new REST API Server also implements an endpoint for RonSQL that is providing much faster aggregation queries. As mentioned it also implements the TTL purge service and Rondis. More about these in separate sections.
We removed support for GRPC in new REST API server.
The REST API Server also exports a port to read Prometheus data about latency for requests to REST API Server, RonSQL and Rondis.
Rondis server#
As part of the REST API server (RDRS2) we have also added a Redis server prototype. Rondis supports the following commands:
-
GET
-
SET
-
DEL
-
INCR
-
DECR
-
INCRBY
-
DECRBY
-
GETRANGE
-
SETRANGE
-
STRLEN
-
HGET
-
HSET
-
HDEL
-
HINCR
-
HDECR
-
HINCRBY
-
HDECRBY
-
PING
Rondis also supports a number of commands that sets and gets multiple rows. These are currently treated as a number of independent gets and sets.
-
MGET
-
MSET
-
HMGET
-
HMSET
There are a few configuration parameters, it will be possible to define the behaviour of these commands through these configuration parameters. These configuration parameters are set per database, so the same Rondis can serve multiple applications with different requirements.
As an example we support a dirty flag for increment and decrement operations. Since RonDB is a distributed DBMS with two-phase commits it is possible to update a single row a few thousand times per second. If it is necessary to increment a counter faster than that one can use a dirty flag, this will ensure that Rondis can support hundreds of thousands of increments and decrements per second.
RDRS2 can run both a Rondis server and a REST API server at the same time. They will serve different ports, but they will use the same connections to the RonDB data nodes.u It is possible to disable and enable those to avoid this sharing.
Using the new Rate limit and Quota feature one can limit both the size of a Rondis database as well as the amount of traffic it is allowed to handle.
Rondis reuses the new TTL feature in RonDB to implement TTL support for Rondis. The TTL purger also runs in the RDRS2 binary.
See the documentation for some examples of performance of Rondis.
TTL service#
In RonDB 24.10 one can now define a time-to-live for rows in a table. The TTL is defined on a DATETIME or TIMESTAMP column. The RonDB data nodes will hide rows that are expired. These rows can still be seen using special flags in the request. The expired rows are purged by a TTL purge service that runs in all RDRS2 servers. Thus to use the TTL service it is required to have a set of REST API servers.
This feature is natural in combination with the improved support for aggregation. RonDB 24.10 supports efficient aggregate queries and TTL makes it possible to let rows stay in the database for a specific time, thus we store a window of features that is dynamically moving and we can calculate any aggregates on this time window and thus get online aggregate results which will improve result of the inference queries.
Pushdown aggregation#
Aggregate queries were previously handled by the MySQL Server. This meant that the row data had to be sent to the MySQL Server for all rows. By supporting pushdown aggregates we perform the aggregation in the RonDB data nodes instead. Currently this is supported for aggregates on a single table.
With this feature the aggregation is handled by a special aggregation interpreter in the RonDB data nodes. The feature can be used from the NDB API, but we expect that most users will access it from RonSQL. There is no support for using this feature from MySQL Server.
The Q1 query in TPC-H see a speed up of more than 20x by using this feature through RonSQL.
RonSQL#
A new SQL server is implemented in the REST API server. This means that you can send SQL queries using an endpoint in the REST API server. Currently RonSQL only supports aggregation queries on a single table.
Rate limits and Quotas#
This feature adds the possibility to limit memory and disk usage for a database in RonDB and similarly one can also limit the usage of (rate limit) a database in RonDB. This includes limiting both the number of concurrent transactions and the size of transactions.
Improved recovery by parallelising a number of algorithms#
This feature is a speed up by a factor of 2 of restarts with many tables by doing more things in parallel in the restart phases.
Added many new interpreter functions#
RonDB has an interpreter that can be used to execute simple programs using a native language. The program is defined in the NDB API and sent with the request. The virtual machine has 8 registers that can store 64-bit signed integers. There are instructions to add, subtract, multiply, divide, shift left, shift right, NO, AND, OR and XOR bitwise operations. One can load constants into the register of various sizes.
A memory has also been introduced into the interpreter, this memory is 64 kByte in size.
One can load larger memory into this interpreter memory. This memory can be used to read columns (VARBINARY columns for interpreter memory), write columns, also read and write partial is possible and also appending to a VARBINARY column.
There are instructions to read from memory into a register and write to a register from memory.
Writing into VARBINARY columns requires some instructions to handle conversion of sizes into little-endian format.
There are instructions to send results back to the NDB API, there are also instructions to read input from NDB API in parammeters. This makes it possible to reuse an interpreter program with different parameters.
There is a large range of branch instructions and these instructions need a label defined to jump to if the branch condition is true.
There are also some library functions such as, string length, converting string to 64-bit signed integer, converting from 64-bit signed integer to string, binary search using a sorted array of numbers. There are 7 different sizes of the numbers supported and 5 different binary search algorithms supported. Based on the binary search algorithm it is easy to make searches in a sorted array of intervals. There are instructions to do this for 7 different number sizes and two different range types.
To support this there is a qsort instruction and an instruction to compress 8-byte number arrays to 5-byte and 6-byte number arrays and similarly from 4-byte number arrays compressed to 3 byte arrays.
Finally there is an instruction to search for a specific string. This instruction is not taking character sets into account, it is simply making a simple memory compare.