RonDB REST API Server#
RonDB 21.04.15 introduces a new REST API, version 0.1.0, to retrieve data from RonDB databases. The new REST API improves the flexibility of the RonDB platform by allowing users to retrieve data from a more diverse set of languages/framework clients. Users can now easily retrieve data from Python, Julia or golang applications and are no longer limited to the C++/Java ecosystem.
REST API#
Currently, RonDB REST API only supports single and batched primary key read operations. In the future, we plan to extend the REST API to support write operations, scan operations, and transactions.
Supported Data Types#
Default mappings of MySQL data types to JSON data types are as follows:
| MySQL Data Types | JSON Data Type |
|---|---|
| TINYINT, SMALLINT, MEDIUMINT, INT, BIGINT | number |
| FLOAT, DOUBLE, DECIMAL | number |
| CHAR, VARCHAR escaped | string |
| BINARY, VARBINARY base64 encoded | string |
| DATE, DATETIME, TIME, TIMESTAMP, YEAR | string |
| YEAR | number |
| BIT base64 encoded | string |
Primary Key Read Operations#
It is used to perform a primary key read operation. Assume we have the following table.
CREATE TABLE `my_table` (
`id0` int NOT NULL,
`id1` int unsigned NOT NULL,
`col0` int DEFAULT NULL,
`col1` int unsigned DEFAULT NULL,
PRIMARY KEY (`id0`,`id1`)
) ENGINE=ndbcluster DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
Primary Key Read Request#
A primary key read request would look something like this
POST /{api-version}/test_db/my_table/pk-read
Body:
{
"filters": [
{
"column": "id0",
"value": 0
},
{
"column": "id1",
"value": 0
}
],
"readColumns": [
{
"column": "col0",
"dataReturnType": "default"
},
{
"column": "col1",
"dataReturnType": "default"
}
],
"operationId": "ABC123"
}
Parameters:
-
filters: (required) This is an array of objects one for each column that forms the primary key.
-
readColumns: (optional) This is used to perform projections. If it is omitted, all the columns of the table will be read
-
dataReturnType: (optional) This can be used to control in which format the data is returned, for example, hex, base64, etc. However, in this version (0.1.0) we only support the default return type.
-
operationId: (optional) It is a string parameter and it can be up to 64 characters long.
Primary Key Read Response#
Body:
Primary Key Read Error Handling#
| Code | Error Message |
|---|---|
| 200 | OK |
| 400 | Bad Request. The error message will contain more information about which part of the request is invalid. For example, if incomplete primary key is specified |
| 401 | Access denied due to invalid API KEY |
| 500 | Server Error. For example, database failure |
Batched Primary Key#
You can use this to batch multiple primary key operations across different databases.
Batched Primary Key Request#
POST /{api-version}/batch
Body:
{
"operations": [
{
"method": "POST",
"relative-url": "my_database_1/my_table_1/pk-read",
"body": {
"filters": [
{
"column": "id0",
"value": 0
},
{
"column": "id1",
"value": 0
}
],
"readColumns": [
{
"column": "col0",
"dataReturnType": "default"
},
{
"column": "col1",
"dataReturnType": "default"
}
],
"operationId": "1"
},
},
{
"method": "POST",
"relative-url": "my_database_2/my_table_2/pk-read",
"body": {
"filters": [
{
"column": "id0",
"value": 1
},
{
"column": "id1",
"value": 1
}
],
},
},
]
}
Parameters:
- relative-url: (required) This represents the url the given pk-read would have in a single request (omitting the api-version).
Batched Primary Key Response#
Body:
[
{
"code": 200,
"body": {
"operationId": "1",
"data": {
"col0": 0,
"col1": 0
}
}
},
{
"code": 200,
"body": {
"data": {
"col0": 1,
"col1": 1
}
}
}
]
Batch Key Read Error Handling#
| Code | Error Message |
|---|---|
| 200 | OK. The batch operation was executed and an array of sub-responses is returned. Each sub response contains its own status code. For example, some suboperations may return code 200 while other may return code 404 |
| 400 | Bad Batch Request. Server fails for unmarshal the request |
| 401 | Access denied due to invalid API KEY |
| 500 | Server Error. For example, database failure |
RonSQL query execution#
NOTE: RonSQL query execution as described in this section is not yet fully implemented.
RonSQL is used to execute queries against a RonDB cluster using pushdown technology, thus achieving higher performance than what is possible via MySQL Server.
Assume we have the following table.
CREATE TABLE `my_table` (
`id0` varchar(10) NOT NULL,
`id1` int unsigned NOT NULL,
`col0` int DEFAULT NULL,
`col1` int unsigned DEFAULT NULL
) ENGINE=ndbcluster DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
Query request#
A RonSQL query can be submitted like this: POST /{api-version}/ronsql
Body:
{
"query": "SELECT id0, id1, SUM(col0), MAX(col1)
FROM my_table
WHERE id1 > 7
GROUP BY id0, id1;",
"database": "test_db",
"explainMode": "ALLOW",
"outputFormat": "JSON",
"operationId": "ABC123"
}
Parameters:
-
query: (required) The RonSQL query to be executed. See the RonSQL documentation for a description of supported queries.
-
database: (required) Name of the database containg the table read by the query.
-
explainMode: (optional) How to handle
EXPLAINqueries. The value can be one of:-
ALLOW: (default) Allow all query types, currently meaning
SELECTandEXPLAIN SELECT. -
FORBID: Allow only
SELECTqueries. Return an error code if given anEXPLAIN SELECTquery. -
REQUIRE: Allow only
EXPLAINqueries. Return an error code if given aSELECTquery. -
REMOVE: Reinterpret any
EXPLAIN SELECTquery as aSELECTquery. -
FORCE: Reinterpret any
SELECTquery as anEXPLAIN SELECTquery.
-
-
outputFormat: (optional) Output format. The value can be one of:
-
JSON: (default) JSON format with UTF-8 encoding, and with most characters represented by themselves. Set response content type to
application/json; charset=utf-8. -
JSON_ASCII: JSON format with ASCII encoding. Use
\uescape sequences. The output stream will only contain characters0x0a,0x20--0x7e. Set response content type toapplication/json; charset=US-ASCII. -
TEXT: For query output, mimic mysql tab-separated output with headers and set response content type to
text/tab-separated-values; charset=utf-8; header=present. For explain output, use a plain text format and set response content type totext/plain; charset=utf-8. -
TEXT_NOHEADER: For query output, mimic mysql tab-separated output without headers and set response content type to
text/tab-separated-values; charset=utf-8; header=absent. For explain output, use a plain text format and set response content type totext/plain; charset=utf-8.
-
-
operationId: (optional) A string parameter, up to 64 characters long. Can only be supplied when
outputFormatisJSONorJSON_ASCII.
Query response#
The query response depends on whether the query was executed as SELECT
or EXPLAIN SELECT, and on the requested output format.
Response body to SELECT query with "outputFormat": "JSON":
{
"operationId": "ABC123",
"data": [
{"id0": "EE", "id1": 10, "SUM(col0)": 13, "MAX(col1)": 180},
{"id0": "AÅ", "id1": 25, "SUM(col0)": 83, "MAX(col1)": 357}
]
}
Response body to SELECT query with "outputFormat": "JSON_ASCII":
{
"operationId": "ABC123",
"data": [
{"id0": "EE", "id1": 10, "SUM(col0)": 13, "MAX(col1)": 180},
{"id0": "A\u00c5", "id1": 25, "SUM(col0)": 83, "MAX(col1)": 357}
]
}
Response body to SELECT query with "outputFormat": "TEXT":
Response body to SELECT query with "outputFormat": "TEXT_NOHEADER":
Response body to EXPLAIN SELECT query with "outputFormat": "JSON":
{
"operationId": "ABC123",
"queryType": "pushdownAggregation",
"parseTree": {
"SELECT": [
{"name": "id0", "expression": {"type": "column", "column": "id0"}},
{"name": "id1", "expression": {"type": "column", "column": "id1"}},
{"name": "SUM(col0)",
"expression": {"type": "aggregation",
"function": "Sum",
"argument": {"type": "column", "column": "col0"}
}
},
{"name": "MAX(col1)",
"expression": {"type": "aggregation",
"function": "Max",
"argument": {"type": "column", "column": "col1"}
}
}
],
"FROM": "my_table",
"WHERE": {"type": "op", "op": ">", "left": "id1", "right": 7},
"GROUP BY": ["id0", "id1"]
},
"aggregationProgram": [
{"instr": "Load", "dest": "r00", "src": "C02",
"description": "r00 = C02:`col0`"},
{"instr": "Sum", "dest": "A00", "src": "r00",
"description": "A00:SUM <- r00:`col0`"},
{"instr": "Load", "dest": "r00", "src": "C03",
"description": "r00 = C03:`col1`"},
{"instr": "Max", "dest": "A01", "src": "r00",
"description": "A01:MAX <- r00:`col1`"}
],
"scanInfo": {
"mode": "tableScan",
"reason": "No candidate plans for index scan"
},
"postProcessingProgramLength": 23
}
Response body to EXPLAIN SELECT query with "outputFormat": "TEXT":
Query parse tree:
SELECT
Out_0:`id0`
= C0:`id0`
Out_1:`id1`
= C1:`id1`
Out_2:`SUM(col0)`
= A0:Sum(`col0`)
Out_3:`MAX(col1)`
= A1:Max(`col1`)
FROM my_table
WHERE
>
+- `id1`
\- 7
GROUP BY
C0:`id0`
C1:`id1`
Aggregation program (4 instructions):
Instr. DEST SRC DESCRIPTION
Load r00 C02 r00 = C02:`col0`
Sum A00 r00 A00:SUM <- r00:`col0`
Load r00 C03 r00 = C03:`col1`
Max A01 r00 A01:MAX <- r00:`col1`
Execute as table scan.
(No candidate plans for index scan.)
Output in UTF-8 encoded JSON format.
The program for post-processing and output has 23 instructions.
With an index scan, the scanInfo part would look like this:
{
"mode": "indexScan",
"column": "id1",
"indexName": "my_table_id1_index",
"ranges": [{"lboundType": "exclusive", "lboundValue": 7,
"rboundType": null, "rboundValue": null}],
"filter": null
}
For an index scan, the WHERE condition is split into a set of ranges
and a filter. The filter may therefore be simpler than the original
WHERE condition. In the example above, the filter is null because
the WHERE condition is simple enough to be expressed as a range.
Configuration Parameters#
{
"Internal": {
"APIVersion": "0.1.0",
"BufferSize": 327680,
"PreAllocatedBuffers": 1024,
"GOMAXPROCS": -1,
"BatchMaxSize": 256,
"OperationIDMaxSize": 256
},
"REST": {
"Enable": true,
"ServerIP": "localhost",
"ServerPort": 4406
},
"GRPC": {
"Enable": true,
"ServerIP": "localhost",
"ServerPort": 5406
},
"RonDB": {
"Mgmds": [
{
"IP": "localhost",
"Port": 1186
}
],
"ConnectionPoolSize": 1,
"NodeIDs": [],
"ConnectionRetries": 5,
"ConnectionRetryDelayInSec": 5,
"OpRetryOnTransientErrorsCount": 3,
"OpRetryInitialDelayInMS": 500,
"OpRetryJitterInMS": 100
},
"RonDBMetadataCluster": {
"Mgmds": [
{
"IP": "localhost",
"Port": 1186
}
],
"ConnectionPoolSize": 1,
"NodeIDs": [],
"ConnectionRetries": 5,
"ConnectionRetryDelayInSec": 5,
"OpRetryOnTransientErrorsCount": 3,
"OpRetryInitialDelayInMS": 500,
"OpRetryJitterInMS": 100
},
"Security": {
"TLS": {
"EnableTLS": true,
"RequireAndVerifyClientCert": false,
"CertificateFile": "",
"PrivateKeyFile": "",
"RootCACertFile": ""
}
"APIKey": {
"UseHopsworksAPIKeys": true,
"CacheRefreshIntervalMS": 10000,
"CacheUnusedEntriesEvictionMS": 60000,
"CacheRefreshIntervalJitterMS": 1000
}
},
"Log": {
"Level": "info",
"FilePath": "",
"MaxSizeMB": 100,
"MaxBackups": 10,
"MaxAge": 30
},
"Testing": {
"MySQL": {
"User": "rondb",
"Password": "rondb",
"Servers": [
{
"IP": "localhost",
"Port": 3306
}
],
},
"MySQLMetadataCluster": {
"User": "rondb",
"Password": "rondb",
"Servers": [
{
"IP": "localhost",
"Port": 3306
}
],
}
},
}
Configuration Parameters
Internal
-
APIVersion: Current version of the REST API. Current version is 0.1.0. Users are not expected to change versions manually.
-
BufferSize: Size of the buffers that are used to pass requests/responses between the Go and C++ layers. The buffers should be large enough to accommodate any request/response. The default size is 327680 (32 KB).
-
PreAllocatedBuffers: Numbers of buffers to preallocate. The default value is 1024.
-
GOMAXPROCS: The GOMAXPROCS variable limits the number of operating system threads that can execute user-level Go code simultaneously. The default value is -1, that is it does not change the current settings.
-
BatchMaxSize: Max number of suboperations in a batch operation. Default is 256.
-
OperationIDMaxSize: Max length of operation ID. Default is 256 characters.
REST
-
Enable: Enable/Disable REST Server Interface
-
ServerIP: Binds the REST server to this IP. The default value is localhost
-
ServerPort: REST server port. The default port is 4406
GRPC
-
Enable: Enable/Disable GRPC Server Interface
-
ServerIP: Binds the GRPC server to this IP. The default value is localhost
-
ServerPort: GRPC server port. The default port is 5406
RonDB
-
Mgmds:
-
IP: RonDB management node IP. The default value is localhost.
-
Port: RonDB management node port. The default value is 1186.
-
-
ConnectionPoolSize: Connection pool size. Default 1. Note current implementation only supports 1 cluster connection
-
NodeIDs: This is an optional list of node ids to force the connections to be assigned to specific node ids. If this property is specified and connection pool size is not the default, the number of node ids must match the connection pool size
-
ConnectionRetries: Connection retries
-
ConnectionRetryDelayInSec: Connection retry delay in sec
-
OpRetryOnTransientErrorsCount: Number of times retry failed operations due to transient errors.
-
OpRetryInitialDelayInMS: Initial delay used in exponential backoff for retrying failed operations.
-
OpRetryJitterInMS: Jitter is added (or subtracted) from the retry delay to prevent multiple failed operations from being retried at the same time
RonDBMetadataCluster
This is an optional parameter. In normal operations the user data and metadata data (API Keys) are stored in the same RonDB cluster. It is possible to store data and metadata in separate clusters. Use the RonDBMetadataCluster parameter to configure a dedicated RonDB cluster containing user metadata. For example, if RonDB and RonDBMetadataCluster parameters are set then pk-read and batch operations will be performed on a cluster defined in the RonDB parameter and metadata operations will be performed on a cluster defined in RonDBMetadataCluster parameter.
Security:
-
TLS:
-
EnableTLS: Enable/Disable TLS. The default value is true.
-
RequireAndVerifyClientCert: Enable/Disable TLS client certificate requirement. The default value is true.
-
RootCACertFile: Root CA file. Used in testing that use self-signed certificates. The default value is not set.
-
CertificateFile: Server certificate file. The default value is not set.
-
PrivateKeyFile: Server private key file. The default value is not set.
-
-
APIKey:
-
UseHopsworksAPIKeys: Enable/Disable Hopsworks API Key for authentication
-
CacheRefreshIntervalMS: The API Keys are cached and refreshed periodically. CacheRefreshIntervalMS can not be set to zero.
-
CacheUnusedEntriesEvictionMS: Unused API Keys are automatically evicted from the cache. Eviction time can not be less than cache refersh time (CacheRefreshIntervalMS).
-
CacheRefreshIntervalJitterMS: It prevents simultaneously updates to the cached API Keys if the keys were added to the cache in a very short interval
-
Log
-
Level: log level, Supported levels are panic, error, warn, info, debug, and trace. The default value is info.
-
FilePath: log file location. The default value is stdout.
-
MaxSizeMB: max log file size. The default value is 100.
-
MaxBackups: max number of log files to store. The default value is 10.
-
MaxAge: max-age of log files in days. The default value is 30.
Testing
MySQL server is only used for testing
-
MySQL: MySQL server is only used for testing
-
Servers:
-
IP: MySQL Server IP. The default value is localhost.
-
Port: MySQL Server port. The default value is 3306.
-
-
User: MySQL Server user. The default value is rondb.
-
Password: MySQL Server user password. The default value is rondb.
-
-
MySQLMetadataCluster: Similar to RonDBMetadataCluster this is an optional parameter used to connect to cluster containing user metadata.
Running RonDB REST Server#
The REST Server binary is named rdrs and it is packaged in the bin of the RonDB tar ball. You can run the REST server follows
The REST server is also compatible with NDB. The REST Server is built on RonDB native API which is identical to NDB native API, except for the version number. As REST API server dynamically loads the native API, it is possible to use the REST server with a NDB cluster. Copy the rdrs binary to ndb cluster bin folder, and librdrclient.so and librdrs_string.a libraries to lib folder of NDB cluster install dir. After copying these files you will be able to use the REST server with a NDB Cluster.
Security and Authentication#
Currently, the REST API server only supports Hopsworks API Keys for authentication and authorization. In the future, we plan to extend MySQL server users and privileges to the REST API. Add the API key to the HTTP request using the X-API-KEY header. Of course, you should enable TLS when using API Keys to ensure that eavesdroppers do not steal your API Key
RonDB REST API at Hopsworks AB#
Using RonDB REST API Hopsworks has built REST API for Feature Stores. Feature Store is a platform that connects enterprise data to analytical and operational ML systems. The REST API is used to retrieve data for real-time inference from the online feature store and RonDB databases.