TTL Purge REST API#

The TTL purging program runs inside every rdrs2 node that has the REST API enabled. It scans the TTL tables and deletes the rows that have expired, see TTL Service. The REST endpoints described here report what the purge worker of one rdrs2 node is doing, and change its settings.

Base URL and scope#

All endpoints are below

http://<rdrs-host>:4406/0.1.0/ttl-purge

where 4406 is the default REST.ServerPort of rdrs2. Use https when TLS is enabled for the REST server (Security.TLS.EnableTLS). The ttl-purge endpoints exist only under API version 0.1.0, see API Versions.

Every rdrs2 node has its own purge worker, its own settings and its own metrics. A request returns the values of the node that answers it. To see the whole cluster, query every rdrs2 node.
Metrics are kept in memory. They start from zero when rdrs2 starts.
Settings changed with PUT /config are not stored. When rdrs2 restarts, it takes enabled and active_window from TTLPurge.Enable and TTLPurge.ActiveWindow in its configuration file, and the other settings from their defaults.
The GET endpoints do not check API keys. PUT /config requires a valid API key in the X-API-KEY header when Security.APIKey.UseHopsworksAPIKeys is true, which is the default. Any valid API key is accepted.
The TTL purge metrics are not part of the Prometheus endpoint /metrics.

Endpoints#

Request	Returns
`GET /0.1.0/ttl-purge`	Configuration, status and metrics in one object
`GET /0.1.0/ttl-purge/config`	Current configuration
`PUT /0.1.0/ttl-purge/config`	Changes the configuration
`GET /0.1.0/ttl-purge/status`	What the purge worker is doing now
`GET /0.1.0/ttl-purge/metrics`	Counters for all TTL tables
`GET /0.1.0/ttl-purge/tables`	Counters per TTL table, one page at a time
`GET /0.1.0/ttl-purge/tables/{db}/{table}`	Counters for one TTL table

All responses are JSON objects.

Configuration#

GET /0.1.0/ttl-purge/config returns the configuration of the purge worker:

{"enabled":true,"min_batch_size":5,"max_batch_size":50,"sleep_interval_ms":1500,
 "reconcile_interval_sec":30,"active_window":""}

enabled: whether this node purges. When false, the worker stops before the next table and reports the state disabled. The value at startup is TTLPurge.Enable.
min_batch_size and max_batch_size: the limits for the number of rows the worker deletes from a table in one batch. The worker adapts the batch size of each table: when a batch deleted as many rows as the batch size in less than one second, the next batch is 5 rows larger, otherwise 5 rows smaller. The defaults are 5 and 50.
sleep_interval_ms: the pause between two rounds. The actual pause is chosen at random within one third of this value, so that several rdrs2 nodes do not run in step. When the batch size of any table has reached max_batch_size, the next round starts without a pause. The default is 1500.
reconcile_interval_sec: how often the worker lists all tables to find TTL tables that were created or dropped without a schema event, for example by ndb_restore into a running cluster or by an NDB API application. 0 turns this off. The default is 30.
active_window: the daily active window of this node, as "HH:MM-HH:MM" in UTC, or "" for none. A valid window in mysql.ttl_purge_ctrl takes precedence over it, see TTL Service. The value at startup is TTLPurge.ActiveWindow.

Changing the configuration#

PUT /0.1.0/ttl-purge/config takes a JSON object with the fields to change. Fields that are left out keep their current values, and unknown fields are ignored.

curl -X PUT -H "X-API-KEY: <api-key>" -H "Content-Type: application/json" \
     -d '{"enabled": false}' \
     http://<rdrs-host>:4406/0.1.0/ttl-purge/config

curl -X PUT -H "X-API-KEY: <api-key>" -H "Content-Type: application/json" \
     -d '{"active_window": "03:00-05:00", "max_batch_size": 100}' \
     http://<rdrs-host>:4406/0.1.0/ttl-purge/config

# Remove the window of this node
curl -X PUT -H "X-API-KEY: <api-key>" -H "Content-Type: application/json" \
     -d '{"active_window": ""}' \
     http://<rdrs-host>:4406/0.1.0/ttl-purge/config

The response contains the configuration that is now in effect:

{"message":"Configuration updated","config":{"enabled":true,"min_batch_size":5,
 "max_batch_size":100,"sleep_interval_ms":1500,"reconcile_interval_sec":30,
 "active_window":"03:00-05:00"}}

The values are checked as follows:

active_window must be a string of exactly the form "HH:MM-HH:MM", with hours from 00 to 23, minutes from 00 to 59, and a start that differs from the end, or the empty string. The window wraps past midnight when the start is later than the end, for example "22:00-02:00". Any other value is rejected with 400 and the error active_window must be a string "HH:MM-HH:MM" (UTC) or empty, and none of the fields in the request are changed.
A value of the wrong JSON type for enabled or one of the numeric fields is ignored.
min_batch_size below 1 is set to 1, and max_batch_size below min_batch_size is set to min_batch_size. sleep_interval_ms below 100 is set to 100. reconcile_interval_sec above 86400 is set to 86400.

An empty body is rejected with 400 and the error Request body is empty, and a body that is not a JSON object with 400 and Invalid JSON or Failed to parse JSON.

The worker reads the configuration at the start of every round. A change of enabled to false is also checked before every table, so the worker stops within one batch.

Status#

GET /0.1.0/ttl-purge/status shows what the purge worker is doing:

{"state":"running","schema_watcher_running":true,"purge_worker_running":true,
 "current_table":"mydb/events","current_partition":3,
 "active_window":"03:00-05:00","active_window_source":"ttl_purge_ctrl"}

state is one of:
- stopped: the worker has not started.
- running: the worker is purging.
- paused: the worker runs but has no TTL table to purge. This is also the state right after the worker starts.
- disabled: enabled is false.
- outside_window: an active window is in effect and the current time is outside it.
- error: the worker failed more than 10 times in a row and stopped. It is started again automatically.
paused and stopped are normal states, not faults.
schema_watcher_running and purge_worker_running: whether the thread that tracks TTL tables and the thread that purges are running.
current_table and current_partition: the table, as "db/table", and the partition that the worker purges now. "" and 0 while the worker is outside its active window or has no TTL table. While the worker is disabled, they can still show the last table it purged.
active_window: the window that is in effect, as "HH:MM-HH:MM" in UTC, or "" when purging runs around the clock.
active_window_source: where that window comes from: "ttl_purge_ctrl" for the cluster-wide window, "config" for the window of this node, or "".

The worker updates state, current_table and the window fields only while it takes part in purging. When mysql.ttl_purge_nodes lists other nodes but not this one, the worker does not purge and /status keeps showing the values it had before, for example paused right after startup. When the worker is disabled, the window fields show the window from the last time it was enabled.

Metrics#

The worker purges in rounds. In a round it deletes one batch of expired rows from one partition of each TTL table, and moves on to the next partition of that table in the next round. GET /0.1.0/ttl-purge/metrics returns counters over all TTL tables:

{"tables_count":12,"rows_purged_total":184220,"rows_purged_last_round":50,
 "last_round_duration_ms":87,"last_purge_time_epoch_ms":1767225600000,
 "rounds_completed":5321}

tables_count: the number of TTL tables this node knows.
rows_purged_total: the rows this node has deleted since rdrs2 started.
rows_purged_last_round: the rows deleted in the last completed round. It is a gauge that drops to 0 after a round without expired rows. Do not add it up over time; use rows_purged_total for totals.
last_round_duration_ms: the duration of the last completed round, without the pause after it. It covers all tables, so it does not show which table is slow. There is no timing per table.
last_purge_time_epoch_ms: when the last round completed, in milliseconds since the Unix epoch. It is updated even when the round deleted no rows, so it shows that the worker is working, not that it deletes rows. It does not change while the worker is disabled, outside its active window, or has no TTL table.
rounds_completed: the number of completed rounds.

Per-table metrics#

GET /0.1.0/ttl-purge/tables returns the TTL tables one page at a time, ordered by database and table name. The query parameters are offset (default 0) and limit (default 20, at most 100). A value that is not a number is ignored and the default is used:

curl "http://<rdrs-host>:4406/0.1.0/ttl-purge/tables?offset=0&limit=2"

{"total":12,"offset":0,"limit":2,"tables":[
 {"database":"mydb","table":"events","table_id":17,"ttl_sec":86400,
  "ttl_column_no":1,"current_partition":4,"partition_count":8,
  "current_batch_size":50,"rows_purged":120344,
  "last_purge_time_epoch_ms":1767225600000},
 {"database":"mydb","table":"sessions","table_id":21,"ttl_sec":3600,
  "ttl_column_no":2,"current_partition":0,"partition_count":8,
  "current_batch_size":5,"rows_purged":981,
  "last_purge_time_epoch_ms":1767225600000}]}

GET /0.1.0/ttl-purge/tables/{db}/{table} returns the object of one table. It returns 404 with the error Table db.table not found when this node does not know the table as a TTL table. A 404 right after a TTL table was created is expected: the node detects a new table within a few seconds.

Each request reads the current list, so the pages are not a consistent snapshot. When tables are created or dropped between two requests, entries move from one page to another. With up to 100 TTL tables, read them all with one request and limit=100.

database, table and table_id: the table. table_id is the NDB table id, which helps to match table ids in the cluster log to table names.
ttl_sec: the TTL of the table in seconds.
ttl_column_no: the position of the TTL column in the table, counting from 0.
current_partition and partition_count: the partition the worker purges next, and the number of partitions.
current_batch_size: the batch size for the next batch of this table.
rows_purged: the rows of this table that this node has deleted since rdrs2 started. It starts again from 0 when the table loses its TTL or is dropped and later comes back. rows_purged_total in /metrics is not affected.
last_purge_time_epoch_ms: when the worker last purged a batch of this table, in milliseconds since the Unix epoch, or 0 if it has not yet.

A table appears in the list as soon as the node detects it, with its counters at zero.

The response does not show whether the worker reads a table through ttl_index or scans the whole table. If purging costs more than expected, check that each TTL table has an index named ttl_index with the TTL column as its first column.

Combined view#

GET /0.1.0/ttl-purge returns the responses of /config, /status and /metrics in one object:

{"config":{...},"status":{...},"metrics":{...}}

HTTP status codes#

Code	Meaning
200	OK
400	`PUT /config`: the body is empty, not valid JSON, or has an invalid `active_window`
400, 401	`PUT /config`: the API key is missing or not valid
404	`/tables/{db}/{table}`: the node does not know the table as a TTL table
503	The purge worker could not be initialized on this node (`TTL purger not initialized`)

Common tasks#

Stop purging on one node for a while

Send {"enabled": false} to PUT /config and check that /status reports disabled. Send {"enabled": true} to start again. A restart of rdrs2 also returns the node to the value of TTLPurge.Enable.

Find out which active window is in effect

Read active_window and active_window_source in /status on each purging node.

Check whether purging keeps up

When a table has more expired rows than the worker deletes, its current_batch_size stays at max_batch_size round after round, and the worker runs rounds without pausing. Increase max_batch_size, add purge nodes, or widen the active window. When the active window closes while the worker is in this state, the rdrs2 log also shows a warning that a backlog of expired rows likely remains.

Measure the purge rate

Read rows_purged_total twice, a known time apart. The difference divided by the time is the purge rate of this node. Compare the sum over all purge nodes with the rate at which rows expire. For the total of the cluster, add up rows_purged_total of all rdrs2 nodes.

List every TTL table

Call /tables?limit=100, and use offset for more than 100 tables.

Monitor the purge worker

Alert when state is error, when purge_worker_running or schema_watcher_running is false, or when last_purge_time_epoch_ms stops advancing while state is running.

Follow one table

Call /tables/{db}/{table} a few times. rows_purged grows while expired rows are deleted, and current_partition moves through all partitions of the table that this node purges.