Hot backups and recovery while writes continue

A backup that needs the database to stop is a backup that gets postponed. InventDB takes its backups while the instance keeps serving reads and writes, sends them to object storage in the same region, and checks that a backup is complete before it will let anyone restore from it.

Backups are on from the start. Every new instance gets a daily backup at 02:00 in the time zone set for the instance, or UTC if none is set, and keeps the 14 most recent. This article covers how the copy is made, why it stays consistent while data keeps changing, how a backup carries proof that it finished, and what happens, step by step, when you restore one.

A copy taken while writes continue

A backup is a copy of the instance's data files, made file by file while the database keeps running. Each source file is opened for reading only and without a lock, so writers never wait for the backup and the backup never waits for writers. Nothing is flushed or paused first. The copy runs in five phases, always in this order:

1. Analysis. The engine walks the data directory and sorts every file into one of the four copy phases.
2. Id index. The index that maps each record id to its place in storage.
3. Records. The record files, which the engine only ever appends to.
4. Indexes. Property, columnar and vector indexes. This is the only optional phase, because everything in it can be rebuilt from the records.
5. Metadata. Everything else, including the write-ahead log and each type's list of segments, followed by a manifest that is written last.

The order is what makes a copy of a moving store usable. The id index is copied before the records, and record files only grow, so every record the copied id index knows about is present in the copied record files. The other indexes are copied after the records, so they are never behind them. A restored copy may lack some of the writes that landed while the backup was running, but it cannot hold a row that its own indexes fail to find.

Three small files are left out on purpose: the lock that names the running process, the marker that says the last shutdown was clean, and the marker that records how far the indexes are known to be complete. They describe the process that was running rather than the data. Without them, a restored copy opens the way a database opens after a power cut and runs the same recovery: it replays the write-ahead log it carries and re-indexes the segment that was being written.

Straight to object storage

Each file goes directly from the data volume to object storage in the same region, into a folder that belongs to your instance alone. Nothing is staged on local disk first, so a backup needs no free space on the instance, and a nearly full volume can still be backed up. Within a phase, eight files upload at a time by default; the phases themselves still run strictly one after another, because their order is the consistency rule.

The files are the engine's encrypted files, copied as they are, and object storage applies its own server-side encryption on top. A copy of a backup on its own cannot be read.

Knowing that a backup is complete

A backup of a live database races the database. While the copy runs, the engine seals segments, compacts and cleans up files, so a file listed during analysis may be gone by the time its turn comes. The backup treats the two kinds of trouble differently:

• A file that no longer exists was deleted by the database itself and is no longer part of the data. The backup counts it as skipped and carries on.
• A file that still exists but cannot be copied is tried up to three times, with a short pause between attempts. If it still fails, the backup counts it as failed and keeps going, so everything else is still copied, but the backup can never be used for a restore.

The folder in object storage carries its own proof of completeness, independent of the instance that wrote it. Before the first data file is uploaded, the backup writes an unfinished marker into the folder. The manifest, which records how many files were copied, skipped and failed, is written after every data file. The marker is removed only when the run has finished with no failed files. Anyone looking at the folder, including a restore into a replacement instance, can tell a finished backup from a partial one without asking the instance that made it.

Each backup's record shows its state, the files copied, skipped and failed, the bytes moved and how long it took. If the instance stops in the middle of a backup, the next start marks that record as failed and says why, rather than leaving it looking as if it were still running.

Schedules, time zones and retention

Every new instance starts with one schedule: daily at 02:00, keeping 14 backups. Administrators can change it or add more, with these settings:

• Frequency. Hourly, daily, weekly on a chosen day, or monthly on a day from 1 to 28.
• Time zone. Any IANA zone, such as Asia/Kolkata or Europe/London. The scheduler turns the local time into an exact UTC instant, so 02:00 in Kolkata means 02:00 in Kolkata whatever the server's clock is set to. On a day when clocks change, a local time that does not exist or occurs twice is skipped for that day.
• Retention. A count of backups to keep. Each time a schedule fires, the oldest finished backups beyond that count are deleted from object storage together with their records. A backup that is still running is never deleted.
• Indexes. Whether to include the optional index phase. A backup without indexes is smaller, and the indexes it leaves out can be rebuilt from the records.

The scheduler checks every 60 seconds and works from each schedule's last run rather than from the clock alone. If the instance was stopped or asleep when a backup was due, the backup runs as soon as the instance is up again, at most once for each missed period.

Instances sleep when they are idle, and a sleeping instance cannot run its own scheduler. Each instance therefore reports its next backup time to our control plane, which wakes a sleeping instance in a window from five minutes before that time to twenty minutes after it. While a backup is running, the instance counts as busy, so it is not put to sleep halfway through the copy.

Restoring a backup

A restore is an administrator action. It replaces the whole database with the backup's contents, so it runs in stages, and each stage can fail without touching your data:

1. Refuse what cannot be trusted. A backup that has not finished, or that failed to copy any file, is refused before anything is downloaded.
2. Stage. The backup is downloaded into a staging area on the instance's data volume. The live database keeps running.
3. Check the bytes. The staged copy is checked on its own terms: the unfinished marker must be absent, the manifest must be present and must report no failed files, and the files staged must not fall short of the count the manifest records. Any failure deletes the staged copy and returns the reason.
4. Mark it pending. The instance records that a restore is waiting and replies that it will be applied when the service next starts, which includes the next time the instance wakes from sleep.
5. Swap before opening. On that start, before the database opens any file, the server moves the current data aside and moves the staged backup into its place. If any move fails, it puts everything back, clears the pending restore and starts on the previous data.

After a successful restore, the previous data stays aside on the volume, so a restore that turns out to be the wrong choice can still be rolled back. The same path rebuilds a replacement instance: a fresh instance with an empty volume can download a backup and start on your data.

Limits

• Whole instance only. A restore replaces the entire database. There is no restore of a single table, no point-in-time restore between backups, and no restore into a database while it runs.
• A restore needs a restart. Nothing changes until the service next starts.
• A backup is as recent as its last run. Writes after a backup are not in it. With the default daily schedule the newest backup can be up to a day old; an hourly schedule narrows that to an hour.
• Retention counts backups rather than days. Failed backups count toward the total, so after a run of failures fewer restorable backups remain. Retention also applies across all of an instance's backups, using the count of whichever schedule fired, so give every schedule the same count.
• Set-aside data uses space. The previous data kept by a restore stays on the volume until it is removed.

How we test it

The backup tests run against the real engine with encryption on, and they open the result as a database rather than only counting files:

• A writer thread inserts records in batches of 100 throughout a hot backup. The restored copy must open; every row a full scan returns must be fetchable by id with byte-exact contents; COUNT(*) must equal the scan; and filters on a string, a numeric value, a numeric range, a boolean and an oversized text column must all agree with the records.
• A backup taken while files are deleted underneath it must finish, with skipped and failed counts accurate enough for the restore checks to rely on.
• A backup of a quiet database must contain every file and restore to identical answers.
• A restored copy must open on a machine where the original process's lock would otherwise block it, and must not carry the clean-shutdown or index markers.

Using it

Backups work the same way on InventDB Serverless and InventDB SOAR, through the same API. In InventDB SOAR, the Safekeeping & Health page shows the last backup, how many are kept and the daily backup time, with actions to back up now, restore a backup or remove one. Over the API, an administrator can take a backup before a risky change and follow its progress:

POST /api/backup/start
Authorization: Bearer <admin token>
Content-Type: application/json

{ "description": "Before the price import", "includeIndexes": true }

// 202 Accepted: { "ok": true, "backupId": "..." }
GET /api/backup/status/<backupId>

Moving the daily backup to 23:30 in Kolkata and keeping 30 backups is an update to the schedule; retentionDays holds the number of backups to keep:

PUT /api/backup/schedule/<scheduleId>
Authorization: Bearer <admin token>
Content-Type: application/json

{
  "name": "Daily backup",
  "frequency": "Daily",
  "timeOfDay": "23:30",
  "timezone": "Asia/Kolkata",
  "retentionDays": 30
}

A restore is one call, POST /api/backup/<backupId>/restore, and it answers only after the backup has been downloaded and checked. If it says the restore is prepared, the swap will happen on the next start; if it refuses, the reason says which check failed and your current data is untouched.