Backup file format: Billfish

The Billfish backup file format

Billfish backs up to one file that you create and keep. This page says exactly how that file is built, so that anyone can read it, check it, or write a program that opens it. The format described here is version 1.

There are two kinds of file. A plain backup is a zip file. A password-protected backup is a container that holds the same zip, encrypted. A reader tells them apart by the first eight bytes of the file, never by its name.

The zip

The file name is billfish-backup-YYYY-MM-DD-HHmm.zip, in the local time of the export. Readers never rely on the name, so a file written under the earlier name straightedge-backup-YYYY-MM-DD-HHmm.zip opens the same. A plain backup opens in any zip tool.

These are the only entries. There are no nested archives, paths use /, and no path has .. or a leading /.

Entry	Stored as
`manifest.json`	Deflated. At most 64 KiB plus 1 KiB per entry, and never over 16 MiB
`data/app.db`	Deflated. A copy of the database made with `VACUUM INTO`
`files/logo.png`	Stored. Only when the business profile has a logo
`files/signatures/<uuid>.png`	Stored. Each signature's image file that exists
`files/photos/YYYY/MM/<uuid>.jpg`	Stored, not recompressed. Photos inside the chosen window
`files/attachments/<sha256>.pdf`	Stored. Each attached PDF whose file exists and whose bytes hash to its name
`files/badges/<name>.png`	Stored. The trade badges, whatever the photo window. The name uses only letters, digits, `_` and `-`

Zip64 is used when an entry or the file needs it. A backup holds at most 100,000 entries.

The manifest

manifest.json is a UTF-8 JSON object.

Field	Meaning
`format_version`	Integer, 1. Raised only when a reader of the previous version would misread the file
`schema_version`	The database's `PRAGMA user_version`
`app_version`	The `version+build` of the app that wrote the file
`platform`	`android`
`created_at`	UTC, ISO 8601, ending in `Z`
`source_install_id`	An identifier of the exporting install
`photo_window`	`none`, `7d`, `30d`, `90d` or `all`. Photos are chosen by their creation time
`counts`	The number of live rows in each of several tables, and the number of photos included and in total. Informational
`files`	Every entry except the manifest, each as `{"path", "bytes", "sha256"}` with the hash in lowercase hex

Unknown fields are ignored, so format 1 can gain optional fields. A reader accepts a missing counts and an unknown photo_window.

What a reader checks

Each check, in this order, ends the import with a plain reason and changes nothing on the phone.

The zip opens and has manifest.json.
The manifest is within its size limit, parses, and has the fields above.
format_version is no higher than the app's, judged first and alone; then schema_version is no higher than the app's.
The entries in the zip equal the entries in files, plus the manifest: none extra, missing or duplicated, and every path matches the layout.
Each entry's size and SHA-256 match the manifest. Sizes come from the manifest and are counted while reading, never trusted from the zip's headers.
data/app.db opens as SQLite and its user_version equals schema_version. After the database is brought up to date, SQLite's integrity and foreign-key checks pass.

The checksums show that a file was not damaged on its way. They do not show who wrote it.

The password-protected container

With a password, the zip above is wrapped in a container. The file name is billfish-backup-YYYY-MM-DD-HHmm.sebackup. The container hides the entry names and sizes, and detects any change to the file as a whole.

The header is 105 bytes. Integers are big-endian.

Offset	Bytes	Field
0	8	Magic, ASCII `SEBACKUP`
8	1	Container version, 1
9	1	Key derivation id, 1 = PBKDF2-HMAC-SHA256
10	4	Iterations. Written: 600,000. Accepted: 100,000 to 5,000,000
14	16	Salt, random
30	7	Nonce prefix, random
37	4	Chunk size, exactly 1,048,576
41	32	Key check value
73	32	Header checksum: SHA-256 of bytes 0 to 72, unkeyed
105	...	Chunk records

Keys

The algorithms are PBKDF2 with HMAC-SHA256 for the key derivation, HMAC-SHA256 to label keys, and AES-256-GCM for the encryption.

master = PBKDF2-HMAC-SHA256(password, salt, iterations, 32 bytes), with the password as UTF-8 exactly as typed (it is not normalized).
key = HMAC-SHA256(master, "straightedge-backup-v1 key")
check = HMAC-SHA256(master, "straightedge-backup-v1 check"). The header holds check.

Chunks

The plaintext is the zip, cut into pieces of the chunk size; the last piece holds 1 to 1,048,576 bytes. Record i is AES-256-GCM(key, nonce_i, aad = the 105 header bytes, piece_i) with its 16-byte tag appended.

nonce_i is the nonce prefix (7 bytes), then i as a 4-byte unsigned integer, then one byte: 0x01 for the last record and 0x00 for every other.
Every record but the last is exactly the chunk size plus 16 bytes. A record is the last if and only if it ends at the end of the file, and only the last may carry 0x01.
Each file has a fresh salt, so each file has a fresh key and nonces never repeat under a key.

Reading a protected file

In order, each failure ending with its own plain message and nothing kept: the magic (if it is missing, try the file as a plain zip); the container version in byte 8 (a newer one reads as "update the app"); the header length and checksum (a damaged header is never reported as a wrong password); the key derivation id; the chunk size and iteration count ranges; the key derivation; the check value, compared in constant time (a mismatch is a wrong password); then each record decrypted in order. A failed tag, a record cut short, or a file whose last record lacks 0x01 is damaged. Only then do the zip checks above run.