# Five-Year Diary sealed format — five-year-diary/3 Version 3. Published with Five-Year Diary 1.0 for Android, 2026. A sealed file is a Five-Year Diary library in a single file: every diary, every entry and every photograph, encrypted with a passphrase the user chooses. The format is published so that the file can be opened without the app — on any platform, years from now, by anyone who holds the passphrase. This document is meant to be enough on its own. A developer on Windows, macOS, iOS, Linux or Android should be able to write a reader, and a writer, from it alone, using only what ships with the platform: a JSON parser, base64, PBKDF2-HMAC-SHA256, AES-256-GCM and zlib. Two files are published beside it: - **`five_year_diary_open.py`** — a reference reader in Python, standard library only. AES-256 and GCM are implemented in the file, so nothing needs installing. - **`golden-vectors.json`** — seventy-six golden test vectors: twenty files that must open to exactly the records given, and fifty-six that must be refused with exactly the code given. Both are normative alongside this document. The format has been implemented three times, independently — the vectors were sealed with OpenSSL, the reference reader opens them with its own AES and GCM, and the app seals and opens them with its platform's own cryptography — and all three agree on every vector. **Conventions.** Refusal codes are written `like-this` and defined under *Refusals*. Section names are in italics. MiB is 1,048,576 bytes. LF is U+000A. **Whitespace**, wherever this document uses the word, is exactly U+0020, U+0009, U+000A and U+000D — the set JSON allows between tokens — and nothing else. ## At a glance ``` {"header":{"format":"five-year-diary/3","version":3,"kdf":"PBKDF2-HMAC-SHA256","iterations":600000,"salt":"","cipher":"AES-256-GCM","nonce":"","segment":65536,"compression":"zlib","created":"2026-09-29T00:00:00Z"}, "segments":[ "", "" ]} ``` | Layer | Choice | Why this one | |---|---|---| | Container | One JSON object, UTF-8: **`header`, then `segments`** | Every platform parses JSON with no dependency. A readable header is a feature — a person can see what sealed their diary. **The header comes first** so that a streaming reader checks it before it reads a segment | | Key derivation | **PBKDF2-HMAC-SHA256**, 16-byte random salt, iteration count in the header | Argon2id is the better password KDF, but it is in neither Windows CNG nor Apple CryptoKit, so requiring it would make every reimplementation start by sourcing a library | | Cipher | **AES-256-GCM**, 16-byte tag | Built into every platform this document addresses, and authenticated | | Segments | **The compressed payload cut into `segment` bytes, each sealed as its own GCM message**, under a nonce made from the header's random nonce, the segment's index and a mark on the last — the STREAM construction that `age` and Tink use | Memory is one segment, whatever the library holds. A segment moved, repeated, dropped or added fails its tag, and so does a file cut at a segment boundary | | AAD | The canonical serialisation of the header, for every segment | No header value can be altered without every segment failing | | Payload | **JSON Lines, one record per line, three record kinds** | Streamable, diffable, and extensible without a new version | | Compression | **`zlib`** (RFC 1950, the wrapped form) or `none`, over the whole payload before it is cut | zlib and raw DEFLATE (RFC 1951) are different byte streams, and several standard libraries default to raw. The format names the wrapped one, and a raw body is refused as `compression`, never as a wrong passphrase | | Encoding | Standard base64 with padding, **strict and canonical** | Not base64url: standard base64 is the lowest-friction choice in every standard library | ## The container The file is UTF-8 text holding **one JSON object**, with nothing but whitespace before or after it, and no byte-order mark. The object has exactly two members, in this order: 1. **`header`** — an object; see *The header*. 2. **`segments`** — a non-empty array of strings, each the base64 of one sealed segment; see *Segments*. Nothing else is allowed beside them. The container's member order is checked: `header` must be the first member. **The writer's layout.** A writer puts the header alone on the first line and each segment on a line of its own, so that the first line of a sealed file says what sealed it: ``` {"header":, "segments":[ "", "", … "" ]} ``` One line per segment, each but the last followed by a comma, and every line ending with LF, the last (`]}`) included. A reader accepts any whitespace JSON allows between tokens and does not depend on this layout. **One JSON object per file, and one file per library.** Every diary is inside the one file, which is what the `diary` records are for. A sealed file is a single file, never a folder. ## The header A version-3 writer writes exactly these ten keys, in exactly this order: | # | Key | JSON type | Value | |---|---|---|---| | 1 | `format` | string | `five-year-diary/3` | | 2 | `version` | integer | `3` | | 3 | `kdf` | string | `PBKDF2-HMAC-SHA256` | | 4 | `iterations` | integer | 1 to 4,000,000. The app writes 600,000 | | 5 | `salt` | string | Strict base64 of 16 random bytes, fresh for every file | | 6 | `cipher` | string | `AES-256-GCM` | | 7 | `nonce` | string | Strict base64 of 12 random bytes, fresh for every file | | 8 | `segment` | integer | Plaintext bytes per segment, 1,024 to 1,048,576. The app writes 65,536 | | 9 | `compression` | string | `zlib` or `none` | | 10 | `created` | string | When the file was sealed — a timestamp (*Dates and timestamps*) | The strings are matched exactly, case included. **`segment` is 65,536 in the app, as in `age`.** It is a header value only so that the test vectors can use 1,024 and cross segment boundaries in a few kilobytes; other writers should write 65,536 too. **Integers as written.** `version`, `iterations` and `segment` must be JSON integers *as the file writes them* — matching `-?(0|[1-9][0-9]*)`, of any length. `1000.0`, `1e3` and `"1000"` are not integers, and are refused as `integer` by name, before any key is derived, rather than failing the tag later as a wrong passphrase. A reader whose JSON parser turns every number into a double cannot tell `1000.0` from `1000`, so it needs the number's text. The range checks hold whatever the magnitude written: a 30-digit iteration count is refused as `iterations`, and must not overflow a reader. **Member order in the file.** A writer emits the ten keys in the order above. A reader does not depend on the order of the header's members in the file — it re-serialises the header for the AAD — but it reads `format` before any other key (*Opening, in this order*). **Keys outside the ten.** A later version may add keys to the header. Each must be a string, a boolean, or an integer as written within signed 64 bits (−9,223,372,036,854,775,808 to 9,223,372,036,854,775,807), with no lone surrogate in its name or its value; anything else — `null`, a float, an array, an object — is `header`. A reader **authenticates** a key it does not know, because it is part of the AAD, and otherwise ignores it. A version-3 writer writes none. ### The canonical serialisation — the AAD The AAD of every segment is exactly these bytes: - the UTF-8 of the header object alone — `{`, its members as `"name":value` joined by `,`, then `}` — with no `"header":` before it and no whitespace in it; - the ten keys first, in the order of the table; then every other key, in ascending **Unicode code-point** order of its name, whatever order the file gives them; - strings escaped exactly as *JSON text* writes them; integers in shortest decimal (`-0` as `0`); booleans as `true` and `false`. It is a **re-serialisation**, not bytes lifted from the file: recovering an exact substring is awkward in several JSON libraries, while re-serialising is trivial everywhere. A writer emits the header in exactly this form, so in a file a writer made, the header's text and the AAD are the same bytes. **Code-point order is not UTF-16 order.** Java's, Kotlin's, JavaScript's and .NET's ordinal string comparisons compare UTF-16 code units, and so put U+1F511 (🔑) before U+FF5E (～); code-point order puts it after. The vector `unknown-header-key` stands its keys out of code-point order in the file and holds both. **Why unknown keys are authenticated.** A key a later version adds — an Argon2id memory cost, say — cannot then be lowered without breaking every segment's tag. The other side of the same rule: a key added to a header after sealing changes the AAD, segment 0 fails, and the file is refused as `passphrase` (vector `unauthenticated-header-key`). Two AADs from the vectors. `single-ascii`: ``` {"format":"five-year-diary/3","version":3,"kdf":"PBKDF2-HMAC-SHA256","iterations":1000,"salt":"EZaKoTx4JIZj0PQCZSeE/A==","cipher":"AES-256-GCM","nonce":"PGnHIwQBPgeLNtRL","segment":1024,"compression":"zlib","created":"2026-09-29T00:00:00Z"} ``` `unknown-header-key`, whose file gives its extra keys as `🔑`, `later`, `～`: ``` {"format":"five-year-diary/3","version":3,"kdf":"PBKDF2-HMAC-SHA256","iterations":1000,"salt":"GxrnaaLYZtjueI/NW7Kuug==","cipher":"AES-256-GCM","nonce":"q6r7O6rWjZpYYFgc","segment":1024,"compression":"zlib","created":"2026-09-29T00:00:00Z","later":"a key this version does not know","～":1,"🔑":true} ``` ## JSON text Every JSON object in the format — the container, the header, every record — is written and read by the same rules. ### Writing - Keys in each object's printed order; absent optional keys **omitted, never `null`**; **no whitespace** anywhere. - **Non-ASCII characters literal, as UTF-8 — never `\uXXXX`.** - Strings are escaped exactly as Python's `json.dumps(…, ensure_ascii=False)` escapes them, and in no other way: - `"` as `\"` and `\` as `\\`; - U+0008, U+000C, U+000A, U+000D and U+0009 as `\b`, `\f`, `\n`, `\r` and `\t`; - every other character from U+0000 to U+001F as `\u00XX`, with the hex in **lowercase** (`\u001f`); - nothing else — not `/`, not DEL (U+007F), not U+2028 or U+2029. - A lone surrogate, which some platforms' strings can hold and UTF-8 cannot carry, is written as U+FFFD. Many serialisers escape more than this by default — non-ASCII characters, `/`, or characters such as `<` and `&`. What they write is still JSON, and a reader can open it, but it is not this format's serialisation: the test vectors' payloads do not reproduce with ASCII escaping on. Check the setting, or write the few lines by hand. ### Reading Reading is strict. A reader refuses: - a **duplicate key**, comparing names after their escapes are decoded; - a raw control character (U+0000 to U+001F) inside a string; - anything but whitespace after the value; - anything that is not JSON, the literals `NaN` and `Infinity` included; - nesting past the bound (*Nesting*). It accepts what JSON allows: `\/`, uppercase hex in `\u` escapes, and whitespace between tokens. In the container these refusals are `container`; in a payload line they are `record`. A `\u` escape of half a surrogate pair decodes to a lone surrogate, which no Unicode text holds. Where this format requires text — the header's keys and values, and the values of a record's own keys — a lone surrogate is refused (`header`, `record`). Many platform parsers keep the last of two duplicate keys without a word, lose member order, turn every number into a double, or recurse without limit. A reader of this format needs all four right, and the vectors test each; a small parser of its own is often the shortest road. ### Nesting **No value sits more than 16 levels inside the header's object or a record's object.** The object itself is level 0: in `{"a":[1]}` the list is at level 1 and the `1` at level 2. A scalar one level too deep is refused just as a list would be. The bound holds whatever the key, unknown ones included, so a later version's additions stay shallow. Past it: `container` in the header, `record` in a record. A reader that parses by recursion stops at the bound; one that parses first and measures afterwards walks the value with a stack of its own, as the reference reader does, so no depth can exhaust it. ## Base64 The salt, the nonce, every segment and every photograph are **standard base64** (RFC 4648 §4, the alphabet `A–Z a–z 0–9 + /`) with `=` padding to a multiple of four characters, **strict and canonical**: - nothing but the alphabet and the padding — no whitespace, no line break, no URL-safe characters; - re-encoding the decoded bytes gives the same text, so the bits under the padding are zero, and exactly one text stands for any bytes; - the empty text is the empty byte string. Anything else is `base64`. A decoder that skips what it does not expect is not strict — .NET's `Convert.FromBase64String` skips whitespace, and many decoders ignore stray bits under the padding. The simplest check is to decode, re-encode and compare. ## The passphrase and the key ### The passphrase's bytes A passphrase becomes bytes one way everywhere: 1. **NFKD** (Unicode compatibility decomposition); 2. every run of whitespace — U+0020, U+0009, U+000A, U+000D — becomes one U+0020, with none left at either end; 3. **UTF-8**. **Case is kept.** Nothing else is changed. The whitespace rule is there so that a phrase typed with a doubled or a trailing space opens the file rather than failing as a wrong passphrase. The set is spelled out because every platform's idea of whitespace differs; NFKD has already folded the no-break space (U+00A0) and the ideographic space (U+3000) into U+0020. Any other separator — U+2028, for one — is kept, and makes a different phrase. These all become the bytes of `correct horse battery staple`: ``` "correct horse battery staple" " correct horse battery staple " "correct horse\tbattery\r\nstaple" "correct\u00a0horse battery\u3000staple" ``` and these do not: `"Correct horse battery staple"`, `"correct\u2028horse battery staple"`. The vectors hold every one of them, and an NFC/NFD pair (`café canción`, composed and decomposed) that must each open the one file. To the format a passphrase is any text. The app offers a generated passphrase of random words; that is a convenience of the app, not part of the format. ### The key The key is **PBKDF2-HMAC-SHA256** (RFC 8018) over the passphrase's bytes, the header's 16-byte salt and its iteration count: **32 bytes, derived once per file.** The random salt already makes the key the file's own, so there is no second derivation — not per segment, not per record. 32 bytes is implied by AES-256 and stated here because this document claims to be enough on its own. Use a PBKDF2 interface that takes **bytes**. One that takes characters turns them into bytes in its own way — Java's `PBEKeySpec` takes a `char[]`, and how those characters become bytes is the provider's decision — which may not be this one. PBKDF2 over a plain HMAC-SHA256 is a few lines wherever no byte interface exists. The derivation is the slow step by design. At the cap of 4,000,000 iterations it takes about twenty seconds on a phone that takes three at 600,000, so a reader should let the user cancel it, and everything after it. ## Segments ### Cutting the payload The compressed payload — or, under `none`, the payload itself — is cut into segments of **exactly `segment` bytes. The last holds the remainder, 1 to `segment` bytes**, so a payload exactly *k* segments long is *k* segments, never *k* + 1 with an empty one. An empty payload — possible only under `none`, with no records — is one empty segment. The cut is unique, which is what lets the vectors reproduce byte for byte. A file holds at most 2³² segments. ### Sealing a segment Segment *i* of *n*, counting from 0, is sealed with **AES-256-GCM** (NIST SP 800-38D) — 12-byte nonce, 16-byte tag — under the key, with the AAD of *The canonical serialisation*, and with the nonce `nonce XOR (00 00 00 00 00 00 00 ‖ i as four bytes, big-endian ‖ 01 on the last segment, 00 on every other)` — the header's 12 random bytes, with the index folded into bytes 7 to 10 and the mark into byte 11. The stored segment is **its ciphertext followed by its tag**, written as base64: ``` mask(i) = 00 00 00 00 00 00 00 ‖ uint32_be(i) ‖ (i = n − 1 ? 01 : 00) nonce(i) = header.nonce XOR mask(i) sealed(i) = AES-256-GCM(key, nonce(i), aad = canonical header, plaintext(i)) = ciphertext(i) ‖ tag(i) -- 16-byte tag, last segments[i] = base64(sealed(i)) ``` For example, under the header nonce `000102030405060708090a0b`, segment 3 as the last segment is sealed under `00010203040506070809090a`. **Why each part is there.** The index keeps segments apart inside a file, so a segment moved, repeated or dropped fails its tag. The mark makes a file cut at a segment boundary fail rather than look complete, and a file extended past its last segment fail too. The random nonce keeps files apart even if a salt ever repeated. And sealing each segment on its own keeps a writer and a reader to one segment of memory at any library size: several platforms' GCM implementations release no plaintext until the whole message's tag is checked, and a library with a photograph a day for five years is hundreds of megabytes. A **streaming writer** holds one segment back until it knows whether another follows, because only then does it know which is the last. ### Opening a segment A reader decodes each segment's base64 (`base64`), then checks its length **before** it opens it. The decoded bytes less the 16-byte tag must be: - exactly `segment` for a segment before the last; - 1 to `segment` for the last; - 0 to `segment` when the last is the only one. Fewer than 16 bytes, or any other length, is `segment-length`. Then the tag: - a tag that fails on a **later** segment fails after segment 0 has opened under this key, so the passphrase is right and the file was altered, reordered, cut short or extended — `altered`; - a tag that fails on **segment 0** is tried once more under **the other mark**. If it opens then, the passphrase is right and the file was cut short or extended after its first segment — `altered`. If not, the passphrase is wrong or the first segment was altered, and **nothing can tell the two apart** — `passphrase`. ## Compression Under **`zlib`**, the whole payload is compressed as **one** zlib stream (RFC 1950: the two-byte header, a DEFLATE body, the Adler-32 checksum) before it is cut. Under **`none`**, it is not compressed at all. **Writers compress at level 9.** The compressed bytes are the zlib library's, and another implementation may choose other matches and still be correct, so **no reader depends on them**, and the byte-for-byte claim of *The cross-platform contract* covers everything but this stream. The vectors' streams were made by the zlib library at level 9: levels 7, 8 and 9 reproduce them, and 6 — the default of Python's `zlib.compress()` and Java's `Deflater` — does not. That matters only to a writer that wants to reproduce the vectors' streams as well as their payloads. **zlib, not raw DEFLATE.** The two are different byte streams. .NET's `DeflateStream`, Java's `Inflater` and `Deflater` with `nowrap = true`, Python's `zlib` with `wbits = -15`, and Apple's Compression framework with its `COMPRESSION_ZLIB` algorithm all read or write **raw** DEFLATE; a reader built on one of them cannot open this format's files, and a writer built on one makes files no reader of this format opens. Use the wrapped form — .NET's `ZLibStream`, Java's `Inflater` and `Deflater` as they default, Python's `zlib` as it defaults, the system zlib library — or write and check the two-byte header and the Adler-32 yourself. A reader inflates **in bounded steps**, checks the Adler-32, and refuses as `compression`: - a stream that is not zlib — a raw DEFLATE body under a valid tag included; - a stream that asks for a preset dictionary; - a checksum that fails; - a stream still unfinished when the last segment ends; - bytes after the stream's end. A raw body under a valid tag is refused as `compression`, never as a wrong passphrase: the tag has already opened. The inflation bound is under *Bounds*. ## The payload The payload is **JSON Lines**: zero or more records, **each a JSON object on one line, followed by LF — the last included.** It is UTF-8, with no byte-order mark. - Writers emit no whitespace in or around a line (*JSON text*). - A line is parsed as JSON, so a reader accepts whitespace JSON allows inside or around it — a CR before the LF included. - An empty line, or bytes after the last LF, is `record`. A line that is not UTF-8 is `utf8`. - A line may not exceed 16 MiB, not counting its LF (`bounds`). - Records stand in the order of *Record order*. ## The three record kinds Each record is a JSON object on one line. **Keys appear in exactly the order printed; absent optional keys are omitted, never `null`.** ```json {"kind":"diary","id":"main","name":"Five-Year Diary","created":"2026-01-01T00:00:00Z"} {"kind":"entry","diary":"main","date":"2026-09-20","text":"…","created":"…","updated":"…","photo":"p1","asked":"d0920"} {"kind":"photo","id":"p1","mime":"image/jpeg","bytes":""} ``` **Every value of a key defined here is a string** of Unicode text, with no lone surrogate (`record` otherwise). Every key printed is required (`missing-key`) except the entry's `photo`, `asked` and `question`. ### `diary` — one per diary | Key | Value | |---|---| | `kind` | `diary` | | `id` | The diary's id (*Ids*) | | `name` | The name the user gave the diary: any text, but not empty and not only whitespace (`empty`) | | `created` | When the diary was made (*Dates and timestamps*) | The record carries the user's chosen name so that it never has to survive as a file name. The name may be edited freely and may hold anything typeable — slashes, control characters, bidirectional overrides, a reserved device name — which is why the `id` is generated by the writer and never derived from it. ### `entry` — one per diary per date | Key | Value | |---|---| | `kind` | `entry` | | `diary` | The id of the diary the entry belongs to; that diary's record must be in the file (`no-diary`) | | `date` | The day the entry belongs to (*Dates and timestamps*) | | `text` | The entry, **exactly as typed**. Not empty (`empty`) | | `created` | When the entry was first written | | `updated` | When it was last changed | | `photo` | *Optional.* The id of a photograph; its `photo` record must be in the file (`no-photo`) | | `asked` | *Optional.* The id of the question from the app's question deck that the entry answered (*Ids*) | | `question` | *Optional.* A question in the person's own words that the entry answered. Not empty and not only whitespace (`empty`) | - **An entry belongs to a day, not to an instant.** One entry per date per diary; a second is `duplicate-date`. - **The text is UTF-8 exactly as typed — no normalisation, no BOM.** Normalising would be rewriting a line the user wrote; the `emoji-combining` vector carries a decomposed `été`. The text may hold line breaks, and text of nothing but spaces is allowed; only the empty string is refused. It has no length limit of its own beyond the line bound. - **An entry holds at most one question: `asked` or `question`, never both** (`record`). Neither means the entry answered no question. The words of a deck question belong to the app and are not carried in the file; a reader that does not know an `asked` id — a later deck's — keeps it as read. ### `photo` — one per photograph | Key | Value | |---|---| | `kind` | `photo` | | `id` | The photograph's id (*Ids*) | | `mime` | `image/jpeg`, `image/png` or `image/webp` (`photo-type`) | | `bytes` | The image, strict base64 (`base64`), not empty (`empty`), beginning as its type begins (`photo-type`): `FF D8 FF` for JPEG; the eight-byte signature `89 50 4E 47 0D 0A 1A 0A` for PNG; `RIFF` at byte 0 and `WEBP` at byte 8 for WebP | - Photographs travel **inside** the file, so a sealed file is complete on its own. Photo ids belong to the whole file, not to one diary, and several entries may name one photograph. - A photograph no entry names is checked like any other, and a reader may then ignore it; the app does not keep it. - A writer never names a photograph it does not carry. A writer that cannot include one drops that entry's `photo` key and keeps the entry. ### Keys this version does not define **A reader ignores a key it does not know** — any key not listed for that record's kind — so a later version's additions do not break it. Such a key may hold any JSON value; it is still bound by the reading rules of *JSON text* (no duplicate key, the nesting bound). A reader that hands records on hands them on as read, unknown keys and all; the vectors compare them that way. A reader does not refuse a record whose keys stand in another order; a writer never writes one. ## Ids **Diary ids and photo ids** follow one grammar: `a`–`z`, `0`–`9` and `-`, one to 36 characters, not starting with `-` — ``` [a-z0-9][a-z0-9-]{0,35} ``` — and not a name Windows reserves for a device: `con`, `prn`, `aux`, `nul`, `com1`–`com9`, `lpt1`–`lpt9`. An id outside the grammar is `id`, whether it stands in `diary.id`, `entry.diary`, `entry.photo` or `photo.id`. The grammar exists because an id may become a folder or a file name on any platform, and ids arrive inside files a stranger may have written. A writer generates ids and never derives one from a name. The app names its first diary `main`, as the vectors do. A reader that saves photographs can name each one by its id, with an extension taken from its `mime`. **Question ids** (`entry.asked`) are one of: ``` d(0[1-9]|1[0-2])(0[1-9]|[12][0-9]|3[01]) the question for a calendar day: d, then a month and day that exist in a leap year — d0229 is one, d0230 and d0431 are not r[0-9]{4} a question from the deck's reserve: r, then 0001 to 9999 — not r0000 ``` The digits are ASCII only: a reader that matches with a Unicode-aware `\d` lets other digits through, and the vector `question-id-unicode-digits` refuses them. An id outside the grammar is `id`. An id within it that the reader does not know is kept as read. ## Dates and timestamps **Dates** are `YYYY-MM-DD` in ASCII digits, with no time and no time zone: a date that exists in the proleptic Gregorian calendar, from `0001-01-01` to `9999-12-31`. Anything else is `date`. **Timestamps** are RFC 3339 in UTC, in whole seconds: `YYYY-MM-DDThh:mm:ssZ`, with `T` and `Z` in capitals, the date by the date rule, hours `00`–`23`, minutes and seconds `00`–`59` — so no leap second — and no fractional part. Anything else is `timestamp`. The header's `created` and every record's `created` and `updated` are timestamps. ## Record order **Record order is deterministic:** all `diary` records by id, then all `entry` records by diary id and then date, then all `photo` records by id, compared by code point. Ids and dates are ASCII, so code-point order here is also byte order, and a fixed-width date sorts by time. **A reader holds every file to it.** Each record has a place — its kind, then its id; or, for an entry, its kind, its diary and its date — and each place must be greater than the place before it. An equal place is `duplicate-date` for an entry and `duplicate-id` for a diary or a photograph; a smaller one is `order`. The streaming checks rest on this order: every diary is known before its first entry, and every photograph an entry names is known before the photographs arrive. ## Sealing, step by step 1. The records become JSON Lines by the rules of *JSON text* and *The three record kinds*, in the order of *Record order*, with **LF after every line**, the last included. 2. The whole payload is compressed as **one** zlib stream at level 9 — or not at all, under `none`. 3. The result is cut into segments as *Cutting the payload* says. 4. A fresh 16-byte salt and a fresh 12-byte nonce are drawn from a cryptographically secure random source, and `created` is the moment of sealing. 5. The key is derived as *The key* says, once. 6. The header is serialised as *The canonical serialisation* says; those bytes are the AAD of every segment, and the header as written. 7. Each segment is sealed as *Sealing a segment* says, and written as the base64 of its ciphertext followed by its tag. 8. The container is written in the writer's layout (*The container*). ## Opening, in this order **The order is part of the format.** It decides which refusal a file gets, and it is why a file of another format or another version is refused **by name** rather than read as a wrong passphrase. 1. **The container.** - The file must be UTF-8 — bytes that are not are `container`, even when the file was cut inside a character. It must parse, under the reading rules of *JSON text*, as one JSON value with nothing but whitespace around it. A syntax error, a duplicate key anywhere in it, text after it or a byte-order mark is `container`. Text that ends before its value closes — a copy that stopped part of the way — is `incomplete`, and so is a file that is empty or holds only whitespace. - The value must be an object whose first member is `header`, and `header` must be an object nested no deeper than the bound (*Nesting*). Otherwise `container`. - Then **`header.format` is read before any other key.** The string `five-year-diary/3` goes on. `five-year-diary/N` for a whole N above 3, written in ASCII digits with no leading zero, is a later version's file: `version`. Anything else — absent, another string, not a string — is `format`. Either refusal ends the reading; nothing more in the file is checked. - Then the container must hold exactly `header` and `segments`, in that order, and `segments` must be a non-empty array of strings. Otherwise `container`. 2. **The header**, before any key is derived, in this order: 1. all ten keys present — `header`; 2. `version`, `iterations` and `segment`, in that order, JSON integers as written — `integer`; 3. `version` is 3 — `version`; 4. `kdf`, `cipher` and `compression`, in that order, the strings *The header* names — `algorithm`; 5. `iterations` within 1 to 4,000,000 — `iterations`; then `segment` within 1,024 to 1,048,576 — `segment-size`; 6. `salt` strict base64 (`base64`) of 16 bytes (`header`); then `nonce` strict base64 (`base64`) of 12 bytes (`header`); 7. `created` a timestamp — `timestamp`; 8. every other key, its name and its value as *The header* allows for keys outside the ten — `header`. 3. **The key**, derived once (*The key*), and cancellable like everything after it. 4. **Each segment in turn**: its base64, its length, then its tag, as *Opening a segment* says. 5. **The zlib stream**, under `zlib`, as *Compression* says, inflated in bounded steps under the inflation bound (*Bounds*). 6. **The records**, line by line, in the order *Refusals* gives for the checks within a line. When the payload ends: bytes after the last LF are `record`; then every photograph an entry named must have had its record — `no-photo`. **One fault, one answer.** Every vector holds one fault. A file with several is refused for the first fault the reader meets, and two correct readers can meet them in a different order. One that streams the container finds text after it, or a copy cut short, only when it gets there; one that parses the whole file first finds it at once. Likewise a streaming reader works through steps 4 to 6 together and may refuse a record in the first segment before it has decoded the second, while one that opens every segment first meets the faults in the order listed. **Nothing counts until the last segment has opened.** A streaming reader sees plaintext before the file is proven whole, so everything it reads is provisional — staged, not stored — until the last segment, the zlib stream and the container have all ended as they should. A refusal at any point leaves whatever the reader keeps exactly as it was. ## Refusals A sealed file is attacker-controllable input, and **every refusal has a code**. The codes belong to the format, not to any app: the vectors name them, the reference reader prints them, and an application turns each into its own message. Two implementations refuse a file with the same code, and a test compares a code rather than prose. | Code | Refused | |---|---| | `container` | Not UTF-8; not one JSON value with only whitespace around it — a syntax error, a byte-order mark, text after the value, a duplicate key anywhere in it; not an object whose first member is the `header` object; a header nested past the bound; `segments` not a non-empty array of strings; any member beside the two | | `incomplete` | Text that ends before its value closes — a copy cut short — the empty file included | | `format` | `header.format` absent, or neither `five-year-diary/3` nor a later version's identifier — a version-2 file included | | `header` | A header key missing; a salt that is not 16 bytes or a nonce that is not 12; a key outside the ten whose value is not a string, a boolean or a signed 64-bit integer, or whose name or value holds a lone surrogate | | `integer` | `version`, `iterations` or `segment` not a JSON integer as written — refused by name, before any derivation, rather than failing the tag later as a wrong passphrase | | `version` | A `version` other than 3; a later version's identifier, `five-year-diary/N` with N above 3 | | `algorithm` | A `kdf`, `cipher` or `compression` that is not a string *The header* names | | `iterations` | Outside 1 to 4,000,000: a hand-edited header cannot make a reader derive for ever, and a zero cannot make it derive for nothing | | `segment-size` | Outside 1,024 to 1,048,576 | | `base64` | A salt, nonce, segment or photograph that is not strict, canonical standard base64 | | `segment-length` | A segment under 16 bytes; one before the last that does not hold exactly `segment` bytes; a last one over `segment`, or empty after others | | `passphrase` | Segment 0's tag fails under both marks: a wrong passphrase, or the first segment altered | | `altered` | A later segment's tag fails, or segment 0 opens only under the other mark: the file altered, reordered, cut short or extended | | `compression` | Not a zlib (RFC 1950) stream — a raw DEFLATE body under a valid tag included; a preset dictionary; a checksum that fails; a stream unfinished when the last segment ends, or with bytes after its end | | `utf8` | A payload line that is not UTF-8 | | `record` | An empty line, or bytes after the last LF; a line that is not a JSON object, or nests past the bound; a duplicate key; a `kind` that is not a string; a key of the record's kind whose value is not a string, or holds a lone surrogate; an entry holding both `asked` and `question` | | `kind` | A record kind other than `diary`, `entry` and `photo` | | `missing-key` | No `kind`; a key the kind requires, missing | | `id` | A diary or photo id outside the grammar of *Ids* — `diary.id`, `entry.diary`, `entry.photo` and `photo.id` alike; an `entry.asked` outside the question grammar, a day no year has or digits other than ASCII among them | | `date` | A date that is not `YYYY-MM-DD`, or does not exist | | `timestamp` | A timestamp that breaks the rule of *Dates and timestamps*, a fractional second included | | `empty` | An entry with no text; a diary whose name, or an entry whose `question`, is empty or holds only whitespace; a photograph with no bytes | | `photo-type` | A photograph that is not `image/jpeg`, `image/png` or `image/webp`, or whose bytes do not begin as that type begins: `FF D8 FF`, the eight-byte PNG signature, or `RIFF` with `WEBP` at byte 8 | | `order` | A record whose place is before the place of the record above it | | `duplicate-date` | Two entries for one date in one diary | | `duplicate-id` | Two diaries, or two photographs, with one id | | `no-diary` | An entry whose diary has no `diary` record | | `no-photo` | An entry's photograph with no `photo` record, known when the payload ends | | `bounds` | A payload line over 16 MiB; more than 1,000 diaries; more than 36,600 entries in one diary — a century of days; a payload inflating past sixteen times the sealed bytes read, plus 64 MiB | **Within a line, the checks run in this order**, so a line with two faults has one answer: 1. its length — `bounds`; 2. UTF-8 — `utf8`; 3. a JSON object, by the reading rules, within the nesting bound — `record`; 4. `kind`: present — `missing-key`; a string — `record`; known — `kind`; 5. every key the kind requires, present — `missing-key`; 6. every key of the kind that is present, a string with no lone surrogate — `record`; 7. an entry with both `asked` and `question` — `record`; 8. the ids — `diary.id`; `entry.diary`, then `entry.photo`; `photo.id` — then `entry.asked` against the question grammar — `id`; 9. the entry's `date` — `date`; 10. `created`, then `updated` — `timestamp`; 11. an entry's `text`, then its `question`; a diary's `name` — `empty`; 12. a photograph's `mime` (`photo-type`), its bytes' base64 (`base64`), their presence (`empty`), then their signature (`photo-type`); 13. its place — `order`, `duplicate-date`, `duplicate-id`; 14. an entry's diary, known — `no-diary`; 15. the counts: diaries in the file, or entries in this entry's diary — `bounds`. After the last line: the photographs entries named — `no-photo`. ## Bounds | Bound | Limit | Refusal | |---|---|---| | Iterations | 1 to 4,000,000 | `iterations` | | Segment size | 1,024 to 1,048,576 bytes | `segment-size` | | Segments in a file | 2³² | — (the index is four bytes) | | Nesting | 16 levels inside the header's or a record's object | `container`, `record` | | One payload line | 16 MiB (16,777,216 bytes), its LF not counted | `bounds` | | Diaries in a file | 1,000 | `bounds` | | Entries in one diary | 36,600 — a century of days | `bounds` | | Inflation | The payload inflated so far may not exceed sixteen times the sealed bytes of the segments opened so far — decoded, tags included — plus 64 MiB | `bounds` | **The bounds are what keep memory and storage flat on the reading side.** A reader holds one segment, the inflater's window and the current line, so a line's bound, not the library's size, is what a reader needs; a line that grows past it is refused as soon as it does. An application that stages a whole file before keeping it holds more — every entry until the last segment has opened, which the diary and entry bounds cap — and should stage photographs on disk rather than in memory. The inflation bound keeps a small hostile file from becoming a large one on the way in: honest text inflates a few times and a photograph barely at all. The app re-encodes the photographs it keeps, so each is a small fraction of 16 MiB. ## Reading a hostile file - **"The passphrase is right" holds for a file sealed honestly.** GCM does not commit to its key, so a file built to deceive can make one segment open under many chosen keys, and the difference between `passphrase` and `altered` would then tell whoever watched whether the user's key was among them. The file is the user's own and opening it sends nothing anywhere, so the format carries no key commitment, and a reader's messages should say no more than an honest file allows. - **Text is hostile too.** Diary names, entries and questions may hold control characters, bidirectional overrides, path-like strings and reserved device names; the `hostile-book-names` vector carries `../../etc/passwd`, `CON`, control characters, a right-to-left override and a 300-character name. Never build a file or folder name from them — the ids are there for that — and show control characters and bidirectional overrides as escapes rather than obeying them, as the reference reader does. - **Nothing counts until the last segment has opened** (*Opening, in this order*). - **The derivation is cancellable**, and a header cannot ask for more than 4,000,000 iterations. ## Versioning The header names its own algorithms and carries a version, so **Argon2id can be added later as a second permitted KDF without breaking a single existing reader**, and the iteration count can rise with no format change. **A key a reader may ignore can arrive without a new version**: in the header the AAD authenticates it either way, and in a record a reader passes over it. **A key a reader must understand arrives with a new version.** **A later version is refused by name.** Its file says `five-year-diary/N` with N above 3 in `header.format`, and that is read before anything else in the header, so a newer file is told as newer — `version` — and never mistaken for a damaged file or a wrong passphrase. N is compared as digits, so no length of N can overflow a reader. A `/3` file whose `version` is not 3 is `version` too; the field is authenticated, so it cannot be edited undetected. **Earlier identifiers.** Versions 1 and 2 were drafts and never shipped. A file that names either — or names anything else that is not this format — is `format`. The vectors hold a version-2 file (`format-v2`) and a later version's (`format-v4`). ## The file: its name and its type | | | |---|---| | Suggested name | `five-year-diary-sealed-.json`, the date being the day of the export. **A diary's name never reaches a file name**: it is user text, it may hold anything typeable, and this is the file most likely to be mailed or copied to a memory stick | | Media type | **`application/json`** — the container *is* one UTF-8 JSON object. An invented extension such as `.fyd` would have some storage providers append the media type's extension and others not, giving two spellings of the one file whose whole purpose is to be recognisable in five years | | What identifies it | **`header.format` inside the file, never the extension.** A reader opens whatever file the user picks, reads it as UTF-8 and decides by that field, so a renamed or extension-less file opens, and a `.json` file of some other kind is refused **by name** as `format` — *not this format* must never be hidden inside a wrong-passphrase message | | Choosing a file | A reader that offers a file picker should let the user choose any file, not only those reported as JSON: providers disagree about what they report, and a user who renamed the file must still be able to choose it | ## The cross-platform contract Two implementations of this format agree when: - **from the same records**, they write the same payload, byte for byte; - **from the same header**, they compute the same AAD, byte for byte — which is also the header as written; - **from the same passphrase, salt and iteration count**, they derive the same key; - **from the same compressed stream** (or, under `none`, the same payload), key, nonce, segment size and AAD, they write the same segments, byte for byte, and the same base64 for each; - **from the same file**, a reader returns the same records — every key, unknown ones included, in the file's order — or, for a file with one fault, refuses it with the same code. Two things are outside the contract: **the compressed stream**, which belongs to the compression library (*Compression*), and **the container's whitespace**, which writers take from the writer's layout and readers ignore. The vectors are what make the claim checkable rather than stated (*Test vectors*): a reader that ignores the last-segment mark fails every openable vector; one that ignores the index fails the openable vectors that span segments; one that accepts loose base64 fails `base64-noncanonical` and `base64-with-newline`; and a writer that escapes non-ASCII does not reproduce the `cyrillic` and `emoji-combining` payloads. ## Test vectors `golden-vectors.json` holds **seventy-six vectors: twenty that open and fifty-six that must be refused**, covering every refusal code but `bounds`, whose fixtures would weigh megabytes — test those with fixtures of your own. The vectors use 1,000 iterations and 1,024-byte segments on purpose: **they pin the construction, not its cost**, so a full run is fast, while the app seals at 600,000 and 65,536. Their salts and nonces are fixed so that the corpus is reproducible; a real writer draws both at random for every file. ### Its shape The corpus is one JSON object: | Key | Meaning | |---|---| | `format` | `five-year-diary/3`. Check it before anything else | | `note` | Prose | | `passphrase` | The passphrase of every vector that does not name its own: `correct horse battery staple` | | `iterations` | The iteration count the vectors are sealed with, 1,000. Each vector's header carries its own | | `vectors` | The seventy-six vectors | Each vector: | Key | Meaning | |---|---| | `name` | Its name, unique in the corpus | | `container` | The container, as a JSON object inside the corpus — what a parser of the file would give. It need not be a well-formed container: that may be the fault | | `file` | In place of `container`: the file's complete text, as a JSON string, for what a parsed container cannot carry — a copy cut short, a duplicate key, text after the object, a number as it was written, nesting past the bound | | `expected_records` | For a vector that opens: every record as the file holds it, unknown keys and all, key order included | | `must_fail` | `true` for a vector that must be refused | | `refusal` | With `must_fail`: the code it must be refused with | | `passphrase` | *Optional.* This vector's passphrase, in place of the corpus's | | `passphrases` | *Optional.* Several passphrases, each of which must give the vector's result on the one file | | `later_writer` | *Optional*, `true`: a file a later version may write. It must open, and a version-3 writer need not be able to make it | | `note` | *Optional.* Prose | Parse the corpus with a parser that keeps member order — the strict one your reader uses will do — since both the container's member order and the records' key order are part of what the vectors test. ### Using it to test a reader For each vector, and for each of its passphrases — its `passphrases`, else its `passphrase`, else the corpus's: 1. Take the container: from `file`, by reading the text as *Opening, in this order* reads a file; from `container`, by using the object as the parsed container. A reader that takes only text may serialise the object back to JSON — keeping member order — and read that. 2. Open it with the passphrase. 3. A vector with `expected_records` passes when the records equal them exactly: the same number, in the same order, each with the same keys in the same order and the same values. A vector with `must_fail` passes when it is refused with exactly its `refusal`. A vector with several passphrases passes only when every one of them gives that result. `python3 five_year_diary_open.py --vectors` runs exactly this over the whole corpus (*Reference reader*). ### Using it to test a writer For each openable vector **not** marked `later_writer`: 1. Serialise its `expected_records` by the rules of this document. The result must be, byte for byte, the vector's payload — the plaintext of its segments, inflated under `zlib`. 2. Serialise its header. The result must be the AAD the reader computes. 3. Seal the vector's **own** compressed stream — the plaintexts of its segments, concatenated (under `none`, its payload), not a stream you compress again — under its passphrase, salt, nonce, iteration count, segment size and `created`. The result must be exactly its segments. A writer that compresses with the zlib library at level 9 will also reproduce each vector's compressed stream; one that does not may write a different, equally valid stream, which is why step 3 reseals the vector's own. ### What it holds The twenty that open: | Vector | What it holds | |---|---| | `empty-diary` | A diary with no entries | | `single-ascii` | One entry in ASCII | | `multiline` | An entry with line breaks | | `cyrillic` | Cyrillic text | | `emoji-combining` | Emoji with a variation selector, a joined emoji sequence, a composed `Café` and a decomposed `été` | | `no-compression` | `compression: "none"` | | `five-years-same-day` | The same calendar date in five successive years | | `leap-day` | 29 February 2028, and 28 February of a year without a 29th | | `multiple-diaries` | Two diaries | | `with-photograph` | An entry with a photograph | | `passphrase-nfc-nfd` | One file, opened by a passphrase composed (NFC) and decomposed (NFD) | | `passphrase-whitespace` | One file, opened by the passphrase typed with extra and unusual spaces | | `many-segments` | A photograph across four segments | | `exact-multiple` | A payload of exactly two segments, and no empty third | | `questions` | The questions entries keep: a dated question, the reserve's, the person's own words, none, the leap day's | | `question-beyond-this-deck` | A reserve question id the app's current deck does not hold | | `hostile-book-names` | Diary names `../../etc/passwd`, `CON`, control characters, a right-to-left override, 300 characters | | `unknown-header-key` | *Later writer.* Unknown header keys, standing out of code-point order | | `later-records` | *Later writer.* Records with keys this version does not know, beside a photograph no entry names | | `nesting-at-limit` | *Later writer.* A record nested exactly to the bound | The fifty-six that must be refused, by code: | Code | Vectors | |---|---| | `container` | `no-segments`, `container-duplicate-key`, `trailing-text`, `header-too-deep` | | `incomplete` | `incomplete-file` | | `format` | `format-v2` | | `header` | `missing-header-key`, `short-salt` | | `integer` | `iterations-as-string`, `iterations-as-float` | | `version` | `format-v4`, `unknown-version` | | `algorithm` | `unsupported-kdf` | | `iterations` | `hostile-iteration-count` | | `segment-size` | `segment-size-out-of-range` | | `base64` | `junk-in-base64`, `base64-with-newline`, `base64-noncanonical` | | `segment-length` | `short-inner-segment`, `empty-final-segment` | | `passphrase` | `unauthenticated-header-key`, `tampered-tag`, `passphrase-other-whitespace`, `passphrase-case` | | `altered` | `tampered-later-segment`, `dropped-final-segment`, `cut-to-first-segment`, `reordered-segments`, `extended` | | `compression` | `raw-deflate`, `data-after-zlib`, `preset-dictionary` | | `utf8` | `not-utf8` | | `record` | `no-final-lf`, `not-json`, `duplicate-key`, `nesting-too-deep`, `empty-line`, `lone-surrogate`, `question-both-keys` | | `kind` | `unknown-kind` | | `missing-key` | `missing-key` | | `id` | `id-outside-grammar`, `question-day-no-year-has`, `question-id-unicode-digits` | | `date` | `malformed-date` | | `timestamp` | `fractional-timestamp` | | `empty` | `empty-text`, `question-without-words` | | `photo-type` | `photo-signature`, `photo-type` | | `order` | `out-of-order` | | `duplicate-date` | `duplicate-date` | | `duplicate-id` | `duplicate-id` | | `no-diary` | `entry-without-diary` | | `no-photo` | `photo-reference-missing` | | `bounds` | — | ## Reference reader `five_year_diary_open.py` is the proof that the format is not a lock-in: anything that runs Python can read the diary, with nothing to install. It needs **Python 3.8 or later and its standard library only**. AES-256 and GCM are implemented in the file: its AES-256 reproduces FIPS-197 appendix C.3, and its GCM test case 16 of the GCM specification. It makes every refusal in *Refusals*, in the order this document gives, and its functions follow this document's sections — `passphrase_bytes`, `canonical_header`, `segment_nonce`, the header checks, the record checks — so it can be read beside it. ``` python3 five_year_diary_open.py prints every entry; the passphrase is read from the terminal, never from an argument python3 five_year_diary_open.py --vectors [golden-vectors.json] opens every vector in the corpus and checks its records, or the refusal it names ``` - **Opening a file**, it prints each entry on a line — the diary's name, the date and the text, with `[photograph]` when the entry names one and `[asked …]` for its question — then the number of entries, diaries and photographs. Control characters and bidirectional overrides are shown as escapes, not obeyed. - **A file that cannot be opened** is refused as `: `, printed as `refused — : ` on standard error, with exit status 1. The codes are the format's own. - **`--vectors`** prints `PASS` or `FAIL` and the name of each vector, then a count — `76 of 76 vectors as the corpus says.` when all pass — and exits with status 0 only when every vector passes. Without a path it reads the `golden-vectors.json` beside it. It proves the format; it is not a tool for a large archive. Its cipher is pure Python and reads about **100 KiB a second**, derivation included, so 100 MB of photographs takes about 17 minutes. ## Licence This specification, the reference reader and the test vectors are dedicated to the public domain under [CC0 1.0](https://creativecommons.org/publicdomain/zero/1.0/): to the extent possible under law, Lustig Fairchild Ltd. has waived all copyright and related rights to them. Anyone may copy, publish, translate and implement them, for any purpose, without asking. Published from Bulgaria.