atomic-json-store: Local State That Survives a Crash

What happened

Every workshop accumulates state files. A counter here, a last-run timestamp there, a list of things already processed so a job does not repeat itself. They start as one line of json.dump into an open file because that is one line and it works.

Then it stops working in one of three ways, and the order is predictable.

First, a process dies between opening the file for writing and finishing the write. The file is now empty or half a document, and the next reader raises a parse error at the worst possible time. Second, two processes touch the same file: a scheduled job and an interactive command, or two workers that were never supposed to overlap. Each reads, changes its part, and writes the whole document back. One of the changes vanishes with no error. Third, a new version of the program changes the shape of the document, and every machine that still has an old file breaks on upgrade.

We hit all three in a private tool during the same week. None of the failures was exotic. Each has a well-known fix. The point of this release is that the fixes belong together.

Why it mattered

The individual remedies are well documented: write to a temporary file and rename it into place, take an advisory lock around read-modify-write cycles, carry a version number and upgrade old documents on load. Any experienced engineer can write each one. The trouble is that they are written again in every project, usually one at a time, usually after the incident that made each one necessary, and usually with a subtle gap. The temp file lands on a different filesystem so the rename is a copy. The lock is taken for the write but not the read. The version check exists but there is no path from version 1 to version 3.

A state file that is written by one process with no lock, no fsync, and no version is fine until it is not. Fixing it after the fact means finding every writer. Packaging the three mechanisms behind one class means the fix is the same line of code as the original mistake.

What it does

atomic-json-store exposes one class, AtomicJsonStore, with a small surface: load, save, update, transaction, get, set, reset, and info. The document on disk is wrapped in a plain JSON envelope that records the library format, the schema version, and the last update time. Your data lives under data, so any language can read the file.

from atomic_json_store import AtomicJsonStore

store = AtomicJsonStore("state.json", schema_version=2,
                        migrations={1: lambda doc: {**doc, "tags": []}})

def bump(doc):
    doc["runs"] = doc.get("runs", 0) + 1

store.update(bump)          # locked read-modify-write, atomic publish

with store.transaction() as doc:
    doc["last"] = "2026-09-06"   # discarded if the block raises

The library has no dependencies beyond the Python standard library. It runs on Python 3.11 and newer, and CI covers Linux and macOS on three interpreter versions. A command-line tool ships with it for shell scripts, with distinct exit codes for store errors, usage errors, and missing keys.

atomic-json-store state.json set service.port 8080 --json
atomic-json-store state.json get service.port        # 8080
atomic-json-store state.json info

How it works

Atomic publication. Every write serializes the whole document to a string first, so a non-serializable value raises before the filesystem is touched. The bytes go to a temporary file created in the same directory as the target, the file is fsynced, and os.replace swaps it into place. On POSIX the directory is fsynced afterwards so the rename itself is durable. A reader that opens the old path keeps the old inode; a reader that opens after the rename gets the new document. There is no moment at which a partial file is visible under the real name. The test suite patches os.replace to fail and asserts that the original document is intact and no temporary file is left behind.

Locking. Writers take an exclusive advisory lock on a sidecar file next to the store; readers take a shared one. The lock file is never deleted, which sidesteps the classic race where one process unlinks a lock file another process is about to open. Locks are re-entrant within a thread, so an update callback can call load without deadlocking, and they time out instead of hanging. The concurrency tests are blunt on purpose: six separate processes each perform thirty locked increments on the same counter, and the test asserts the total is exactly one hundred and eighty.

Schema versioning. The constructor takes the schema version the program expects and a mapping of migrations, each from one version to the next. When a file at an older version is loaded, the migrations run in order and the upgraded document is written back under the exclusive lock. A file at a newer version than the program understands raises SchemaVersionError before anything is written, so an old binary never silently downgrades data. A plain JSON file that was never written by the library is treated as version zero, which gives a clean path for adopting existing files.

Corruption. An unreadable file raises by default. With the quarantine policy it is renamed aside with a timestamp and the store restarts from its default document. Nothing is overwritten in place.

What changed

The pattern moved out of one private tool and into a released library with a test suite that encodes the guarantees rather than describing them. Version 1.0.0 is on GitHub under AGPL-3.0 with a README, an assistant-oriented STARTHERE bootstrap, an idempotent setup script, and a tagged release workflow. The internal tool that motivated it now depends on the public package.

The limits are stated in the README because they matter. Atomicity relies on os.replace being atomic on the target filesystem, which is true of local POSIX filesystems and NTFS but not guaranteed on every network mount. Locking is advisory: a program that ignores the lock file can still race. On Windows every lock is exclusive and the platform is not covered by CI. The whole document is rewritten on every operation, which is the right trade for configuration and small state and the wrong one for large datasets.

What comes next

The scope will stay small. Candidates for later versions are an async interface and a hook for typed validation on load. Neither is needed to make a state file safe, which is the only job this library has.

Why open source

This is a utility with no proprietary mechanism in it. Publishing it costs nothing in competitive position and saves other people the same week we spent. It sits beside sqlite-checkpoint in the OpenForge catalog: one for when the state is a SQLite database, one for when it is a JSON file. The release ran through the OpenForge release path, and the public source note lists the claims and the validation that backs them.

Get started

git clone https://github.com/GreyforgeLabs/atomic-json-store.git && cd atomic-json-store && ./scripts/setup.sh

The repository, the v1.0.0 release, and the STARTHERE guide are the public entry points.