Outside evidence, gathered deliberately, and measured against what this build actually does. Every entry states what the industry converged on, where Herumon Tower differs, and whether that difference is a decision or a defect. A finding with no comparison against our own code is not research, it is reading.
The question. The server decides a battle's drops and returns them inline in the battle response. The client adds them to its inventory and uploads a whole-save blob every 10–20 s. If the save carrying that drop never lands — crash, network loss, a 429, a save over the size ceiling — the item is gone. Does the player get it back on the next sync?
The answer in this build is no, and worse than no: invSync reconciles the server's item table by following the client's save, deleting any uid the save no longer carries. So the next successful sync erases the server's own record that the drop was ever granted. The drops_granted log ages out at 7 days, and there is no re-delivery path anywhere.
create_time, expiry_time, claim_time and a can_claim flag, and states that claims and grants are atomic "to avoid partial updates and double claims".IdempotencyId, with a separate durable transaction history. Steam keeps inventory on Valve's servers and warns that the client-side GenerateItems is "only intended for prototyping". Nakama's docs are explicit: "wallets can only be directly updated from the server to make it more difficult for unintended or exploitative transactions to occur", with an append-only ledger.ConsumeItem, PlayFab has SubtractInventoryItems, Nakama appends a negative changeset. Event-sourced designs use tombstones and per-item state machines — granted → delivered → claimed → consumed — so a gap is never read as a deletion.IdempotencyId for 14 days and then garbage-collects it, while keeping transaction history for up to four years. Steam's requestid gives at-most-once execution and returns a replayed flag, but documents no retention window and explicitly re-attempts an original that failed. The durable answer to "did this player ever receive grant X" is a pending row or a ledger, never the idempotency key.| Industry pattern | What this build does | Verdict |
|---|---|---|
| All rewards delivered through a claimable pending queue | The grant lane does exactly this — mint → pending → ack. Battle drops bypass it and are returned inline. | Defect. The correct machinery already exists and drops do not use it. |
| Acknowledgement stored server-side | The ack (grantAck) is stored in the save. pendingGrants = minted, not revoked, not in the stored save's grantAck. | Right answer, fragile reason. It survives rollback because the ack rolls back with the save. Correct by colocation, not by design. |
| Destruction recorded as an explicit event | invSync deletes any server row whose uid is absent from the incoming save. | Defect. This is set-difference reconciliation; it cannot tell "dismantled" from "never received", and it erases the evidence. |
| Server-authoritative inventory | The client holds the bag and uploads it. The server now mirrors the full per-instance state (2026-08-03) but the save is still the authority. | Known, ruled on. D's ruling 2026-08-03: "the bag needs to be fill-able from the server. Local << server." |
| Idempotency/retention window of 14–30 days | drops_granted prunes at 7 days. | Too short. Shorter than every comparable window found, and it is the only record of a lost drop. |
| Durability tiered by stakes | Real-money Wraithgold already runs on an append-only wallet_ledger with server authority; gameplay drops do not. | Correct, and matches the industry. Purchases are held to the stricter standard already. |
| Idempotent grant application | /api/hatch is idempotent per egg id off an HMAC keyed stream; grants are keyed on a gid. | Correct. The hatch design is textbook — an egg hatches into exactly one creature, forever. |
This build already has all three primitives the industry answer is made of, and they are simply not wired together:
xp, mut, g, extra),_dismQ), where the client says what it destroyed and the server verifies each row against its own previous save before honouring it.The dismantle claim is the important one, because it is the first place in this codebase where destruction is a positive statement rather than an inferred absence. Generalising that shape to every removal path is what resolves the ambiguity: an item gone with a consume event was destroyed; an item gone without one, still sitting unclaimed in pending, was never received.
invSync: never drop a server item row merely because a uid is absent from an incoming save. Extend drops_granted retention from 7 days toward the 14–30 day window the industry uses. Both are small and neither changes player-facing behaviour.Cost, because it is the usual objection. Durable Object SQLite bills rows written at $1 per million against a 50-million monthly inclusion, and rows read a thousand times cheaper — and an indexed column adds a second billed row per write. One pending row per battle at 30 battles/hour is affordable, and coalescing a battle's drops into a single row plus batching currency into a ledger keeps it that way. Keep the hot pending table minimally indexed. Revisit if grant writes approach the monthly inclusion.
What would change this. If the save could be made transactional and rollback-proof, inline delivery would be tolerable — it cannot be, over a browser on a phone. If drop volume were a handful per session, a per-grant mailbox row would beat a cursor for flexibility.
Sources: Genshin Impact Wiki (mail expiry and capacity); Heroic Labs Hiro documentation (Reward Mailbox); Microsoft Learn / PlayFab Economy v2 (inventory operations, IdempotencyId 14-day enforcement, transaction history); Steamworks Inventory Service (requestid, at-most-once, replayed); Nakama documentation (server-only wallet updates, ledger); Cloudflare Workers pricing (Durable Object SQLite rows written/read, index write amplification); Wikipedia and RuneScape Wiki (2003 RSC duplication); Old School RuneScape update history (2020, 2022, 2024 rollbacks).
invSync no longer deletes a server item row because its uid is absent from the incoming save; it writes a gone timestamp instead. The row is kept for 30 days and pruned after. This does not yet decide why an item left — it stops us destroying the ability to ask.gone on the next sync via the same ON CONFLICT update, and its origin survives untouched.GET /api/backup/inv now returns rows (live) and tombs (what the account used to hold, newest first). That second list is the point of not deleting — it is the only way to answer "was this granted and then lost?" after the fact.drops_granted retention 7 → 30 days. Not an imported number: hatches already prunes at 30, so 7 was the outlier, and the drop log is the only record of a lost drop.Still outstanding from the sequence: Stage 2, minting drops into the pending lane and moving the ack server-side, and Stage 3, making the save a projection rather than the authority.
The recommendation to "move the ack server-side" was wrong for this architecture, and I did not implement it. pendingGrants already defines pending as minted, not revoked, and not listed in the STORED save's grantAck — so the ack is only believed once the save containing it has actually landed. Moving it into a server table would make things worse: the server would mark a grant delivered before the save carrying it was durable, which is precisely the failure being fixed. The ack living in the same atomic unit as the item is the property that makes rollback self-healing, and it is a design, not an accident.
drop:<eventId>) — one grant per battle, coalescing all of that battle's drops into a single row, as the cost note recommends.vicBar window rather than on a later drain.items.csv, so the Worker resolves at drain time. That keeps the cost on the rare path — a lost save — instead of adding a round trip to every battle that drops something.Verified (tools/probe_droplane.py, 5/5): a raw descriptor minted as {lane:'gear', tier:1, grade:'Worn'} is offered while unacked, comes back resolved as a real item row (Crude Signet, ITM-0181), and disappears from pending once an ack reaches the stored save.
Now verified, and the earlier claim here was wrong. This paragraph previously said the mint-on-battle leg could not be tested because the chance tables would not load locally. Both halves of that were false. The tables load fine; the real cause was that the harness fired every battle on ONE chain, which drains its bank — and once a battle is uncredited, drops never roll, because the roll lives inside the credited branch. One chain per battle and it drops on the first try: Faded Rune of Aegis, Rough Garnet, each returning both the resolved drop and its dropGid, and each showing up as pending until acked. probe_droplane is now 8/8 including the battle leg itself.
How the correction was found is the point. rollDrops records exactly why it declined — flag('drop_data', note) — and until today nothing in this system could read a flag. 60 flag() call sites across 57 distinct kinds had been writing to the anomalies table since it was created, and FROM anomalies appeared zero times in the worker. Every guard recorded its reason and then discarded it, which is why a production question had to be answered by reasoning about source — the thing the standing rule forbids and could not actually satisfy. A read door (GET /api/backup/flags, with a kind histogram) was added, and it answered the question on first use: the flags said rate_cap, then spawn_budget, and never once said drop_data.
The question. The client holds the whole player state as one JSON blob and uploads all of it every 10–20 s. The server refuses anything over a ceiling. Three of four live accounts have crossed that ceiling by playing normally, and one accumulated eight and a half hours of play that never reached the server, because a refused save is silent and the game keeps running happily from the copy in the browser.
The pattern: nobody stores mutable player state as one blob under one limit. Everyone shards it, and that single decision solves three things at once — you write only the key that changed, so deltas stop being a problem to solve; growth never meets a single wall; and the total budget is far larger anyway.
Split by how often a thing changes rather than by what it is, because that is where the payoff sits:
| Key | Bytes | Share | Changes |
|---|---|---|---|
| HOT — run + account (floor, xp, wallet, streak) | 1,367 | 0.9% | every battle |
| HOT — party + souls | 7,325 | 5.0% | every battle (creature XP) |
| WARM — breeding pens | 2,623 | 1.8% | on breeding |
| COLD — inventory (bag + bagAdd) | 112,920 | 77.5% | on drops / dismantles |
| COLD — sockets | 9,521 | 6.5% | rarely |
| COLD — codex | 11,022 | 7.6% | on first defeat / acquire |
A battle today rewrites 145,626 bytes. Split, it rewrites 8,692 — six per cent of that. The save is huge and written constantly only because creature XP (tiny, moves every battle) shares a document with the inventory (enormous, moves rarely).
Taken on the real save rather than from published ratios:
JSON.stringify(d).length — the parsed object — so compressing the transit body does not move the ceiling at all. Nor can it be applied at rest: six call sites run json_extract(data, …) straight against the stored save (grantAck, invCap, soulCap, maxFloor, heroName, _ledB), and those exist specifically to avoid parsing a 128 KB blob on hot paths. A compressed BLOB breaks all six.keepalive cap, which is a wire-size limit — so the page-close flush, which currently cannot send at all for large accounts, starts working. That is worth having, but it is not the lockout.CompressionStream in the browser, DecompressionStream in the Worker.The provenance block stays. It is 33.5% of the save and the obvious thing to cut, and it must not be: the save carries a rich birth record {src, ts, egg{name,grade,uid}, parents, pen, floor} while the server's items.origin holds only the string 'seed'. Stripping it would destroy every creature's origin permanently. Compression makes cutting it unnecessary anyway.
Sources: Microsoft Learn / PlayFab (entity objects, the 100K legacy limit, Economy v2 limits); Unity Cloud Save documentation (5 MB across default slots, 2000 keys); Steamworks ISteamRemoteStorage and Steam Cloud documentation. All byte figures measured directly against a live account's save on 2026-08-03.
Backfilled 2026-08-04. This read was done on 2026-08-01 during the cross-device rollback
investigation and was recorded only in OPS.md, which is a task list. It belongs here, because the
fence that ships today rests on it.
The question. A player's second device, or a long-open tab, was overwriting newer server state with months-old progress and the save door accepted it. What do shipped save services do when a stale writer arrives?
onlyIf/ETag. Google Play Games hands the app both conflicting snapshots and offers a
highest-progress resolution policy rather than picking silently.Where Herumon Tower differs, and it is now a decision rather than a defect. The save door
originally stored the client body verbatim, with only two refusals (an older client ts, and
playerXp below half of the previous value), and the ratchet clamp rewrote the save downward and
answered 200 — a silent rollback by construction. What shipped instead follows the research directly:
base_ver optimistic concurrency answering 409 with the authoritative copy; a per-instance epoch
minted at GET /api/save and required on every later request, because two tabs share one cookie and
one session row and only an instance-scoped token separates them; Web Locks tab ownership so a losing tab never
claims the epoch at all; and grow-only treatment for maxFloor, playerXp and
classXp. One gap is recorded rather than closed: base_ver proves the client
read the latest version, not that its content derives from it.
The question. Entry 2 concluded the save should be split by write frequency, and the split is
now built as far as a dual write: save_parts holds seven keys beside the authoritative
saves blob. An audit then found that six code paths write saves and exactly one
of them writes the parts — the save door. The bypassing five are the ops restore door, the R2
backup restore, bulk dismantle, the ledger-reconcile sweep and admin rollback. Worse, resetRealm
deletes saves, save_hist, chains, ledger,
progress and more, and there is no DELETE FROM save_parts anywhere in the
Worker. So: do we make all five writers dual-write, or do we treat the parts as derived?
SetObjects as an improvement over its old model, where a partial failure left data
inconsistent with no way to tell which key failed. Unity's batch set is all-or-nothing. Nakama offers
cross-object transactions only through a specific server-runtime call. Firestore caps an atomic batch at
500 documents. Three vendors treat as a headline feature the thing we get for nothing.Kleppmann's 2015 articulation is the canonical one: application code updating two independent systems — a database and a search index, a database and a cache — works at first and then desynchronises, because partial failure and reordering have nowhere to be resolved. The prescribed fixes are a single ordered log or change data capture. The transactional outbox pattern is the practical form, and the reason it works states the boundary exactly: the business table and the outbox table are in the same database, so both writes ride one ACID transaction. The anti-pattern is about crossing a transaction boundary between two systems.
We do not cross one. saves and save_parts are two tables in one
SQLite database inside one Durable Object, which is single-threaded and whose storage is transactional and
serializable. Writing both in one transaction is not a dual write in Kleppmann's sense — it is what the
outbox pattern deliberately engineers. So the objection that would normally rule out “make every writer
write both” does not apply to us. What remains is not a consistency problem but a coverage
problem: six call sites, one of which is correct.
save_parts.save_parts. Today
that is inert because saves is authoritative. At read cutover it resurrects the deleted run
— the reset silently does not happen.max(parts.ver), because writeSaveParts only rewrites rows whose content hash
moved. On a live account saves.ver read 3430 while the highest part version read 3429. The
watermark has to be stored explicitly.What this changes about the plan. The two shapes are not alternatives — the correct
build is both: a single internal write function that every writer calls, updating saves and
save_parts in one transaction and stamping the watermark; plus a load path that trusts the parts
only while the watermark equals saves.ver and rebuilds from the monolith when it does not. The
facade is the fix; the watermark is what survives the writer nobody remembered. Two invariants come with it:
saves.ver must be monotonic across every writer, or a restore that lowers it passes the
check silently; and the reset must both bump the generation and delete the parts.
And stage 5 changes what the watermark means, which is worth stating now. A projection-with-fallback design assumes an authority that will eventually stop existing. Through stages 1–4 the monolith stays authoritative and the watermark answers “are the parts stale?”. When the monolith is retired the parts become the system of record and the per-key version becomes an optimistic-concurrency token for the client's per-key upload — which is exactly the PlayFab / Unity / Nakama model. That is a deliberate semantic flip, not a deletion.
The question. Two releases shipped in one day and the Discord notice was written for one of them and forgotten for the other; two of five changes reached release with no codex entry. The failure was not knowing what to write — it was that writing it was an afterthought rather than a step.
“How Patch Notes Get Made”, Path of Exile forum, 28 December 2020, written by Nick, GGG's senior Narrative Designer. Its central point is the one that matters most here: patch notes are not generated from commits. A human reads roughly twenty thousand commits per expansion, keeps only what a player would notice, traces each one back to its work item to find out what it actually means, and then goes into the game and reads every before/after number by hand — the step he says he dreads, because it is the one where things get missed. Finished notes run 7,000–14,000 words and take about eighty hours.
Herumon Tower is competitive with leaderboards, and players measure. The documented incidents are all about omission, not about the change itself: Destiny 2's Edge of Fate undocumented nerfs, where Bungie's community lead answered “we would be completely out of our minds if we thought we could slip something under the rug”; and Helldivers 2, where a fire-buildup value moved from 2.0 to 2.1 undocumented and the community found it by testing — the studio's own explanation was that small balance changes were not being collated properly. A missing line in a leaderboard game is indistinguishable from cheating. Even GGG has been caught by this: PoE2's 0.2.0 withheld exact numbers, players read it as hiding nerfs, and a game director conceded there were blatant mistakes.
Accepted practice, and it pays for itself: name the reporter, with permission. Riot does it in Valorant notes. The reasoning is that players who see their report land keep reporting, and players who report into a void stop. With eleven alpha players the rule needs to be consistent or it reads as favouritism — first clear reproducible report gets the credit line.
The question. A Cloudflare spend projection built from measured usage put rows written at 82% of the bill at 100,000 players — $44,072 of a $53,521 month, because Durable Objects charges $1.00 per million rows written against $0.001 per million read, a thousand to one. Nothing in the build records which system produces those rows. The billing telemetry cannot answer it either: durableObjectsPeriodicGroups resolves to object and day and no finer, and there is no route dimension anywhere in the dataset.
Nobody writes on every change. The write-behind (write-back) cache is the standard game-backend shape: mutations land in fast in-memory state immediately, queue in an asynchronous buffer, and a worker flushes batched mutations to durable storage on a fixed interval — commonly every 5 to 30 seconds. The stated benefit is not latency but shock absorption: a traffic spike grows the batch instead of the write rate, so the database is never sized for peak.
The write/skip decision is made per event class, not per field. The long-standing MMO rule is to write the events a player would be furious to lose — trades, deaths, level-ups, zone changes — immediately and under a transaction, and to accumulate everything that happens per tick in memory, persisting a snapshot periodically. The test offered is bluntly practical: how loudly will players complain if this is lost. AWS's Games Industry Lens gives the same advice in queue form — ingest into a durable queue and have a consumer insert in batches — and says plainly that this reduces cost because the database no longer has to be scaled to peak write throughput.
Cloudflare's own guidance points the same way, and it is specific to this runtime. The Rules of Durable Objects guide names in-memory state as the tool for batching and aggregating before persisting, and warns that in-memory state is lost on hibernation, so anything needed later must reach storage first. It also says the runtime already coalesces writes behind output gates, and that blockConcurrencyWhile() on every request is an anti-pattern — at ~5 ms per call it caps a single object near 200 requests per second. Indexes are called out as a deliberate trade: faster reads, slightly slower writes, and on SQLite-backed objects an indexed column adds a second written row per write, one for the table and one for the index.
Attribution itself is treated as a tracing problem, not a storage problem. The current practice is a request-scoped span carrying attributes, with child work inheriting the parent's context automatically, so one tag at the top of a request covers everything beneath it. The FinOps community's “cost per feature” work is explicitly this: combine cloud billing data with distributed tracing to attribute spend at the feature level across asynchronous service chains, and adjust statistically for sampling. Two design rules travel with it — attributes are near-free where the backend bills per event rather than per attribute, so accumulate onto one span rather than creating many; and keep tag cardinality low, because high-cardinality values belong on the span, not on the resource.
sql.exec call sites, zero storage.list(), no KV facade. 123 of those statements write, across 46 tables — 71 INSERT, 12 UPDATE, 40 DELETE. The widest single target is kv at 18 statements.usage(period, kind, n) with an ON CONFLICT … DO UPDATE SET n=n+? upsert, read through the key-gated /api/usage dashboard. It counts two kinds, r2_write and rank_write. The mechanism was built; it was never wired to the write paths.workerd locally with sql.exec wrapped — the cursor returns rowsRead and rowsWritten — gave, on an empty account: /api/me 4 read / 0 written; /api/save (GET) 3 / 1; login 3 / 6 across four tables; /api/grants/pending 6 / 9, six of those nine being an automatic bug report writing feedback, flag_rept and anomalies at two rows each.The per-player figures in this entry were divided by the wrong number, and the population they describe is not a normal one. Both are corrected here rather than edited away, because the original numbers were quoted to D.
| hours/day | 1,000 | 10,000 | 100,000 |
|---|---|---|---|
| 13.5 (alpha, as measured) | $721 | $7,914 | $79,845 |
| 4 (very heavy) | $160 | $2,290 | $23,603 |
| 2 (heavy) | $56 | $1,106 | $11,762 |
| 1 (typical live-service) | $6 | $514 | $5,842 |
| 0.5 (casual) | $5 | $218 | $2,882 |
The original $53,521 figure sat between the 4-hour and 13.5-hour rows because it used the wrong denominator on a 13.5-hour population. The honest statement is a range: roughly $2,900 to $80,000 a month at 100,000 players, set almost entirely by hours played per day, and writes are the dominant term at every point on it — $66,106 of the 13.5-hour case, $9,751 of the 2-hour case. Autoplay is a designed feature, so the real average will sit above a normal RPG's; it will not sit at 13.5.
A per-write audit trail is paid for in the exact currency it is measuring. One stored row per write doubles the dimension that is 82% of the bill, and the existing helper is worse than a bare insert: spend(kind, n) runs a SELECT on wmeta, a SELECT on usage, then the upsert — two reads and a write per call. Wired into every write path it would add 10–30% to the number under investigation.
The resolution the industry has already reached is that measurement granularity and storage granularity are different decisions. Tag per request, aggregate in memory, and persist per period — one row per system per period, never one row per event. The wrap belongs in one place (the object's constructor) rather than at 123 call sites, because a per-call-site tag misses every statement issued from a helper, and this codebase issues many.
observability is already enabled in wrangler.jsonc. One console.log per request carrying the system tag and its row totals costs zero database writes and no storage, and a single day of traffic answers the whole question. This is the cheapest correct first step and it is reversible by deleting a line.setTimeout/setInterval in the object class are banned outright by the hibernation rule and do_guard fails the build on them, and alarms are flagged.rowsRead at no extra cost, and reads-per-request is the figure currently climbing.Sources: Cloudflare Developer Docs — Rules of Durable Objects (in-memory batching, output-gate write coalescing, the blockConcurrencyWhile throughput ceiling, index write amplification) and Durable Objects / Workers pricing pages (rows read and written rates, included allowances, verified 2026-08-05); AWS Well-Architected Games Industry Lens (write-queueing with batched consumer inserts, and the cost argument for it); GameDev.net, MMO database consistency (write-important-events-immediately, accumulate per-tick state in memory); horizOn engineering blog (write-behind cache, 5–30s flush, spike absorption); FinOps Foundation X conference session, Engineering a Cost per Feature Capability with OpenTelemetry (billing data joined to distributed tracing, sampling adjustment); Honeycomb / Jessitron, Span or Attribute? (accumulate attributes on one span; attributes are near-free where billing is per event); OneUptime, span and resource attribute design (cardinality placement). Herumon Tower figures are from the Cloudflare GraphQL analytics API and a local workerd run of the shipped worker, both on 2026-08-05.
The question. Entry 6 established that rows written is the bill and that attribution needs a tagging layer. This is the mechanics pass: exactly how a row becomes a billed row, what the platform charges to measure itself, and what each lever is actually worth.
setAlarm() bills one row written. The key-value methods on a SQLite-backed object store into a hidden SQLite table and bill through the same row accounting.cursor.rowsWritten under workerd, written because the question kept coming up on the Durable Objects Discord with no experimentally-validated answer. His results:INSERT costs 1 for the row + 1 per index that needs updating.TEXT PRIMARY KEY without WITHOUT ROWID creates a hidden index, so the INSERT costs 2, not 1. His headline recommendation is to add WITHOUT ROWID to every table with a TEXT or compound primary key — he reports no downside. INTEGER PRIMARY KEY aliases the rowid and costs 1 either way.(a,b) costs 1 extra write and still serves leftmost-prefix lookups on a; two single-column indexes cost 2.UNIQUE creates a hidden index: +1 per INSERT.INSERT OR REPLACE costs 1 for the row rather than delete-plus-insert, but rewrites all indexes — 4 on a three-index table. A narrow UPDATE beats it.WHERE status=1) bill only when the written row matches the filter — near-free for sparse flags.DELETE reports rowsWritten 1 regardless of index count. He flags that he could not confirm whether index cleanup is genuinely unbilled or merely not surfaced by the cursor.Batching does not reduce billed rows. Entry 6 cited AWS write-queueing and Cloudflare's in-memory aggregation approvingly and left the impression that batching lowers the bill. It does not, not directly: a multi-row INSERT … VALUES bills exactly the same total as the same rows inserted one at a time. Batching coalesces the commit — latency, atomicity, output-gate round trips. Billed rows are always rows touched plus index updates. Batching only lowers the bill when it removes redundant row touches: collapsing many per-tick whole-row rewrites into a single flush of the final state. That is dirty-flag coalescing, and it is a different mechanism from a batch INSERT. Stating it loosely will produce a saving that does not arrive.
usage table is a billed write, permanently. Diagnostic counters added there are a recurring cost, not a one-off.writeDataPoint() is hibernation-safe. It is fire-and-forget, returns immediately, needs no await and no waitUntil(), and opens no outbound connection — so it is none of the things the Durable Object lifecycle documentation lists as blocking hibernation (an awaited fetch(), the WebSocket API, an outbound TCP or WebSocket connection, an in-progress request, a timer). Checked against that list deliberately, because the hibernation rule is a thousandfold cost difference here.writeDataPoint calls per invocation, 3-month retention. Currently unbilled; the announced price is $0.25 per million written and $1.00 per million read, with 10 million written per month free.wrangler dev — guard the call.INSERT with no ON CONFLICT re-inserting rows that already existed, roughly 910 million redundant writes. Normal usage for that account was about 200 writes an hour.WHERE turned a single-row UPDATE into a full-table UPDATE running repeatedly — over $5,000 in under ten seconds.An append-only design costs more writes than a snapshot when state changes often and each event is small: every action is at least one row plus its indexes, where a snapshot rewrites one row only when you choose to flush. For an autoplay grinder that is a write amplifier, not a saving. Compaction is itself a write; with reads a thousand times cheaper than writes, compacting to speed reads almost never pays here, though compacting to control storage at $0.20 per GB-month can, on a coarse cadence.
Keep append-only for what must survive a rollback — grants, wallet, entitlements, drops — written immediately, idempotency-keyed, minimally indexed, with a compound primary key under WITHOUT ROWID so the dedup key costs one write rather than two. The Durable Object is already the single-writer boundary, so no distributed lock is needed for exactly-once processing; exactly-once delivery remains impossible and the idempotency key is what covers the retry.
MultiUpdate combines account updates, storage writes and deletes, and wallet updates into a single atomic transaction — the canonical shape for coalescing related authoritative writes into one commit./api/grants/pending writes nine rows, six of them an automatic bug report. A polling endpoint should write nothing. Highest leverage because autoplay polls.INSERT OR REPLACE.WITHOUT ROWID on TEXT and compound primary keys, compound indexes in place of several single-column ones, inline UNIQUE reconsidered. The kv table, at 18 write statements, is first.usage table.Not yet measured on this build: which of these levers is worth what. The ranking is by mechanism, not by measurement, and it rests on the unverified assumption that whole-row rewrites and per-poll writes dominate the 10.3 rows written per request. The instrumentation in entry 6 is what settles it, and nothing should be promised as a saving before it runs.
Sources: Cloudflare Developer Docs — Durable Objects and Workers pricing, SQLite storage API, Rules of Durable Objects, Durable Object lifecycle, Workers Analytics Engine (pricing, limits, sampling); Larry Maccherone, Every index costs you $1 per million rows: measuring DO SQLite write costs, lumenize.com, 23 Feb 2026 (36 experiments against cursor.rowsWritten); Little Bear Apps (Nathan Schram), My $5/month Cloudflare bill hit $4,868, 18 Mar 2026; ofsecman.io, How Cloudflare D1 Burned $5000 in 10 Seconds; Cloudflare Community (the $3,200 missing-index report, via Little Bear Apps); James Gwertzman, No Free Lunch: Transactions in Online Games, HPTS 2013; Heroic Labs Nakama documentation (wallets, MultiUpdate); Bo Ingram, How Discord Stores Trillions of Messages, Discord Engineering, 2023. Prices verified 2026-08-05 and subject to change.
The question. Entry 7 ranked the levers by mechanism and said plainly that lever value was unmeasured on this build. The plan was to instrument, then collect a day, then rank. D's correction: fourteen days of traffic already exist, and the shape of the bill can be recovered from them without waiting.
From the Cloudflare GraphQL analytics API, durableObjectsPeriodicGroups:
| date | rows written | date | rows written |
|---|---|---|---|
| 2026-07-29 | 8,088 | 2026-08-03 | 100,990 |
| 2026-07-30 | 35,861 | 2026-08-04 | 175,727 |
| 2026-07-31 | 40,899 | 2026-08-05 | 280,142 |
| 2026-08-02 | 70,745 |
A 35× rise in a week against a static roster of 8–11 active players. That is not growth; it is per-player cost. Grouped by object name, player objects are 98.6% of it and the WorldDO is 1.4%, which rules out the cross-player surfaces and points the whole question at the save path.
The billing data has no route dimension, but the save ver counter is monotonic and the nightly R2
sweep snapshots it at ~00:00 UTC. So Δver between consecutive nightly snapshots is an
exact count of that player's saves for that day, and it can be divided into the same player's rows written
for the same day. One window per player per day, boundary nearest midnight:
| day | players | saves | rows written | rows per save |
|---|---|---|---|---|
| 2026-08-02 | 3 | 3,740 | 66,882 | 17.9 |
| 2026-08-03 | 5 | 3,493 | 88,735 | 25.4 |
| 2026-08-04 | 3 | 2,915 | 91,569 | 31.4 |
| 2026-08-05 | 3 | 2,847 | 109,244 | 38.4 |
Pooled: 12,995 saves, 356,430 rows written, 27.4 rows per save — and the figure is doubling across four days.
The figure above is total rows written by that player's object, divided by the number of saves it posted. Every other write the object made that day — battles, drops, hatching, breeding, dismantles, flags — is in the numerator. Calling it “rows per save” invites the next session to attack the save handler and find only a quarter of the bill there. It is a rate normalised against saves, not a unit cost, and the distinction matters because the two lead to different work.
The unit costs were then measured directly, per table, by extending the step 1 tags to a per-statement
breakdown and driving the real worker under workerd:
| endpoint | tables written | rows |
|---|---|---|
| POST /api/save (steady state) |
saves 3, save_hist 2, save_parts 2, save_parts_mark 2 |
9 → 7 after WITHOUT ROWID on the two derived tables |
| POST /api/battle (steady state) |
ledger 2, chains 2, events_seen 2, spawn_budget 1 |
7 |
So one battle in production, which posted two saves until 2026-08-06, costs about
7 + 7 + 7 = 21 rows; with the client fix that is 14, and with the derived-table migration 12. The
remainder of the 27.4 is the rest of the day's activity — drops minting item_grants and
drops_granted, hatching, breeding, dismantles, and flags. Flags were guessed at twice before being
measured, and both guesses are kept here rather than deleted because both were stated to D first. The first said
flags were “most of the gap”. The second said “about 4 rows per save”, on the reasoning that a
heavy account fires two flags per save. Both are wrong, the second by roughly eight times.
flag() collapses repeats to one row per account per kind per MINUTE, not per write, and
the live evidence had been read the wrong way round: entries at 04:00, 04:01, 04:02 are a per-minute cadence, not a
per-save one. Measured on the heaviest live account over 129 consecutive minutes:
3.10 anomaly rows per minute against roughly 6.5 saves per minute, which is
0.48 rows per save. Still worth moving off the billed store under entry 7 lever 4, and
nowhere near the majority.
The throttle was then proved rather than read (tools/probe_flagstorm.py 6/6). The
storm shape is the one the 2026-08-06 incident produced — a client retrying a call the server keeps refusing.
With the rate cap deliberately disabled so the burst actually reaches the object (with it on, 41 of 60 requests were
answered 429 before the object woke, and the run would have been testing the rate cap instead):
120 refused requests in 3.3 seconds wrote 13 rows in total — 0.11 rows per request.
anomalies 4, feedback 2, flag_rept 2, none of them scaling with the burst.
The known weakness stays named: the flag() window lives in isolate memory, so an outage
that also cycles objects resets it. flag_rept, which gates the auto-reports, is stored and does not.
What the unit costs point at, which the rate did not: three of the four tables on the battle path
— ledger, chains, events_seen — each pay one extra row for a hidden
primary-key index, exactly as save_parts did. WITHOUT ROWID on those three takes the battle
path from 7 rows to 4, and on saves and save_hist takes the save path from
7 to 5. Unlike the derived parts tables these hold real state, so each needs
create / INSERT SELECT / verify / rename inside one transaction rather than
a drop and a create. events_seen is the largest of them at roughly seven days of battles, and it repays
its own rebuild in about a week.
The save-split dual write landed 2026-08-03 and widened on 2026-08-04 when all six writers of saves
were routed through writeSave(). Every save now also writes save_parts and
save_parts_mark. Both carry compound TEXT primary keys, and WITHOUT ROWID appears
zero times in the worker — so by entry 7's measured rule each of those rows bills twice.
Mid-migration the game pays for both save shapes and benefits from neither, because the read cutover
has not happened. This is not an argument against the split: entry 2 recommendation 1 remains right, and a
split battle write is 8,692 bytes against 145,626. It is an argument that the half-finished state is the worst one to
sit in, and it has been sat in for three days.
tools/schema_audit.py, step 4)45 tables, 78 write statements. 40 of the 45 carry a primary key that creates a hidden index that
WITHOUT ROWID would remove — every INSERT against them bills 2 rows where it could bill 1. No table
in the build has two or more single-column indexes, so that lever is already clean. kv is the worst
single target at 12 INSERT OR REPLACE sites: that shape bills 1 for the row but rewrites every index,
where a narrow UPDATE of a non-indexed column rewrites none.
The one-time rebuild cost, which is what step 5 is gated on: WITHOUT ROWID
cannot be added in place — it is create-new, INSERT SELECT, drop, rename, writing every existing
row once per table per object. Measured across the whole roster today: items 1,833,
item_grants 5,142, anomalies 4,034+ (a floor — the read door caps at 500 per account),
save_hist 280 (capped at 20 versions per realm). Everything else is single digits per account.
Total on the order of 11,000 rows, about one US cent. Retention is what keeps this small:
drops_granted and anomalies prune at 30 days, events_seen at 7,
save_hist at 20 versions. The rebuild is cheap now and scales with the roster, so the argument
for doing it at fourteen accounts rather than at a hundred thousand is the whole argument.
The save path is the bill. Not battles, not the leaderboard, not the WorldDO. Two of the three levers that follow
are already identified and neither needs new research: halve the number of saves (the client sent two per
battle until 2026-08-06), and stop each save costing double (finish the split cutover, and put
WITHOUT ROWID on the 40 tables). The third — per-system attribution inside a request — still
needs the WRCOST tags from step 1 to be on release, because the historical billing data has no route dimension
and never will.
Sources: Cloudflare GraphQL analytics API durableObjectsPeriodicGroups (rows written by date and
by object name, 2026-07-24 to 2026-08-06); this game's own ops doors for save versions, inventory, grants and flags;
tools/schema_audit.py against the shipped worker. Write-cost rules from entry 7. All figures measured
2026-08-06.
The question. Two players queued, one was shown JOIN MATCH and the other never was, the match never started, and both were then left “searching” while the ops door showed no queue node for either. Commissioned after the cause turned out to be one player's PC clock, which asked the larger question: what do other games do so that a player's own machine cannot decide whether a match is real?
Matchmaking is client-driven peer discovery. Each client writes {n, r, ts} to
queue/<pid> once a second with ts = its own Date.now(), reads the whole
queue, and filters candidates on Date.now() − candidate.ts < 12000. The
lower pid creates the match; the other side validates it with
Date.now() − meta.ts < 90000. Every one of those tests compares one
machine's wall clock against a timestamp written by a different machine.
Measured on the live door, not inferred: Farmaca's node read age_ms -220103 — his clock ran
3m 40s ahead. His client therefore judged every legitimate match to be 220s stale, discarded it,
wiped its own pairing marker and kept queueing in silence. His opponent sat out the 28s team wait and was told the
opponent never readied. Which of the two symptoms appears depends on which side holds the lower pid: with the
fast clock on the creating side, no match is made at all.
Reproduced end to end in tools/probe_pvpclock.py — two browser contexts, separate pids, one
clock pinned +220s to the measured offset. Pristine build 16/16 reproduces; both pid orderings covered.
| fault | who it hit |
|---|---|
| cross-machine clock comparison in the 12s and 90s windows | any skewed clock, either direction |
a future ts passes every upper-bound test, so a dead node advertises as live forever | any fast clock |
_pvpJoining never released on the abandon branch | everyone, perfect clock included |
_pvpJoining never released by startPvpGame, the real-PvP entry | everyone — first completed match kills the queue until reload |
leaderboard at is a client stamp; server 403s anything >60s future | every player >60s fast, silently absent from all three boards |
The two latch faults are not clock bugs at all. The fourth had never fired only because no two humans had yet finished a real match.
Nobody uses peer discovery. Every production matchmaker is ticket-based, and the client polls only its own
ticket. Open Match: the frontend creates a Ticket, it enters a Pool, a Director calls FetchMatches, a Match
Function proposes, and an Evaluator de-collides overlapping proposals because one ticket may only appear in one
resulting match. GameLift FlexMatch runs the ticket through
QUEUED → SEARCHING → REQUIRES_ACCEPTANCE → PLACING → COMPLETED.
PlayFab allows one ticket per user per queue precisely so a player can never be in two matches and have to
choose, and recommends polling no faster than every 6s. Nakama cancels a player's pending matchmaking request when
they disconnect, so liveness is structural rather than timestamped.
No production system compares client wall clocks. Firebase writes time with a server sentinel
(serverTimestamp()) and exposes /.info/serverTimeOffset for clients to correct local time
— explicitly for display, never for authority. The distributed-systems position is blunt: using a
wall-clock timestamp as the source of ordering is a named anti-pattern, and the remedy is a server-assigned timestamp
or sequence number. If a client must estimate server time it uses the NTP/Cristian correction
offset = [(T2−T1) + (T3−T4)] / 2. Our code implemented neither
half — it neither stamped server-side nor corrected for offset.
Race-freedom comes from a unique constraint, not from a tie-break rule. The conventional pattern is act-first-then-insert against a uniqueness constraint, because the constraint is the one thing two concurrent writers cannot both satisfy. “The lower pid creates the match” is a client-side convention with no enforcement behind it.
Ready-check exists so both sides move together. FlexMatch bounds acceptance
(acceptanceTimeoutSeconds, max 600); League cancels the match and returns the accepting player to queue
when the other side misses the check, with escalating dodge penalties. The property that matters is that one server
owns the deadline and both transitions, so one player can never hang while the other is told the opponent failed.
Server authority is already an absolute rule here for drops, hatching, breeding and battle rewards — PvP
matchmaking is the one system that was never brought under it. The queue is the last place in the build where a
client both holds the identity (pvpPid() is a random string in localStorage) and makes the
decision. Offline play is banned and the server owns battle outcomes, so the surrounding contract is already correct;
the gap is the matchmaker itself.
Freshness must be a server-owned lease, not a timestamp comparison. Stamping server-side is
necessary but not sufficient — a client still evaluates myNow − serverTs against its
own clock. The permanent form is: the server holds lease_expires_at and a monotonic
lease_seq, both read and compared with the same clock inside the Durable Object, and the client
is told only a status. No comparison ever crosses a machine boundary, so the entire bug class disappears by
construction rather than by bounding.
Hibernation is lost by setTimeout, setInterval, a standard (non-Hibernation) WebSocket,
or an in-progress awaited fetch — so a server tick loop is not available to us at any price.
| mechanism | cost | latency to match | verdict |
|---|---|---|---|
| piggyback (a pass on each incoming request) | ~zero, runs inside a billed request | excellent with ≥2 waiters; unbounded for a lone waiter | primary |
| one demand-scheduled Alarm | billed per invocation; armed only while a ticket waits | bounded backstop | safety net |
| fixed high-frequency alarm | pure waste on an idle queue | good | rejected |
| cron trigger | max three per service, not programmatic | coarse | rejected |
One alarm per object, at-least-once execution, retried on throw with exponential backoff from 2s up to 6 retries, stored transactionally with DO storage. Arm it only to the nearest real deadline (next lease expiry, or the end of an open accept window) and let it lapse when the queue empties, so the object hibernates. This is the pattern Cloudflare Queues uses for itself. Because a lone waiter can wait, the UI must say searching — never attribute it to an opponent.
Eleven accounts. A strict synchronous 1v1 queue will usually find nobody, and no amount of architecture fixes that. Ranked from least to most invasive to competitive integrity: server-side band widening on the ticket's server-measured age, with cross-region pooling (a turn-based game is latency-tolerant, so there is no reason to shard the pool geographically); announced queue windows to concentrate the roster; and asynchronous PvP against a snapshotted opponent team. Async is the honest trade to name: a snapshot cannot make adaptive double-blind commits, so the AI piloting it becomes the real opponent and the skill signal is weaker. It belongs as a labelled, unranked mode that keeps players engaged, and must stay out of the ladder until PvP anticheat lands.
Rating: Glicko-2 carries a deviation and volatility term, which is what makes it right for a roster that plays in bursts; OpenSkill is the public-domain alternative, benchmarked at 87.56% accuracy against TrueSkill's 92.44% but at 0.97s versus 3.41s. Either works for 1v1 — plain Elo is the weakest option here because it has no uncertainty term at all.
| concurrent players | shape |
|---|---|
| 10 | one DO, piggyback only, alarm rarely armed, hibernates whenever the queue empties |
| 1,000 | still one DO — but 1 Hz self-polling is ~1,000 req/s, at the documented soft limit. Lengthen the poll to 3–5s (PlayFab itself caps at 6s) for ~200–330 req/s |
| 100,000 | shard by rating band and/or region behind a Worker router with a KV shard map, promoting long-waiting tickets to a shared overflow object. Cloudflare's own Queues rebuild went 400 → 5,000 msg/s by exactly this route |
Secondary ceilings to watch: 30s CPU per invocation, and the 10 GB SQLite cap on the paid plan.
Each stage is independently shippable and each retires a named fault.
| stage | what it does | what it retires |
|---|---|---|
| 0 | server stamps every stored ts; x-srv-now header; windows bounded both ends; both latches released; board at converted to server time | all five faults above, defensively |
| 1 | replace pvpPid() with the authenticated account key | client-owned identity |
| 2 | ticket model — server owns the queue, client polls only its own ticket; freshness becomes a lease | the clock class, permanently |
| 3 | lifecycle state machine, mutual ready-check, server-assigned match id, unique pair_key, idempotency keys | the hung-side and double-create classes |
| 4 | Glicko-2, server-side relaxation, pooled regions, labelled async mode | empty-queue at alpha size |
| 5 | demand-scheduled alarm for lease expiry and accept timeouts; shard router prepared, not split | lone-waiter latency |
Release status, CORRECTED 2026-08-06 after D's ruling: stage 0 was ABANDONED, not shipped.
His objection was exact — stage 0 leaves the CLIENT correcting itself, which is a bandaid on a design that
lets the client decide at all, and it should not ship ahead of the real fix. Stages 1–3 went to release
together in build 0.36: identity, the ticket model, and the lifecycle with a mutual ready-check and a
UNIQUE-constraint arbiter. Stage 4 (rating and small-population work) and stage 5 (alarm liveness and
sharding) remain design only; the sharding design sits in BACKLOG.md with an explicit trigger. Identity binding in stage 1 must validate
through Steam's own endpoint (ISteamUserAuth/AuthenticateUserTicket, which Steamworks requires be called
from a secure server and never from a client); generic OpenID implementations have documented spoofing pitfalls.
Sources: Open Match (Evaluator and Synchronizer); AWS GameLift FlexMatch MatchmakingTicket;
Microsoft PlayFab matchmaking docs and error cases; Heroic Labs Nakama matchmaker; Firebase server timestamp and
serverTimeOffset; RFC 5905 §8 and Cristian's algorithm (Krzyzanowski, Rutgers CS 417);
Cloudflare Durable Objects lifecycle, alarms, pricing and limits docs, and the Cloudflare Queues rebuild post;
Joshy 2024, arXiv:2401.05451 (OpenSkill); Steamworks user authentication. Measured on this build 2026-08-06 via the
/api/backup/pvpq door and tools/probe_pvpclock.py.
The question. D has specced Auto PVP: a second ladder beside Competitive PVP where the opponent is a once-daily snapshot of a real account's current loadout, piloted by that account's own autoplay settings (ultimate timing, Prioritize Buffs, Prioritize Debuffs, Targeting); the attacker queues whenever they like, may play manually or on autoplay, and carries a visible attacker rating while a hidden defender rating decides who gets served as their opponent. Commissioned before the build: what did the genre converge on, which parts transfer to this backend, and where the spec already differs from the field — decision or defect.
Every large creature- and squad-RPG arena is the same machine: the defender is a stored team the game's AI pilots, the attacker plays live against it, and nobody needs to be online to be fought. Summoners War serves each attacker a list of ten candidates around their rank and lets streaks buff the attacker; RAID: Shadow Legends shows a refreshable candidate list (free every 15 minutes), regenerates one attack token per hour to a cap of ten, settles its weekly period at a fixed Monday 08:00 UTC, and demotes anyone with fewer than 20 offense battles in the period; AFK Journey caps attacks per day and, when an account has been idle two weeks, hands its defense to house NPCs so the ladder never empties; Marvel Strike Force and SW:GoH use no rating at all — beat someone and you swap positions — with rewards paid on a per-player 24-hour timer. Auto-battlers (Super Auto Pets, Backpack Battles, Mechabellum) go one step further and fight recorded ghost builds with no human on either side. Chess correspondence (Chess.com Daily, Lichess) is the one counter-model — both sides human, moves exchanged over days, rated with Glicko — and it is the wrong fit here precisely because it needs the defender to act.
The attacker wins most async fights everywhere, because a human beats a script. The field splits on what that does to rating. RAID puts symmetric points on both sides of every battle, so a normal account's defense bleeds points all week and the deficit is treated as a fact of life. SW:GoH's position-swap ladder never penalises a losing attacker at all — which is why its endgame produced payout-minute sniping and shard chats where top players agree not to attack each other, and why the developers later moved the mode's rewards onto a conventional skill rating. The Auto PVP spec's split — a visible rating moved only by your attacks, a hidden rating moved only by your defenses — is cleaner than either shipped answer: the ladder number never bleeds while you sleep, the matchmaker still learns exactly how hard your defense is, and neither number can be farmed by the other's failure mode. The field supports the shape; no surveyed game ships it this cleanly. A decision, not a defect.
For the update rule itself the small-pool literature is unambiguous: plain Elo needs a K-factor compromise and
never says how sure it is; Glicko-2 carries a rating deviation that starts wide (new accounts converge in
about ten fights), grows again through inactivity, and gives calibrated win probabilities to match on. TrueSkill's
brand is licence-restricted to Microsoft platforms; open equivalents (OpenSkill, already sourced in entry 9)
and Glicko-2 are free. Seasonal inflation is handled everywhere by a linear squish toward a floor — Clash
Royale's release notes state it outright, new = floor + k × (old − floor)
with k toggled between 0.25 and 0.75 across seasons — and each attack should settle as one paired result:
attacker's visible rating against that defense's hidden rating, both updated, nothing else moved.
The dead ladder — nobody to fight, so nobody queues, so there is nobody to fight — is the documented death of small competitive modes, and the async snapshot is itself the cure: every account with a server save is a standing opponent whether its owner ever queues or not, so this roster is seventeen opponents on day one. What the field adds on top, and the spec should too: serve from a wide rating band and keep a lower-rated tail in the deal (the spec already says so); cap how often the same defense can be served to the same attacker — at this population roughly twice per defender per day — and scale rating gain by rating gap so farming the weakest snapshot pays nothing; and settle the week at one fixed UTC instant for everyone. The per-player payout clock is the one prior-art mechanism to refuse outright: in a pool where everyone knows each other it manufactures the sniping and win-trading it produced at scale. Threshold rewards (everyone above a line) beat top-N cutoffs while the roster is this size, and cosmetics-before-power keeps the mode from becoming mandatory grind.
Most of the machine already exists, built for other reasons, and the spec lands on it almost exactly.
| the spec needs | what this build already has |
|---|---|
| per-battle rating writes, ladder parsed once a day | the Competitive ladder already works this way: rating settles in the world object at fight end, and the board is cut from it by the daily fold, skip-unchanged. Two more rating columns ride the same rows and the same fold. |
| a once-daily defense snapshot of every account's loadout | the daily fold already holds every account's full save in hand when it runs; the snapshot is one more skip-unchanged row written from data already carried, zero extra per-account hops. The rank pass's crash rules — bounded keyset pages, a day gate, a write budget, a poison guard — are the standing pattern any new daily writer inherits. |
| defense honours the owner's autoplay settings | those settings live in the save (settings['battle.*'], sparse over defaults), so the snapshot reads them where it already reads the loadout. The ultimate row extends its existing two values to Turn 1 / 2 / 3 / Never. The defender must be driven by the player autoplay brain — the wild-battle AI never fires an ultimate by standing ruling. |
| a defense log and a notification | the settings registry already carries a PvP Defense Results notification toggle, waiting for exactly this. |
| a result the server can trust with one human present | the one genuinely new build. Competitive PvP settles by dual report — two humans cross-report one fight. Async has a single reporter, so the fight must resolve server-side: the battle engine lifted into a shared deterministic module the worker runs (or re-runs) from a stored seed and the attacker's action list. The ground is prepared — the battle RNG already accepts a seed, turn order already breaks ties on a seeded key, and the power port set the precedent and the bar: a pure module, parity-gated against the real client to exact equality before it touches anything. |
Costs stay in the established shape: a replay is a seed plus an action list, small rows; attack tickets bound both the economy and the compute; and a hard turn cap with the tie ruled for the defender is mandatory before any autoplay-vs-autoplay fight exists, because two scripts can stall where a human would concede.
Sources: Com2uS Summoners War arena documentation and the Sky Arena community wiki; Plarium RAID: Shadow
Legends Classic Arena support articles; AFK Journey arena wiki; Scopely Marvel Strike Force arena help pages;
SWGoH.wiki Squad Arena and Grand Arena Championships, and the CG “State of the Galaxy” posts on retiring
rank-swap rewards; Supercell Clash Royale release notes (trophy reset formulas); Glickman's Glicko-2 specification;
Lichess and Chess.com rating documentation; Joshy 2024, arXiv:2401.05451 (OpenSkill); Microsoft TrueSkill licence
terms; Cloudflare Workers and Durable Objects CPU-time limits documentation. Measured against this build 2026-08-14:
the world-object ladder and daily fold as shipped 2026-07-30/31, the settings registry and save
settings block, the seeded battle RNG, and the src/power.js parity port.
The question. D ordered a full project review of Tower (memory and direction files, repo clutter, code against the codex). Commissioned first: what 2025–2026 practice says about auditing a whole codebase with a manager and parallel subagent lanes, and what this review should adopt.
file:line and a concrete failure scenario; otherwise it is
a question, not a bug.MEMORY.md under its 200-line /
25 KB load limit, and cut any line whose removal would not cause a mistake.Full source list and the 15 adopted rules: docs/review/research.md in the client repo
(dfmc-client, branch dev).
docs/review/research.md in the client repo.age_ms -220103 on the ops door). Every freshness and validity test in the matchmaker compared one
machine's wall clock against a timestamp written by another. The industry answer is uniform and we implement none
of it: ticket-based matchmaking where the client polls only its own ticket, freshness expressed as a server-owned
lease rather than a timestamp comparison, and race-freedom from a unique constraint rather than a
“lower pid wins” convention. Five faults recorded, two of which are not clock bugs at all and hit
every player — including one on the success path that would have killed the queue on the first completed
match between two humans. Reproduced 16/16 at the time in tools/probe_pvpclock.py, since
retired with the code it tested. Stages 1–3 shipped in v0.36; stage 0 was abandoned on D's
ruling as a bandaid. The shipped design is verified by tools/probe_mmserver.mjs (33/33,
the real matcher against real SQLite) and tools/probe_mmlive.py (9/9, two real browsers, one
pinned 220 s fast).ver and the nightly R2 snapshots) into per-object rows written gives 27.4 rows per
save, doubling from 17.9 to 38.4 across four days as our own dual write landed. Schema audit: 40 of 45 tables carry
a removable hidden index, and the one-time rebuild is about one cent today.WITHOUT ROWID finding, that a narrow UPDATE only rewrites indexes on
changed columns, and that batching does not reduce billed rows — a correction to
entry 6's phrasing. Establishes that writeDataPoint() does not block hibernation, and records
three published runaway-write incidents ($4,868, $5,000, $3,200) proving there is no platform write cap.
Ranked levers included; none of them measured on this build yet.usage counter and a key-gated /api/usage
dashboard already exist and count only two kinds, and that per-endpoint row cost is measurable today by
wrapping sql.exec under a local workerd run. No code change yet; awaiting D's word.saves and only one writing save_parts, plus a realm
reset that never deletes the parts at all — which would resurrect a deleted run at read cutover. The
research says the dual-write anti-pattern does not apply to us, because both tables sit in one
transactional SQLite database inside one single-threaded object, so a single write facade is correct and
cheap; and that a stored watermark compared against the source version is the conventional staleness test,
which is what covers the writer nobody remembers. Measured and recorded: the watermark cannot be inferred
from max(parts.ver) — a live account read saves.ver 3430 against 3429.OPS.md,
which is a task list and not the record. The fence, the epoch, Web Locks tab ownership and the grow-only
fields all rest on it, so it now sits here with the rest.anomalies table has ever had — 60 flag sites had been writing to it and nothing had ever read one.pendingGrants reads it from the stored copy — which is what makes a lost save self-healing rather than prematurely acked. Verified 5/5; the mint-on-battle leg remains unverified in the harness.hatches already used. Verified 11/11.