tl;dr

Corollary to compaction maps. No code, yet. Update 2026-09-09: Exploratory prototype just to see what breaks. More than can be brushed away, but less than expected. See below.

Old snapshots in table formats are immutable, referencing the same objects they were derived from until they’re deleted. However, given a compaction map, we can peel back versions to invert delta commits between compactions. By rewriting snapshots to reference later compactions, we can reclaim storage by garbage-collecting the old layouts they pin, and improve cache efficacy by leaving fewer distinct objects to cache, without changing the versions we can address.

Compaction maps tl;dr

A compaction rewrites a table’s data files into a more efficient layout without changing what the table contains. Compactions are logical noops; they only conflict with concurrent transactions because they change the layout of the data. Transactions need to know where their input rows were moved in the new layout to repair the layout conflict.

A compaction map effects that translation. For each run of rows the compaction relocated, it records the source file and offset, the target file and offset, and the length. Repairing a conflict is then arithmetic over the position delete vectors- tgt_start + (pos - src_start)- instead of re-execution. The previous post uses this to make compactions commute with concurrent transactions.

Rewriting Snapshots

Compaction maps rebase transactions that commit concurrently with a compaction, but the same translation works in reverse: rewriting snapshots that already committed so they refer to the subsequent compaction, rather than the layout of the snapshot they were prepared against.

A typical history is on the top of the figure. When \(C_B\) compacts the state and commits, the compacted objects are unaddressable by future delta updates: \(T_\alpha\) and \(T_\beta\) can be read, but not referenced by subsequent transactions.

This creates a new possibility: rewriting these old snapshots to refer to the new layout. The snapshot written by \(T_\beta\) is redundant as soon as \(C_B\) commits, but more interesting is its inverse, which we can use to derive its predecessor \(T_\alpha\). The compaction map rewrites \(T_\beta\)’s inserts to position delete vectors in \(T_\beta^{-1}\) against \(C_B\). The position delete vector for \(T_\beta\) looks up what it deleted to create inserts for \(T_\beta^{-1}\). This creates the state of \(T_\beta\)’s predecessor \(T_\alpha\), but against \(C_B\) instead of \(C_A\).

However, the compaction map alone is not sufficient to rewrite \(T_\alpha\)’s inserts. \(T_\beta\) deleted d, one of \(T_\alpha\)’s inserts, so d doesn’t survive into \(C_B\) and the compaction map has no information to recover it. But recall that to generate \(T_\beta^{-1}\), the rewrite had to resolve the rows that \(T_\beta\) deleted, including d. It must know the position of d in \(T_\beta^{-1}\) and the object/transaction that needs the reference (\(T_\beta^{-1}:2\) in the figure, for \(T_\alpha^{-1}\)). The insert of c in \(T_\alpha^{-1}\) is recovered from \(C_A\) by resolving the position delete.

Rolling backward, either inserted rows survived to \(C_B\) and they’re in the compaction map, or they were deleted by a transaction that committed between compactions, in which case they were recovered during the rewrite of that transaction’s inverse.

When this reaches \(C_A\), we can reclaim not only the snapshots made redundant by the compactions, but also old compactions without losing snapshot states.

OK. Why?

Beyond reclaiming old compactions, two immediate benefits come to mind. First, if a snapshot is insert-only and all its data survives, then this process can compress that snapshot to a position delete vector using only the compaction map. Even if only most data survives, this reduces the space required to store the interstitial snapshots and improves the efficacy of caching the target snapshot. Second, GDPR and other “right to be forgotten” regulations require that data be deleted from a table, but it remains available in older, immutable snapshots. As part of the rewrite, purged data can be removed from the transactions that inserted it.

If the layout conflicts introduced by compactions are trivially repaired, then extending the compaction’s runtime by including these rewrites imposes few costs on workloads. Old layouts can’t be immediately reclaimed without causing running transactions to fail, but GC is already asynchronous and separate from compaction. The cases where a rewrite of a particular snapshot is less efficient (or difficult) can leave the old layout undisturbed.

Breaking Changes

Rewrites preserve the sequence of states, but many services downstream of table formats- replication, change data capture (CDC), audit- depend on the original layout because the encoding captures its provenance: data dependencies that reconstruct the table following the same sequence of incremental changes. Returning to the running example, the snapshot written by \(T_\alpha\) was not derived from its successor \(C_B\), but that’s what the new layout describes¹. While recoverable, the rewrite scrambles exactly what the original layout captured: change logs.

The value of table provenance usually decays quickly. Transactions don’t validate against historical snapshots (and only care that some transaction conflicted), async audit/replication consumes changes in minutes, and historical audits can afford to reconstruct the original layout. Still, practical implementations would probably need to avoid rewriting recent snapshots so these services have a chance to run before the layout is rewritten.

Exploratory Prototype

Added 2026-09-09

After scarcely more work than pointing an AI agent at this post and the cmpmap branch the result is still unusable, but the attempt adds enough tension for some assumptions to creak. Some details are in the README and spec. Saliently: internal consistency checks are strategically corrupted massaged so readers accept the rewritten layout and statistics tied to the original layout are dropped. The prototype demonstrates that the compaction map is sufficient to effect the rewrite as described.

The prototype rewrite relies on some undefined behavior, living in the gaps of the Apache Iceberg specification and around the explicit checks in the default client. Since it was “debugged to correctness” around the default client’s validation, other clients might refuse to read the rewritten snapshots and less defensively coded clients might yield incorrect results.

By way of demonstration: during implementation, decreasing sequence numbers caused a ValidationException in v3 but silent corruption in v2; the default client refused to apply deletion vectors with an earlier sequence number to a compaction from a later snapshot. To work around the reader’s check in both v2 and v3, the rewrite uses two sequence numbers for the compaction: the real one for subsequent snapshots and another that satisfies the inequality check for the rewritten snapshots.

Properly supporting archival layout changes like this would require a specification change so clients can support rewrites explicitly. Hacking around validation is not sustainable.

Space Savings

Intuitively, the rewrite should remove a full copy of the compacted region. The old compaction can be deleted- it has the same state as its predecessor and the rewrite removes references to it- and rows inserted between it and the new compaction are rewritten to position deletes. The costs are some extra metadata and converting deleted rows to inserts. Since the deleted rows were present in the region being reclaimed, the cost of restoring deleted rows is roughly a wash. Some metadata need to be explicitly preserved to losslessly restore the original layout², but these overheads are small relative to the size of row data.

Since the same row needs to be deleted from every snapshot that precedes its insertion, the number of positions grows quadratically with the number of snapshots. This is a negligible overhead in v3 deletes when expressed as roaring bitmaps, but in v2 the overhead is higher (~1.2 bytes per position). In practice the overhead depends on sparsity, schema, etc., but the details are out of scope. I don’t have production data to test this against, but anecdotally I’d expect most rows to survive between compactions and the overhead to be small.

Drawing from the favorable, but not delusionally sunny range of synthetic workloads: v2 table, 60k rows, ~228 bytes/row, 5x append 4k rows, delete ~200 rows in runs of ~50 (i.e., deleting ~1% of the compacted region). Rewriting based on the compacted layout reclaimed 98.8% of one copy of the compacted table (96.9% of the old compaction + interstitial commits).

Prototype Takeaways

An ARCHIVE layout should also support coalescing interstitial snapshots into fewer objects, where this transformation is one of many to recover space and reduce the number of objects in a table. After seeing the prototype I’m more confident that an inverted archive layout is useful and feasible, but skeptical that it could be deployed without a specification change.

Conclusion

Compaction maps were a narrow solution to a specific problem, but tracking layout lineage in table formats seems to enable a broader set of repairs and optimizations. A real implementation would also grapple with schema changes, partitioning, equality deletes, etc. I’d love to hear why this doesn’t work. Email me.