Update: Corrected simulations 2026-06-09

[Part 1] [Part 2]

Table formats like Apache Iceberg were designed before conditional operations were widely available in object stores. These operations are sufficient to support Iceberg’s linearizable table update protocol, but how would they perform? Simulating multi-table commit contention at the catalog suggests:

1. Partitions help throughput and VO tail latency. Distributing a uniform workload across 20 tables roughly doubles aggregate throughput for slower providers (1.5–2.5x) and compresses VO mean latency by 5–8x. With zipfian skew, the most popular table converges to single-table performance, but other tables are mostly unaffected by the hot table’s contention.

2. Provider choice is a larger lever than table count. S3 Express One Zone (S3x) sustains 14.6 c/s on a single table- more per-table throughput than spreading a workload across 50 tables on S3 Standard, even with an “instant” catalog. The entire commit pipeline (CAS + manifest I/O) compresses with faster storage; adding tables only helps with catalog contention.

3. Longer-tailed distributions compound under contention. Each attempt requires multiple reads/writes in the object store. Variability extends the hazard window and makes workloads less stable overall. For example, S3 and Azure Premium have similar median CAS latency (61, 64ms), but Azure Standard’s longer tails result in more failures as it approaches saturation.

4. GCS is not viable for catalog-as-file workloads. This follows from raw GCS CAS latency. Commit success degrades above ~0.4 commits/sec, whether the workload is pure FastAppend or a 90/10 mix. Adding tables barely helps: the per-table I/O cost is high enough that 50 tables only lifts the usable rate to ~0.7 c/s.

The commit protocol bottleneck is well-known among table format developers; lifting it whole or in part into a dedicated service is a popular solution. Now that we’ve measured and characterized the protocol, we can explore those tradeoffs in a later post.

Commit Contention in Catalog Files

Previously we simulated single-table commit rates. Now we add another dimension: multiple tables in the catalog. These are not multi-table transactions, but rather independent table updates that physically conflict at the catalog. For example, if transactions T1 and T2 update tables A and B respectively, T1 successfully updating the catalog reference for A could cause T2 to fail its commit to B. However, repairing T2’s commit is cheaper than what we measured last time: T2 only needs to retry at the catalog, not rewrite the table metadata or its manifest list.

This models the “catalog as file” case where the entire catalog is conditionally replaced on every commit. Note that as the number of tables increases, the inter-arrival time is distributed across all the tables in the catalog; be careful not to read it as the arrival rate for a single table, which we measured before.

Experiment Summary

The workload mix is the same as in the single-table experiments, composed of “light” FastAppend (FA) and “heavy” validated overwrite (VO) transactions. The salient difference between FA and VO is the I/O necessary to retry a transaction: a FA transaction needs to re-read only the latest manifest list while a VO transaction needs to read the manifest lists of all new snapshots of that table.

The workload is steady, but optimistic: it assumes no transaction needs to read beyond the manifest list to investigate or repair a conflict before retrying. Real workloads include commit attempts that do more work between attempts, increasing the chance of conflict.

Tables are selected from either uniform or Zipfian distributions, as annotated.

In all experiments, the manifest list and table metadata sizes are fixed (10KiB and 100KiB, respectively). Manifest and metadata I/O uses unconditional GET and PUT operations, not the conditional operations measured earlier. We use the same S3 Standard latencies for experiments 4a/4b as we used in the single-table experiments. We use provider distributions for experiment 4c (i.e., unconditional reads/writes for metadata, conditional writes for the catalog).

Multi-table scaling (4a, 4b)

Before measuring real providers, we sweep the number of tables (1-50) and catalog CAS latency (1-120ms) to establish how much multi-table scaling can buy. Experiments 4a (FA-only) and 4b (90/10 FA/VO) use simulated S3 Standard latencies for manifest I/O, with synthetic CAS latencies. Full heatmaps are in Appendix A.

Uniform distribution: tables move the bottleneck to the catalog

With a fast catalog (1-10ms CAS), distributing FA transactions uniformly across more tables lifts success rates at high load, though the 20ms×50-tables corner is no longer a free lunch: 50 tables at 20ms inter-arrival reach 88% success at 1ms CAS, 70% at 10ms CAS. At 100ms+ inter-arrival (≤10 c/s offered), 10+ tables hit 100% success at CAS latencies up to 10ms.

At CAS latencies closer to real providers (50-120ms), the frontier tightens. At 50ms CAS, the knee (>95% success rate) sits near ~10 c/s with 20+ tables. At 120ms CAS, the knee drops to ~5 c/s- the catalog round-trip dominates retry cost at every table count.

Adding VO: table partitioning reduces per-table retry cost

Adding 10% VO transactions barely changes FA success rates. VO success improves dramatically with table count. Each VO retry reads a manifest list for each snapshot committed to that table since the read snapshot; with more tables, each table sees fewer commits, reducing the per-table retry cost. At 10ms CAS with 50 tables, VO approaches FA success rates above 100ms inter-arrival; at 50ms CAS, VO and FA success rates converge above 10 tables for moderate loads. At 120ms CAS, the catalog limits both FA and VO equally.

Zipfian Skew: the most popular table converges to single-table performance

Under a Zipfian (α = 1.5) distribution, the probability of selecting the kth-ranked table is proportional to 1/k^1.5. The rank-1 table absorbs ~50% of writes regardless of how many tables exist; rank-2 gets ~18%, rank-3 ~10%, and the distribution falls off steeply. The effective table count tops out at ~4.5 even with 50 physical tables.

The rank-1 table behaves approximately like a single table at half the global arrival rate, with a small penalty from catalog conflicts. At low load, rank-1 success rates and latencies converge to the single-table baseline; at high load, catalog conflicts from other tables’ writes consume part of the retry budget, degrading success rates below the single-table equivalent.

Under Zipf, 70% of retries are same-table conflicts (requiring manifest I/O), compared to ~2% under uniform distribution with 50 tables. Adding physical tables beyond 10 barely helps- Zipf with 50 tables performs like uniform with ~5 tables.¹ This is unsurprising, given that additional tables shift diminishing fractions of the workload.

Adding back VO transactions to the zipfian distribution, we see a similar effect: the most popular table converges to single-table performance, VO transactions are more sensitive (particularly at high load) to catalog conflicts, and sustainable single-table throughput with VO transactions is much lower. Catalog conflicts are evenly distributed across tables, but the most popular table also accumulates per-table conflicts where VO transactions struggle to complete at high load.

Takeaway: Uniform distribution shows partitioning is effective, until the CAS latency becomes the bottleneck. When the distribution is skewed (zipfian), retries from popular tables have minimal impact on other tables. These results also suggest that catalog and table conflicts should be handled separately by the retry policy. While these simulations assume a steady arrival rate following a distribution, many real workloads burst in a particular table. Exponential backoff for table conflicts and immediate retry for catalog conflicts would be more effective for that workload.

4c. Multiple tables, varied workload ratio, measured CAS distributions

Real provider CAS latencies (22-170ms) fall well above the 1-10ms sweet spot from 4a/4b, so most workloads will operate in the regime where CAS latency limits throughput. Now we substitute the CAS latencies measured for each provider and published unconditional read/write latencies. The full results are in Appendix B.

We’re assigning labels to these distributions matching observations from each provider, but this is still a model of the commit protocol. We’re interested less in hitting the moving target of real provider performance and more in learning from the model: when does CAS latency become the bottleneck? (When) does storage variability (modeled as a lognormal distribution) impact commit success rates?

The synthetic parameter sweeps in 4a/4b varied CAS latency and workload to measure commit success rates/latency holding the provider (S3) constant. Now we want to see how different provider profiles interact with workload mixes and table counts.

Single-table Provider Performance

In the commit path, I/O latency for metadata is the dominant factor across providers. S3 Express One Zone is in its own class on this workload, delivering 3-6x the throughput of the next tier (S3 Standard, Azure Premium, Azure Standard) and up to 20x the throughput of GCS. Its low latency and low variance for both reads and writes compress the entire commit pipeline.

To accommodate 10% VO transactions, even S3x requires a 2x reduction in throughput to keep success rates above 95%.

Multiple tables

With uniform table selection, partitioned workloads fall into three performance tiers:

s3x (14.9 c/s at 10 tables, up to 36 c/s at 20+ tables) » s3 / azurex / azure (3.7 c/s) » gcp (0.7 c/s simulated, ~0.7 actual)

For S3, Azure Premium, and Azure Standard the usable rate plateaus at 3.7 c/s from 5 tables through 20 tables. The per-table ceiling at S3 median latencies is ~5.7 c/s (five serial S3 ops per attempt), so the aggregate knee lines up with the catalog-CAS rate taking over only when per-table offered load drops far enough. That happens at 50 tables for S3 Standard FA-only, where the knee jumps to 7.2 c/s — but adding even 10% VO pulls it back down to 3.7 c/s. Azure Premium and Azure Standard stay at 3.7 c/s all the way through 50 tables.

This tier is narrower than it looks. Adding tables beyond ~5 buys little for S3/Azure on mixed workloads: the per-table metadata pipeline is the binding constraint, not the catalog round-trip.

Outliers: S3 Express One Zone (S3x) and GCP

S3x benefits from partitioning immediately and keeps benefiting. Its single-table knee is 14.6 c/s FA-only (7.4 c/s mixed); with 2+ tables mixed workloads also sustain 14.9 c/s. Beyond 20 tables FA-only, cross-table CAS contention still dominates but each failure costs only a re-CAS (manifest I/O is skipped for disjoint-table conflicts), so the usable FA-only rate jumps to ~36 c/s at 20–50 tables.

The simulated GCP rate is 0.7 c/s at 50 tables, up from 0.4 c/s at 1 table — a much weaker curve than the other providers. GCS’s per-op latency is high enough (~200 ms GET/PUT, ~170 ms CAS median) that each attempt costs roughly 590 ms even with a fast catalog, so per-table throughput caps at ~1.7 c/s and the catalog CAS caps the aggregate at ~8.5 c/s. The real product is much lower: GCS measured CAS throughput saturates at 0.8–1.4 op/s before any commit-protocol overhead.

Main tier: S3 Standard, Azure Premium, Azure Standard

The other three stores are more interesting. S3 and Azure Premium have similar CAS medians (61 vs 64ms), but Azure Premium’s CAS sigma is 5x larger (0.73 vs 0.14). Azure is worse on both axes: higher median (93ms) and higher sigma (0.82).

In this model, both Azure Premium and Standard have higher CAS variance, but Premium has very tight read/write variance; its I/O is predictable even if the CAS is noisy. At 10 tables all three stores land at the same 3.7 c/s knee — they are all bound by the per-table metadata pipeline, not the catalog CAS. Azure Standard’s wider I/O variance shows up in mean latency (1.33s vs S3/Azure Premium’s 0.82s) and VO mean latency (6.8s vs 4.7s).

Put another way: when Azure Standard retries take 5-10x the median, they’re almost certainly waste. In settings with high variance, commit protocols need to minimize how often a retry attempt samples from a fat-tailed distribution. Azure Standard’s higher variance shows up in FA mean latency (~1.33s vs S3/Azure Premium ~0.82s) and in VO mean latency rather than in the sustainable rate, which lands on the per-table bound at 3.7 c/s just like S3 and Azure Premium.

Takeaway: S3x is in a different class for catalog-as-file workloads. Distributing load across tables is effective, but store variance can drive failure rates up even at low arrival rates.

Conclusion

We tested two levers for improving commit throughput under contention: adding tables and choosing a faster storage provider. Both help, but the gap between them widened with the corrected per-attempt I/O cost.

Provider choice matters more than table count. S3 Express sustains 14.6 c/s (FA-only) on a single table — more per-table throughput than spreading the same load across 20 tables on S3 Standard (3.7 c/s aggregate, ~0.18 c/s per table). Fast storage compresses the entire commit pipeline (CAS + manifest I/O), while adding tables only relieves catalog contention. For providers in the 1–2 c/s single-table tier (S3, Azure Premium, Azure Standard), 5+ tables yield ~2x aggregate scaling to 3.7 c/s and stall there; only at 50 tables does S3 Standard’s FA-only knee climb to 7.2 c/s.

Table partitioning reduces VO retry cost. Each VO retry reads manifest lists proportional to snapshots committed to that table since the read snapshot. With more tables, each table sees fewer commits, and VO tail latency drops accordingly. For S3 Standard 90/10 mix, VO mean latency falls from 20.2s at 1 table to 4.7s at 10 tables and 2.6s at 20 tables. S3 Express drops from 6.7s to 0.8s over the same range. Under Zipfian skew, the hot table still converges to single-table performance, but catalog contention does not impact the less popular tables.

The protocol still matters at single-table scale. On a single table, VO mean latency reaches tens of seconds at moderate throughput regardless of provider (S3: 20.2s, Azure Premium: 21.7s, Azure Standard: 22.6s, GCP: 32.9s). S3 Express’s 6.7s single-table VO mean is the lowest of any provider, though still impractical for most production workloads.

Take the simulated provider experiments with a grain of salt: the labels we’re putting on the storage distributions are drawn from measurements of real systems, but these parameters do not completely describe reality. These are optimistic models of provider performance. It is unlikely that real workloads could sustain these rates without external coordination.

One practical takeaway: catalog and table conflicts should be handled separately by the retry policy. Catalog conflicts are cheap to retry (re-read the catalog, re-apply the CAS) while table conflicts require re-reading manifest lists. Immediate retry for catalog conflicts and exponential backoff for table conflicts would better match the cost structure.

More broadly, these simulations may be sufficient to indict the commit protocol. Writing copy-on-write objects to storage on every commit attempt is a self-imposed obstacle to scaling commit throughput, more than writing to the same catalog object in every commit.

Correction 2026-06-09

The numbers in this post were regenerated on 2026-04-17 after finding some simulator bugs, listed below. The qualitative conclusions survive- the tier ordering and the “partitioning helps” story are intact- but the multi-table knees moved substantially, in both directions: the 5-20 table knee for S3/Azure drops by half, GCP scaling is much weaker than presented, and S3x at 20+ tables jumps to roughly 2.5x the published rate.

• Per-attempt I/O cost corrected. Each commit attempt now issues five S3 round-trips (TM_read + ML_read + ML_write + TM_write + CAS) instead of three. The per-table ceiling at S3 medians tightens from ~7.7 c/s to 5.7 c/s. For S3/Azure at 5–20 tables the knee drops from ~7.4 to 3.7 c/s: the per-table bound bites before the catalog CAS does. Only at 50 tables does catalog CAS again become the bottleneck (for S3 FA-only).
• Cross-table CAS free retry. Multi-table CAS failures caused by commits on different tables no longer charge manifest I/O, and only charge for the CAS conflict. This is why S3 Express at 20+ tables FA-only jumps from 14.9 to ~36 c/s: its CAS throughput was already the binding constraint, and removing the spurious manifest I/O lets more commits through.
• Timing leaks removed. The CAS version check and catalog.read() both split-yield at half-RTT, eliminating a fast path where clients could commit against state propagating faster than the full CAS round trip.
• VO convoy decomposed per-table. The IO convoy was using a global value and ignored the per-table configuration. Corrected, multi-table VO tails are lower.
• GCP scaling is weaker than presented. With the tighter per-table bound, 50 tables only reach 0.7 c/s for GCP (was 3.6 c/s in the old model).
• table_metadata_inlined config drift fixed. An orphaned template flag had silently switched intermediate re-runs to the 1/(3L) inlined bound; the post now uses non-inlined metadata, matching the original intent.

The tier ordering (S3x » S3/Azure Premium/Azure Standard » GCP) is unchanged, which makes sense given the measured CAS latencies. Adding tables improves scalability, but less than the original numbers implied, especially at 5–20 tables.

One presentational change: VO latency figures in this post are now reported as the mean, where the previous version cited VO P99. With the per-table convoy fix, the mean is a more stable summary of the distribution.

Appendix A: Full 4a/4b results

Full heatmaps for experiments 4a (FA-only) and 4b (90/10 FA/VO mix) across table counts, inter-arrival times, and catalog CAS latencies. Both uniform and Zipfian table selection distributions are included.

4a: FA-only, uniform (10ms, 1ms CAS)

4a: FA-only, uniform (50ms, 120ms CAS)

4a: FA-only, Zipfian (50ms CAS)

4b: 90/10 FA/VO mix, uniform (10ms, 1ms CAS)

4b: 90/10 FA/VO mix, uniform (50ms, 120ms CAS)

4b: 90/10 FA/VO mix, Zipfian (50ms CAS)

Appendix B: Full 4c results

Galleries of success rate and latency heatmaps for all 5 storage providers, across all table counts, inter-arrival times, and workload mixes. Click on an image to view the gallery and flip through them.

S3

S3 Express One Zone

Azure Standard

Azure Premium

Google Cloud Storage (GCS)

Update: Corrected simulations 2026-06-09

The root and leaves of table formats- catalog services and replacements for Parquet files- are getting a lot of attention in industry and academia. However, the inner nodes of table formats like Apache Iceberg limit write throughput to 1-2 commits per second, even running well-behaved workloads in optimistic conditions.

1. Sustained commit rates above 1-2 commits/sec are unattainable without making some transactions uncommittable.

2. Storage I/O is the primary bottleneck for single-table workloads, even with an unrealistically fast catalog. Catalog compare-and-set (CAS) latency up to 120ms adds only modest overhead for single-table workloads.

3. IO cascades extend tail latency. Conflicts requiring work proportional to the number of snapshots committed not only increase tail latency, they also create IO convoys that space out commit attempts.

Why bother?

Table formats are designed for read-dominant workloads with low update rates: non-blocking reads, atomic updates, and cheap, scalable metadata at rest. Conflict is rare by construction. If table update rates are steady, it’s usually an ingest workload with a dedicated writer and occasional conflicts caused by maintenance transactions. In multi-writer workloads, write rates are expected to be low, and an exponential backoff will probe around spikes in load.

Table format commit protocols are like an airport without a control tower. Absent coordination, either aircraft land infrequently (read-dominant workload) or each aircraft takes off knowing when it’s supposed to land (dedicated or externally coordinated writers). The table format community seems to have reached a consensus that airports must have control towers (i.e., catalog services) to sustain higher throughput. So: when do we need a control tower?

Given what we learned about the latency of conditional operations in object stores, when does the cost of resolving conflicts limit throughput for a file-based catalog? This post will focus on single-table workloads. We’ll explore multi-table catalogs and architectural alternatives in future posts.

Background: Table Formats and the Catalog

Table formats like Apache Iceberg, Apache Hudi, Delta Lake, and Lance specify conventions used by participants to ensure both that readers access consistent snapshots, and that updates to table data are ordered. Unlike traditional database systems, writers autonomously self-certify the consistency of their transaction against the current state of the table before atomically installing a new snapshot. There is no coordination across transactions outside of storage; in many settings, the set of running transactions is neither recorded nor discoverable.

Write transactions follow a coarse-grained, optimistic 3-phase lifecycle: versioned action, validation, and commit. Versioned actions track the snapshot version for reads and sequester writes outside the tree of objects reachable from the visible table. Commit is a race: transactions attempt to atomically change the current version to reflect their updates. Successful commits update the shared/visible table state; failed commits must retry after repairing their prepared transaction.

Also unlike traditional database pages, data referenced in a snapshot contains only committed data. Readers may filter tombstoned data, but a snapshot never references uncommitted data written by an active transaction.

Catalog

The Iceberg Catalog- the saddest table of all time- manages a tiny amount of state i.e., the locations of the root object for every table¹. Like the root of a CoW B-tree, an update to a table shadows all affected nodes including the root.

The catalog is a natural bottleneck for concurrent transactions… right?

Yes and no. Conflicts on this pointer will cause losing transactions to verify and repair table data committed since their read snapshot. If the data files and manifest files remain valid as-written, then the transaction only needs to merge metadata references. Concretely, it must read the current table metadata (JSON blob at the root of the table; ~1MiB but potentially larger) and a subset of the manifest list(s) committed since the read snapshot (tens, multiples of ~100KiB each). The transaction merges and writes a new manifest list and table metadata, and retries the commit at the catalog.

Conflicts detected at the catalog limit throughput, but the latency of the root is not necessarily the bottleneck.

Workload Mix: FastAppend (FA) and ValidatedOverwrite (VO) Transactions

Our workload will mix “light” and “heavy” transactions from Iceberg.

A FastAppend transaction is our “light” transaction type that appends new data files and metadata (blind writes). Conflicts with other transactions are resolved by reading the table manifest and the latest manifest list, merging and writing both to create a new snapshot, then retrying the commit at the catalog.

A ValidatedOverwrite transaction is our “heavy” transaction type: an overwrite with validation enabled. It rewrites and replaces data files and metadata, so it needs to examine all the manifest lists committed between its read snapshot and the current snapshot to check for conflicts. It can then write the merged manifest list and table metadata and retry the commit at the catalog.

In what follows, we assume that all conflicts are spurious, require no reads beyond the manifest list, and failures are caused only by exhausting the retry budget at the catalog.

Simulating Commit Throughput

Using a discrete event simulator (Endive²), we can submit a transaction workload against a virtual catalog and object store. The simulator models the transaction lifecycle, including the cost of preparing retries after conflicts, and the latency of storage operations.

1a We start with an optimistic baseline: what happens to transaction throughput/latency if we use an implausibly fast catalog (1ms) and only FastAppend (light) transactions in a single table. We sweep arrival rates from 20ms (50 commits/sec) up to 5000ms (0.2 commits/sec).

1b Then we add ValidatedOverwrite (heavy) transactions into the mix. The rate a single table can sustain is lower than whatever commit rate causes these transactions to fail. By analogy, the rate at which small planes land safely is irrelevant if your airport is ringed by wrecked cargo planes that circled until they ran out of fuel. Using the same 1ms catalog, we sweep the ratio of FA (light) and VO (heavy) transactions to see how the workload mix affects success rates and latency.

2a,2b With this baseline for a single table and an instant catalog, we add another dimension: catalog latency. We sweep the compare-and-set (CAS) latency from 1ms up to 120ms. We run this sweep for both FA-only and 90/10 FA/VO workloads to measure success rate and commit latency sensitivity to catalog latency, still for a single table.

Experiment Summary

In all experiments, the manifest list and table metadata sizes are fixed (10KiB and 100KiB, respectively). These do not use the conditional operations measured earlier, but unconditional GET and PUT operations for S3:

All experiments use 5 seeds, retry=10, txn runtime mean=180s. Each run simulates 1 hour, with the first and last 15 minutes excluded as warmup/cooldown. Config counts exclude seeds.

Transactions retry (10x) immediately rather than backing off, since the workload submits transactions at a steady rate. By default, Iceberg transactions retry 4 times with an exponential backoff that starts at 100ms, doubling up to 1 minute between attempts (max 30 minutes). This would be strictly worse than the “immediate retry” strategy for this workload.

Note: Each set of heatmaps often includes more than the two displayed by default. Click on an image to view the gallery and flip through them. The full set of plots is also here.

1a. Single table, FastAppend (FA) workload

In this workload, we have a single table and only FastAppend transactions. The catalog is “instant” with a fixed latency of 1ms. If we’re assuming clients follow the protocol as specified, then the only bottleneck should be the cost of preparing retries.

Around 2-3 commits/sec, transactions start to time out after 10 retries. Each FA commit attempt reads the current table metadata and manifest list, writes a new manifest list and table metadata, and submits a CAS — five S3 round-trips per attempt, ~175ms at S3 median latencies (GET 27ms, PUT 60ms). Successful commits therefore can’t exceed ~5.7 c/s at saturation; failures climb above 1 c/s offered load.

Another way of looking at it: 20 TPS per table (50ms arrival rate) is completely unattainable. Only 33% of transactions succeed, even in this rosy setting.

Takeaway: the most optimistic single-table transaction rate is low (2-3 commits/sec), dominated by the cost of reading, merging, and writing metadata in the object store.

1b. Single table, mixed workload (FastAppend (FA), ValidatedOverwrite (VO))

In this experiment, the arrival rate is shared for a mix of FastAppend and ValidatedOverwrite transactions. The ratio of FA to VO transactions (fast_append_ratio) on the y-axis is swept from 1.0 (all FA) down to 0.0 (all VO). The same arrival rates are swept as the previous experiment, with the same, “instant” catalog. The 1.0 row is the same configuration as in experiment 1a, but with different seeds.

Unsurprisingly, FastAppend transactions are more likely to commit as the mix includes more ValidatedOverwrite transactions; retrying a FA transaction is cheaper. However, as with a mix of 90% FastAppend and 10% ValidatedOverwrite transactions, the maximum sustainable arrival rate drops to around 2 commits/sec.

We’ll continue to measure FA-only as a baseline, but I’d argue that VO throughput is the “real” sustainable commit rate per table, following the protocol as specified. Higher rates require coordination outside the table format protocol; only participating writers (i.e., the catalog operator’s writers) get reliable commit latencies.

The latency heatmaps show that VO transactions are remarkably insensitive to the workload mix; there is almost no difference in p50/p95/p99 latencies for VO transactions for a given arrival rate. Until VO transactions start to fail, even a workload composed of 100% VO transactions looks similar to the 90/10 split. Weird.

To understand why, recall that VO transactions read all the manifest lists committed since the read snapshot to prepare a retry. The first commit attempt needs to read about three minutes of manifest lists (180s mean txn time) to prepare. This takes long enough that it will almost certainly fail. Its next attempt will read fewer manifest lists- only those committed while it was preparing- but it’s probably still too many to succeed on that attempt. When the transaction finally commits, its retries have effectively spaced out commit attempts, with (roughly) exponentially reduced retry work per attempt.

VO transaction retries form IO convoys that, in effect, order VO commit attempts from the workload. This makes sense, but it surprised me; I wouldn’t have anticipated the manifest list processing could be this expensive. Even with an immediate retry, p99 commit latencies approach 2.5 minutes at a 300ms arrival rate.

Takeaway: arrival rates above 2 commits/sec will make some transactions practically uncommittable, even if we assume all conflicts are trivial to repair and require no reads beyond the manifest list. Transactions that need to do work proportional to the size of the conflict- no matter how minor- will form IO convoys in settings with steady commit rates.

2a. Single table, varied catalog latency

Next we establish a new baseline, varying catalog latency for a single table with only FastAppend transactions. Our goal is to understand how sensitive success rates and latencies are to catalog latency for a single table. The bottom row of the success heatmap (i.e., 1ms catalog latency) is the same configuration as in experiment 1a, but with different seeds.

Results match intuition: the rate at which transactions start to fail is only slightly affected by catalog latency, since the bottleneck is the cost of preparing retries. FA-only workloads start to fail at ~2 commits/sec at 120ms catalog latency, vs ~3 commits/sec at 1ms CAS.

Within an arrival rate, latency increases but retries don’t start to cause notable differences below 80ms catalog latency.

Takeaway: for a single table, the cost of preparing retries dominates the bottleneck, so catalog latency has limited impact on success rates and commit latency.

2b. 90/10 FA/VO workload, varied catalog latency

Next we add ValidatedOverwrite transactions back into the mix, with a 90/10 FA/VO workload.

Unsurprisingly, FA transaction success rates are almost identical to 2a (FA-only) at all catalog latencies and inter-arrival times. At higher catalog latencies, VO success rates degrade more rapidly, between 1-2 commits/sec. At 120ms commit latency, success rates drop earlier and more sharply, starting at 1 commit/sec and plummeting thereafter.

Mean VO latencies are modestly higher as the catalog latency increases, but continue to be mostly determined by the arrival rate.

Takeaway: Catalog latency has a more pronounced, but minor effect on VO success rates in a mixed (90/10) workload below 80ms. At rates where VO transactions are viable (above 1s inter-arrival time), catalog latency has a minor impact on commit latency.

Conclusions

Single-table commit throughput is limited by the cost of preparing retries, not by the catalog. With an unrealistically fast (1ms) catalog, FastAppend-only workloads top out at 2-3 commits/sec; adding even 10% ValidatedOverwrite transactions drops the sustainable rate to around 2 commits/sec. Below these rates, catalog CAS latency adds modest per-commit overhead but does not change the success rate. Above them, no catalog- however fast- can help.

The most interesting finding is that ValidatedOverwrite transactions form IO convoys under load. Because each retry must read manifest lists proportional to the number of snapshots committed since the read snapshot, the work to prepare a retry grows with contention. Retries effectively serialize VO commit attempts, with p99 latencies reaching minutes even at moderate arrival rates. This is not a catalog bottleneck or an object store bottleneck- it is the cost of the table format’s own metadata protocol.

These are optimistic results. All conflicts are trivial (no real data to re-read), retries are immediate (no backoff), and the catalog is implausibly fast. Real workloads include compactions, GDPR deletions, and non-trivial overwrites that would only widen the gap. The protocol- as specified- limits single-table commit rates to low single digits per second.

In the next post, we use the CAS latency distributions measured across S3, S3 Express, Azure, and GCS to model multi-table catalogs stored in object storage. Distributing transactions across tables moves the bottleneck from per-table metadata I/O to catalog contention- a bottleneck that conditional operations can address.

Correction 2026-06-09

The numbers in this post were regenerated on 2026-04-17 after finding some simulator bugs, listed below. The bugs mostly cancel each other out leaving the qualitative conclusions unchanged, but the values are slightly more pessimisitic than in the original post. I intended to publish the updated numbers with the next entry in this series, but it was taking too long.

• Per-attempt I/O cost corrected. Each commit attempt now issues five S3 round-trips (TM_read + ML_read + ML_write + TM_write + CAS) instead of three (MF_write + ML_read + ML_write). The old model both double-counted a manifest-file write the protocol doesn’t require and skipped the table-metadata read/write pair entirely. Net effect: the per-table ceiling drops from ~7.7 c/s to 5.7 c/s at S3 medians.
• Timing leaks removed. The CAS version check and catalog.read() now both split-yield at half-RTT. Previously, the simulator allowed the client to read the catalog state without waiting for the message delay.
• VO convoy double-count removed. The IO convoy was charging the per-attempt manifest-list read twice and counting one extra historical manifest list per attempt. With both fixed, single-table VO P99 drops ~7% (203s vs 219s).
• table_metadata_inlined flag corrected. A config-drift bug let intermediate re-runs silently use inlined metadata (future work). This post now uses non-inlined metadata, matching the original intent.
• I/O requests are serial. The intended “up to 4 I/O operations running in parallel” was inaccurate; the implementation ignored this parameter and issued I/O requests serially.

Overall: throughput is slightly lower, P50/P99 latencies are slightly higher. No curve changes shape and no comparison reverses.

“3–4 commits/sec” in the original post has been updated to “2–3 commits/sec”. The per-attempt cost changed from “~300 ms retry” to “five S3 round-trips per attempt, ~175 ms at S3 median latencies”.

Since S3 introduced strong(er) consistency in 2020 and conditional writes in late 2024, table formats like Apache Iceberg can assume stronger guarantees from object storage. Stricter consistency guarantees were already supported by Azure and GCS¹. How do conditional operations perform under load?

We benchmark the atomic/conditional primitives that object stores provide across AWS S3 and S3 Express One Zone, Azure Blob (Standard and Premium), and Google Cloud Storage.

*S3 Express One Zone (S3X) lazily composes appends on read; see details below.

See the graphs for detailed throughput and latency analysis.

Conditional operations are useful building blocks for coordination through storage, but object stores are not designed to be consensus systems². Storage-only coordination is limited to low throughput workloads without an intermediary service to batch and order operations.

Conditions for Coordination in Storage

I just dropped in to see what condition my condition was in

The ostensible Future of Data Systems is a converged data lake. Vendors build toward this vision in path-dependent ways, but table formats like Apache Iceberg are emerging as a common substrate for interoperability with limited (or no) coordination between systems outside of storage.

In the 2010s, we pretended that eventual consistency was usable because it was cheap. Iceberg made the elegant partition of its strongly-consistent Catalog state and the eventually-consistent object store where data and metadata files are stored. Transactions following a pointer from the catalog accessed a fixed, immutable set of objects; the only eventual state to reconcile was an object’s transition from absent to present. From an application perspective, this is like transforming a consistency problem- reasoning about which updates have been applied- to an availability problem- parts of a table snapshot might be temporarily missing.

Modern object stores provide stronger consistency guarantees and atomic operations over objects. So… is any of this useful to the design of table formats? Let’s measure the practical limits of conditional/atomic operations in object stores. They are the building block for linearizable swaps and commit protocols in an open, storage‑only catalog³.

Conditional Operations

Skip to measured results if you don’t want to read a primer on conditional operations.

In a nutshell, clients updating an object provide a token from data they read earlier to the object store, which rejects that write if the token doesn’t match the current state. In a table format, the application coordinating the transaction is saying, “I checked the table in state x. Only apply my update if the table is still in state x; otherwise leave the state of the table unchanged so I can retry, fail, or wander off forever.”

“Wandering off forever” is particularly important for table formats, since concurrent transaction processors could have wildly different resource constraints. The bloated corpse of a large, failed transaction could prevent writers with fewer resources from making progress if they first need to clean up the failed transaction.

Consider two typical Iceberg clients: Florence: a Spark job enriching tens of terabytes of table data using hundreds of nodes and Bob: inserting kilobytes of data from a streaming source. If Florence wanders off, nobody thought to provision Bob with the resources to undo/redo even a fraction of Florence’s work. Installing updates with a single, conditional operation avoids any dependencies between uncommitted transactions that might entangle their resources. It’s a key idea in Iceberg’s design⁴: all work preparing a transaction is completed outside the table before installing it at commit.

Three widely supported conditional operations could enable linearizable updates from uncoordinated writers to a single object.

If-absent

This is probably the most widely available conditional operation. Basically, “write this object only if it does not already exist”. With a naming convention for successors, one can order updates by probing for the latest version and writing the next version in the sequence. Obvious example: if the transaction depends on objname.x then attempt to write objname.(x+1). Some implementations store a recent-ish version that clients blindly overwrite in race after a successful commit, to provide a hint for the “tail” of the log.⁵

I didn’t evaluate this operation as its performance under the synthetic benchmark would be unrealistically pessimistic. Without retries or backoff, each writer would require multiple round trips to discover the current version before attempting its write. The benchmark would tell us less about the capabilities of the object store than the efficiency of the tail-locating protocol.

If-match

Every object is associated with a nonce that uniquely identifies the object at that location. The implementation of a compare-and-set (CAS) is straightforward: keep the nonce- an entity tag (etag; S3 or Azure) or generation (GCS)- when the object is read and provide it to ensure the correct version of that object is overwritten⁶. The write checksum must also protect against partial writes (e.g., closing the stream while unwinding the stack could replace the old version with a partially-written object).

The read-modify-overwrite loop makes this approach impractical as the object gets larger; the read costs get too high. Moreover, not all stores guarantee that concurrent readers can even complete the object read if it is interrupted by an overwrite. Readers should be protected from torn reads (i.e., reading a prefix of one version of the object and the suffix of its successor), but I’ve heard of widely used storage systems that failed to provide this basic protection.

Atomic Append

Object stores usually warehouse immutable objects, but Azure Blob Storage and S3 Express One Zone (S3X) support appending data to an existing object⁷. Both implement a conditional, position-based append: the client reads the current length of the object and attempts to write its data to that offset. As long as it’s the same base object, an append that fails the position check can be retried with the new length, avoiding the costs of reading and merging required by a compare-and-set.

Contrast this with POSIX O_APPEND or GFS/KFS/QFS record append. When the storage service owns the ordering of atomic appends, clients don’t need to know where the tail is when they append to an object. The service will fill holes if clients are slow or fail (avoiding partial writes) and assemble a contiguous object from the ragged tail of appends. Neither Azure nor S3X implement record append.

Instead, both Azure and S3X supply metadata for the clients to order their own operations. Conditional appends must match both the position and etag for the base object and include a checksum to ensure the full write succeeds. Limits on the size and number of appends per object apply: S3X allows up to 10,000 appends up to 5GiB each; Azure allows 50,000 appends up to 4MiB each.

Delegating append ordering to the client simplifies everything except the writer. Server side: replication and concurrent reads at the tail are much simpler if writes that are contiguous in the object arrive ordered. Slow or failed appends can’t be visible to external clients, so in an implementation of record-append: replicas need to handle cases where they disagree on the presence or content of a particular record. Not only in recovery, but in forward operation. Client side: readers also need to handle duplicate records. If a write times out, the writer can’t be certain if its append was applied or not. If the client retries, two duplicate records could be applied at distant offsets.

Acknowledging the implementation complexity, the storage service can reorder writes more efficiently than independent clients. As we’ll see in benchmarks, position append avoids the read and merge costs of CAS, but oblivious coordination across writers will limit its throughput.

Hereafter, I’ll refer to S3 Express One Zone as S3X and Azure with Premium storage as AzureX for brevity.

Performance

CAS (Compare-and-Set)

Google Cloud Storage

Kudos to the team responsible for throttling single-object requests. The single write per second limit is documented. Even the single-client CAS latency is over 700ms, which is quite high. I wrote the GCS team to verify that I was using the SDK correctly and had not overlooked a storage configuration suited to this workload, but did not receive a response.

Given the documentation, it seems likely that this is performing as designed. I did not modify the benchmark to use composite object support to emulate append, as I assume the same limit applies.

A guess: versioning support probably retains multiple versions of the data, and without throttling the system could be overwhelmed. It would be worth testing if larger objects can be read without interruption from concurrent writes e.g., a workload of readers starting 100ms apart, interrupted by a write every 2s. CAS operations in Azure sometimes interrupted readers.

Azure Blob Storage

Goodput remains steady as the rate of failed operations increases with parallelism. Premium supports ~50% more goodput than standard storage.

A pipeline of conditional requests is mostly waste: a successful write invalidates all outstanding requests, so only one can succeed per round-trip. This fundamentally restricts conditional write throughput.

The increasing latency of successful operations in Azure is not innate to conditional writes. The benchmark does not retry failed requests; the increase in latency must be attributed to the store, as every client runs in its own process.

Azure Blob Storage had some odd behaviors worth noting, against the advice of my generous reviewers whose taste is otherwise impeccable.

Azure CAS anomalies (click to expand)

Anomalies

First, the length reported after a CAS is sometimes _zero_. This happens intermittently; subsequent reads return the correct length. It's unclear if this is a consistency issue, client bug, or a bug in my implementation, but this (invalid) state should not be visible to clients between CAS operations.

Second, CAS operations sometimes caused outstanding reads to fail, particularly in the append workload. These errors were infrequent, but this microbenchmark is on small data and reads of larger objects might be more susceptible to interruption by concurrent writes.

Third, throughput using SAS tokens (Shared Access Signatures that delegate access without exposing the storage account key) is significantly higher than the default authentication mechanism, as shown below. At high client counts, errors include timeouts from whatever ancillary service is authenticating requests. Creating different SAS tokens per client has no effect on throughput, so the throttling appears to be from the default auth service.

AWS: S3 and S3X

S3 CAS throughput is similar to Premium Azure Blob Storage. S3X CAS throughput is roughly 5x what we see in any other store. Again, most of the pipeline is wasted work (nearly 600 failed op/s!), but few Iceberg warehouses require this kind of throughput, even across tables.

The S3 and S3X latencies have low variance compared to GCS and Azure. Note that the CAS latency in S3 is within 10ms and in S3X the increase in latency is within 12ms, compared to hundreds of milliseconds separating min/max latencies in GCS and Azure. S3X is neither general-purpose storage nor magic- it restricts several S3 features to deliver this performance- but it delivers what it claims on its label.

Append

Next, let’s measure position-based append performance. There’s a long history of log-structured systems delivering high-throughput over append-only logs. Unfortunately, the prenominate constraints on position append make high throughput unrealizable in practice.

Azure

I’ll indulge in some eyeball statistics and call these pretty similar to the CAS throughput graphs above. The same read‑modify‑write cycle applies here. Because we do not retry operations in this benchmark, we’re not exercising the path where failed appends are retried at the new position; including retries would make latency a noisy proxy for retry rates.

For small objects, the virtues of append are not salient over CAS.

Latency of successful append operations is lower than CAS operations, particularly under contention. Only one or two runs on Azure appended enough data to reach the append limit, so we omit the interleaved CAS results.

S3 Express One Zone (S3X)

Append throughput is a striking ~90 ops/s, but CAS latency over 10 seconds? What is going on? Breaking down the latency of the CAS operation reveals that almost all the time is spent on the initial read of the object before the conditional write.

As we gradually add reads into the mix (0.1%, 1%, 10%, 30%), the mean latency of the CAS operation gradually increases as contention increases:

It’s not a coincidence that the append limits in S3X match the multipart upload limits (10,000 parts of at most 5GiB each)⁸. It appears that S3X accumulates parts (not enforcing the 5MiB minimum part size) until the object is read, and then lazily composes them into a single object. This is quite different from Azure’s implementation, which pays this cost eagerly for each append.

Discussion

First, some basic takeaways from the benchmarks. GCS was not designed to support this workload; the use case we’re interested in- linearizable writes on a single, small object- is explicitly throttled below a rate that could support even modest transaction rates. Azure Blob Storage and standard S3 sustain 10-15 conditional writes per second. The append implementations in both Azure and S3X exhibited anomalies in correctness and performance respectively, which complicates the implementation of optimistic protocols directly in these object stores.

Immutable data objects remain the core use case for object stores, even at premium tiers; stronger consistency and conditional operations are exciting, but insufficient to transform object stores into consensus systems.

Position Append is not (really) Multi-writer

Adding a “record append” API to object stores would allow the client to conditionally append not to a particular offset of a particular object, but to the end of a particular object… wherever it is.

Record append is not an easy API to use well, but it can improve throughput substantially⁹ and allow readers to tail the object coherently. Readers need to tolerate duplicates and implement framing for objects larger than the maximum append size (e.g., Azure Blob Storage’s 4MiB limit). To be fair: evidently the state of the art in position append isn’t that easy to use well either, at least for optimistic concurrency control.

To belabor a point: when a successful action invalidates all outstanding work, conditional operations create a lot of waste under load. Azure and S3 sustain steady throughput under contention, but as we see in these microbenchmarks, another service needs to be in front of the object store to batch operations and reduce conflicts to achieve higher throughput.

On the bright side, we can roll our own multi-writer append on top of conditional writes.

Emulating Record Append

Any service that guarantees the durability and ordering of writes can emulate record append on top of an object store supporting conditional writes. As in the (inferred) implementation of S3 Express One Zone append, primitives like multi-part uploads or composite objects in GCS can stitch together multiple writes into a single object. The trick is to get a consistent view for readers and lazily compact to the object store, coordinated using conditional writes.

AWS Object Lambdas can intercept GET, PUT, and LIST requests to S3 objects. If writes are funneled through a FIFO queue service like SQS, an object lambda can read a checkpoint in S3 and apply the queued appends to a particular object. On GET requests, the lambda replays queued writes over the base object from S3.

Asynchronously, the lambda function can conditionally write the compacted object (or append a prefix of the queue) to S3 and remove the corresponding entries from the queue. Other instances running concurrently may return a stale view, but never an inconsistent one. Conditional writes prevent compactions from losing state.

The object lambda adds cool points for being serverless and transparent to read-only transactions, but there’s no magic here. Clients could access the queue directly or any service could stand in front of S3 to batch and order writes.

Implications for Table Formats

So what does this mean for table formats? There are still opportunities to reduce write amplification, particularly using append. But for coordination through storage: any format needs to ensure real conflicts are rejected, making the rates measured here an upper bound on throughput under contention without an intermediary service.

Object stores are not (yet!) consensus systems. Data systems that use conditional writes should relegate them to a coarse, batch primitive for coordination, unless throughput demands are very low.

Thanks and Blame

Thank you Joe Hellerstein, Owen O’Malley, Tiemo Bang, Natacha Crooks, and Micah Murray for feedback on drafts. Ashvin Agrawal, Carlo Curino, Jesus Camacho Rodriguez, and Raghu Ramakrishnan at Microsoft (Gray Systems Lab) introduced me to table formats and helped shape many stages of a (now abandoned) research project on Iceberg. Mehul Shah’s prophecy steered me away from several pitfalls, except when I ignored his advice and fell into them anyway. Pavan Lanka and Sumedh Sakdeo offered useful insights into Iceberg workloads and ORC and Parquet internals, informing much of the work I’ll probably write about later. The misunderstandings and errors I managed on my own.