Iceberg in Production: What Three Industries Taught us

The market has converged. The question has changed.

Two years ago, choosing a table format was a bet. Today it isn’t. Databricks – the company behind Delta Lake – now supports Iceberg as a first-class format, with Iceberg v3 generally available on its runtime. Snowflake reads and writes managed Iceberg tables and has built its interoperability layer on the Apache Polaris catalog standard. AWS went further and made Iceberg a storage primitive: Amazon S3 Tables provide fully managed Iceberg tables with built-in maintenance. Google and Microsoft support it. The open-source engines – Spark, Flink, Trino – treat it as the default.

When every major vendor converges on the same open format, the strategic risk inverts. The question is no longer “should we adopt Iceberg?” It is “why are we still paying to keep our data in a format only one vendor can read?”.

That convergence has moved the hard questions down a level, from selection to implementation.

• Which catalog owns the tables?
• Which engine runs which workload?
• Who maintains the tables at 2am?
• How do you get business value out in weeks rather than quarters?

Those are the questions our customers actually face – and they are implementation questions. Here is what answering them in production taught us.

Lesson one: one copy of data, every engine

The interoperability benefit of Iceberg is usually described in the future tense – optionality for some later migration. In practice, it pays off in the first month.

For the payments provider, transaction and gaming data arrives daily from multiple point-of-sale and gaming systems across dozens of venues. That data lands once, in Iceberg tables on Amazon S3. From there, Spark on AWS Glue runs the heavy transformation. Athena serves ad-hoc investigation when an analyst needs to trace a single receipt. Redshift serves the governed semantic layer that powers executive dashboards. Three engines, three very different workloads, zero duplicate copies of the data.

What that removed is worth spelling out, because it is where the money is. No overnight sync jobs between a lake and a warehouse. No reconciliation meetings about why two systems disagree. No storage bill multiplied by the number of engines. When the numbers in the dashboard are questioned, everyone is looking at the same table – the debate is about the business, not the plumbing.

The same pattern held at industrial scale. For the coal producer, billions of sensor readings from thousands of plant instrumentation tags – arriving at 30-second cadence – are written once and consumed by streaming analytics, batch reporting, and machine learning models that predict plant performance. The workloads could not be more different; the data platform underneath them hosts the same Iceberg tables.

For a business audience, the takeaway is simple: Iceberg turns “which analytics engine should we buy?” from a five-year commitment into a workload-by-workload decision. That is a procurement advantage, not just an architectural one.

Lesson two: managed tables changed the economics

For executives, architects, and data leaders, the Iceberg discussion should not start with “Do we need Iceberg?”

It should start with a sharper set of questions:

• Is our data ecosystem trustworthy enough to support critical workloads and an Agentic future?
• Can we change schemas and partition strategies without expensive migrations?
• Can multiple engines safely access the same data without duplicating it?
• Can we reproduce historical results when numbers are challenged?
• Can we reduce dependency on proprietary compute while preserving performance and governance?
• Do we have the operational maturity to maintain open table formats at scale?

If the answer to the majority of these questions is no, Iceberg on AWS is not just a technical upgrade. It is a strategic enabler for your organisation.

Lesson three: the boring features carried the day

Iceberg demos love time travel. What actually protected our customers, day after day, were the two least glamorous capabilities in the specification: schema evolution and transactional writes.

Source systems change without asking permission. Point-of-sale vendors add columns and rename fields between versions. Plant systems get reconfigured. On a traditional lake, each of those events is a small emergency – a broken pipeline discovered at 6am, a dashboard silently gone wrong. On Iceberg, a schema change is a metadata operation. Paired with code-based data contracts that make expectations explicit and versioned, schema drift became a managed event rather than an outage. Across three engagements ingesting from more than a dozen distinct source systems, this was the single biggest reducer of operational noise.

Auditability mattered more than expected, because all three customers operate under scrutiny. Gaming venues answer to state regulators. Mining operations answer to production reporting and safety obligations. Land and tenement compliance is, by definition, an evidence business. Iceberg’s point-in-time snapshots mean “what did the data say on the 14th?” is a query, not an archaeology project.

The land-compliance engagement pushed this furthest, and it reframed how we think about Iceberg. There, Iceberg tables are not just analytics storage – they are the change ledger. Authoritative government source data is harvested on a schedule, stored as point-in-time snapshots, and compared as record-level diffs, with the operational application consuming only what it needs. The table format itself provides the evidence trail. That is not a data lake pattern; that is Iceberg enabling a product capability.

What we did differently

Cross-industry lessons are only half the story. The other half is the approach – what Codex did that unlocked the outcomes, and that we now reapply on every engagement.

We started from the outcome, not the platform. On each engagement, the first artefact was not an architecture diagram. It was the measures: the exact reports, KPIs, and decisions the business runs on – venue performance measures for the payments provider, plant recovery and throughput KPIs for the miner, compliance workflows for the software company. We then worked backwards to the tables. It sounds obvious; it is the opposite of how most platform programmes run, and it is why these platforms produced business-visible results in their first release rather than after a year of foundation work.

We productised the pattern. A medallion architecture on Iceberg, code-based data contracts, infrastructure as code, automated quality gates, and orchestration with self-healing built in – codified once, then reapplied. The consequence is speed: capability that took months to establish on the first engagement was stood up in weeks on the next. When a new data source or a new venue comes onboard, it lands on a proven pattern rather than a blank page.

We treated governance as a day-one feature, not a retrofit. Row-level security so that each venue sees only its own data. Least-privilege access wired into the infrastructure code. Data quality alerts on the pipelines from the first release. This is the connective tissue back to our first article: trustworthy, governed, open data is the precondition for the agentic future – and it is far cheaper to build in than to bolt on.

Where this goes

Iceberg on AWS is no longer an early-adopter decision. It is the pragmatic default for organisations that want their data estate open, governed, and ready for AI – and the implementation playbook now exists.

If you are an AWS team with a customer weighing a lakehouse modernisation – or a data leader wondering whether these lessons transfer to your industry – the honest answer from our experience is that they already have: payments, resources, and software are about as different as industries get. We are happy to share the detail behind any of them.