Iceberg Tables: The AI-Chemist for an Agentic Future

The problem Iceberg solves is bigger than performance

 

Many data platform discussions begin with speed: faster queries, better pruning, fewer files, lower cost. Iceberg helps with all of that. But its deeper value is not simply that queries can run faster. The deeper problem is trust.

Traditional data lakes were built on folders and files. A table was often just a directory full of Parquet, ORC, or Avro files, with partition folders acting as a loose convention. That worked until organisations needed to support concurrent writes, schema changes, streaming ingestion, backfills, deletes, auditability, reproducibility, and multiple query engines.

At that point, the cracks became visible. A pipeline could partially write files and leave the table in an inconsistent state. A schema change could break downstream consumers. A partition strategy that worked at 10 TB could collapse at 500 TB. A business user could rerun yesterday’s dashboard and get different results without knowing why. A data engineering team could spend more time repairing tables than creating value.

Iceberg addresses this by shifting the definition of a table away from “a folder of files” and toward “a governed, versioned, metadata-driven object.”

In Iceberg, table state is maintained through metadata files. Changes create a new metadata file and replace the old metadata through an atomic swap. The table metadata tracks the schema, partition configuration, table properties, and snapshots; each snapshot represents the complete state of the table at a point in time.

That sounds technical, but the business implication is simple: the table becomes a reliable contract.

 

How Iceberg is different from older lake patterns?

 

Iceberg’s appeal comes from the fact that it solves several long-standing data lake issues at the table level.

Open Architecture:

Data platforms are inherently complex to manage. Organisations have traditionally built them over time with significant investments through multiple internal and partner resources. These are often poorly documented, require management of data from a number of different technologies and formats, undergo non-standard data updates/patching over time to simulate real-world scenarios, and depend on input from several SMEs to build and maintain cross-system relationships and metric definitions. When the technology landscape evolves, the entire stack needs to be migrated, resulting in significant reinvestment and complexity. If the data storage on the platform is vendor-specific, chances are the compute layer is also tightly coupled and usually dictated by the vendor. This has traditionally posed a big barrier for organisations to exit from existing legacy data ecosystems. Whilst modern catalogs aim to solve for the cross-technology data access constraint, it is the de facto lock-in to compute which poses a greater challenge.

Further, vendor lock-in can pose limitations on agents to scale and be cost-effective, as organisations can be constrained using or building agents on a particular technology. The competitive advantage of a truly agentic ecosystem will ultimately distil down to cost and flexibility, and organisations will need to make their architecture as open as possible to manage the uncertainties of the future.

Iceberg tables, governed by the Apache Software Foundation, are a true open table format. They can be hosted on any hyperscaler (e.g. AWS), and are compatible and interoperable across a range of compute engines like Spark, Flink, Trino, Athena, Redshift, EMR, Firehose, Snowflake, Databricks etc. These compute engines can seamlessly operate on Iceberg tables without any duplication of data.

ACID Compliance:

Advancements in data platforms have enabled horizontal and vertical scaling of data assets, making large amounts of enterprise datasets ready for analytics at speed. The same datasets can be updated through multiple data pipelines at different temporal intervals, for instance, a customer data asset can be updated by CRM and social media feeds simultaneously.

This places more focus on the integrity of data, especially as trust in data is critical for your agents. Further, these agents will amplify the above state and enable reads and writes at a frequency and scale not seen before in our industry. If multiple data processes read and write from the data platform, it needs to ensure that there is isolation for these processes to prevent data corruption and enable consistent and reliable output. Data warehouses were largely ACID compliant, but data lakes dropped the ball, leading to severe data issues.

Iceberg tables act just like a standard database, supporting transaction isolations for data lakehouses. The writes support the replacement of data files in a single operation/transaction, so partial operations are never visible. Concurrency locking ensures any parallel writes do not lead to inconsistent data, and snapshots ensure that reads and concurrent writes are isolated. Iceberg ensures durability, implying the transaction commits are persistent.

Schema Evolution:

Downstream impacts in data platforms due to upstream changes in source data structures, aka schema drift, have been a change management challenge for a long time. This was traditionally handled through multi-party data contract documents, which stipulated the structure and frequency of expected datasets. So when an attribute needed to be added, removed or updated within a dataset, the contract document served as the binding agreement. This worked as a stopgap solution 20 years ago when source system release cycles used to be measured in months, if not years, but through agile practices, the source system development cycles started shrinking, making management of this process not scalable. Further, data lakes encouraged the collection of most internal and relevant external datasets, broadening the impact of this issue to a larger number of datasets.

Advancements in AI will turbo-charge the evolution of these source (and downstream) systems, requiring rapid management of changes to data. This means the source systems like CRMs and ERPs will update rapidly, resulting in data formats and API definitions changing more frequently. This will have downstream impacts on data processes as attributes get added, updated and deleted from the data ingestion feeds at pace. This has always been a reactive problem in the data world but will be amplified by AI percolating all aspects of technology ecosystems.

Iceberg tables support schema evolution by decoupling schema definitions from the physical data files. This means the changes to the structure of data are managed using metadata definition, preventing the rewrite or reshaping of physical data files. The schema changes are tracked through versioning of metadata, reducing significant operational overheads and expensive data management effort. This makes these formats resilient to frequent changes in data structures, with minimal intervention required by users to manage the process. It should be noted that modern, code-based data contracts should still be used and versioned automatically to manage these changes.

Time Travel:

What did our data look like on a particular day? This question resonates with many users who grapple with investigating why the numbers look different now, compared to a previous point in time. Dimensional data modelling helped solve this problem to a certain extent but was still constrained by real-world constraints in data modelling and representation. Any attempts at finding the exact state of data at a point in time meant restoring expensive database backups, which required significant investment of effort and time.

Explainability, relating to structure and content of data will become increasingly crucial as more and more data workloads start getting managed autonomously by agents. There will be increased scrutiny from a governance, audit and regulatory perspective, which will add to the need to understand the exact state of data at a historical point in time.

Iceberg tables allow access to historical snapshots of the table, providing capability to reproduce the exact state of tables instantly without restoring any special backups or saving separate physical snapshots. This allows users to execute time travel queries to simulate the state of the data platform exactly at a point in time. This also enables users to revert the state of their table to an older snapshot to fix and reprocess data-related issues, and obviously help in audit and compliance use cases.

Hidden Partitions & Partition Evolution:

Query performance has often been one of the most important performance benchmarks in the data analytics world. Data partitioning allows data tables to be divided into smaller tables based on a partitioning strategy. These partitions help improve read performance as queries can skip over partitions which do not satisfy the filter criteria; data can be processed independently in parallel from these partitions, making it easier to distribute data in engines like Spark. Previously, partitioning required explicit definition of partitioning attributes and queries requiring partition columns to benefit from partitioning strategies. Further, changing the partitioning strategy resulted in issues in existing queries and data processes.

Iceberg tables support hidden partitioning, which allows Iceberg to handle creation of partition values, removing the need to explicitly specify these attributes. Iceberg hides partition details from users while still using partition information for planning and pruning. Further, a table’s partition strategy can change as data volume, access patterns, or business needs change. Iceberg supports in-place table evolution, including changing partition layout without rewriting all table data or migrating to a new table.

 

But Iceberg is not magic

 

Thought leadership around Iceberg sometimes slips into hype. It is worth being clear: adopting Iceberg does not automatically create a high-performing data platform.

Iceberg introduces powerful metadata structures, but those structures need to be maintained. Tables still require compaction, snapshot expiration, metadata cleanup, manifest management, file-size discipline, and thoughtful catalog design. Poor ingestion patterns can still create small-file problems. Weak governance can still create inconsistent semantics. Bad modelling can still produce bad analytics.

The difference is that Iceberg gives teams the right primitives to manage these problems properly. It makes table health observable. It makes evolution manageable. It makes transactions possible. It makes multi-engine access safer. But organisations still need platform engineering discipline. AWS S3 tables help manage a lot of this complexity.

The winning teams will not be those that simply “move to Iceberg.” They will be those that treat Iceberg as part of a broader operating model: clear data product ownership, strong catalog practices, automated maintenance, governance by design, and workload-aware optimisation.

Why is AWS the best cloud for Iceberg tables?

 

AWS is one of the best clouds for Iceberg tables because it treats them as a first-class storage primitive, rather than another table format. AWS offers the best cost-effective data management and interoperability to support agentic workloads.

Cost-effective, managed Iceberg Storage at Scale:

Amazon S3 provides built-in Iceberg support through fully managed S3 tables, coupled with proven durability, availability, security, and compliance at scale. S3 tables offer much faster writes (up to 10x) and query responses compared to self-managed tables, making them suitable for large analytical workloads. Intelligent tiering, which moves data to cost-effective storage classes based on access patterns, can reduce storage costs by up to 80%. S3 Tables also automatically manage compaction, snapshot cleanups and file maintenance for Iceberg formats, reducing overheads for data teams across thousands of datasets. AWS also provides one of the best price-performance for hosting Iceberg tables, leading to lower TCO for your data estate. A cost-effective way to store and manage your data is going to be critical as AI agents start accessing broader organisational data assets.

Compute Interoperability:

Hosting Iceberg on AWS avoids vendor lock-in and provides interoperability across both AWS-native services like Amazon S3, Glue, Athena, Redshift, SageMaker etc., as well as Iceberg compatible services like Spark, Snowflake, Databricks, Trino etc. Your agentic and analytical workloads can be optimised across different compute engines based on the use case and still work on the same ACID-compliant datasets on Iceberg to ensure cost-effective management of resources. This becomes especially important as your organisation uplifts its AI maturity and preserves investments made in current tooling. Amazon Quick offers native integration with S3 tables, which allows any data to be available instantly for agentic consumption, which can be used in intelligent apps, self-service analytics and many other use cases.

Catalog Integration:

Hosting Iceberg on AWS provides further interoperability in the open lakehouse architecture through catalog integration and compatibility. For instance, AWS Glue Data Catalog can integrate with Databricks’ Unity and Snowflake’s Horizon Catalogs through Iceberg’s REST APIs. This will allow the AI agents to discover and access data assets across your organisational ecosystem, breaking one of the biggest barriers in data management.

 

What leaders should ask before adopting Iceberg

 

For executives, architects, and data leaders, the Iceberg discussion should not start with “Do we need Iceberg?”

It should start with a sharper set of questions:

• Is our data ecosystem trustworthy enough to support critical workloads and an Agentic future?
• Can we change schemas and partition strategies without expensive migrations?
• Can multiple engines safely access the same data without duplicating it?
• Can we reproduce historical results when numbers are challenged?
• Can we reduce dependency on proprietary compute while preserving performance and governance?
• Do we have the operational maturity to maintain open table formats at scale?

If the answer to the majority of these questions is no, Iceberg on AWS is not just a technical upgrade. It is a strategic enabler for your organisation.

Food for thought

The most underrated benefit of Iceberg is optionality. An organisation that standardises on Iceberg is not merely choosing a table format. It is preserving future choices by avoiding vendor lock-in. That matters because data platform requirements rarely stay still.

Today’s priority may be BI cost reduction. Tomorrow’s may be near-real-time personalisation using machine learning. Next year’s may be AI feature generation or regulatory auditability. A platform tied too tightly to one compute engine or proprietary storage abstraction can become a constraint just when the business needs flexibility.

Iceberg does not eliminate platform decisions. It makes them less irreversible.

The name “Iceberg” is fitting. The visible part is a table: rows, columns, SQL, dashboards, models. But the real mass is underneath: metadata, snapshots, manifests, schemas, partitions, transaction history, and evolution.

For years, data leaders focused on what was visible above the surface: faster dashboards, cheaper storage, more pipelines. The next era of data platforms will be won below the surface.

That is where trust lives.

That is where optionality is created.

And increasingly, that is where Iceberg is becoming the standard.

Is your organisation ready?