Apache Iceberg

One Interface, Many Backends: The Design of Iceberg Rust's Universal Storage Layer with OpenDAL

Karthic Rao

Karthic Rao

— 6 min read
One Interface, Many Backends: The Design of Iceberg Rust's Universal Storage Layer with OpenDAL

Apache Iceberg's core promise is to treat files in a data lake on Minio AIStor, S3, GCS, HDFS, or your local disk as if they were rows in a high-performance database table, complete with ACID transactions and schema evolution. It's a powerful abstraction. But how does Iceberg achieve this "data anywhere" capability at the I/O level? What are the design considerations and challenges in building a single, universal storage layer that seamlessly and reliably interacts with a diverse set of underlying systems?

This post dives into the architecture of such a layer, drawing insights from the design patterns observed in the Apache Iceberg Rust reference implementation. We'll explore the fundamental I/O requirements, the complexities introduced by supporting multiple storage backends, the non-negotiable need for robustness, and how a dedicated data access library like OpenDAL is instrumental in crafting a clean and effective abstraction. Let's dissect how this I/O engine is built.

Core I/O Primitives: The Foundation of Iceberg Operations

Before Iceberg can perform any of its table management magic, its storage layer must provide a set of foundational I/O primitives. These are the basics needed to interact with the files constituting an Iceberg table—metadata (JSON, Avro) and data (Parquet, ORC, etc.).

The universal storage layer must, at a minimum, offer:

  1. Reliable File Operations for Metadata: Iceberg commits often involve writing a new table metadata file and then updating a central pointer. While the atomicity of this pointer swap is typically handled by the Iceberg Catalog or commit logic, the storage layer must reliably create these new metadata files (often by writing to unique, new locations) and reliably delete old ones if required. It provides the building blocks for higher-level atomic operations.
  2. Path Resolution and Manipulation: Consistent interpretation of URIs (e.g., s3://bucket/key, file:///path) and the ability to derive new paths is essential.
  3. Basic File Operations: Standard create, read, write, and delete capabilities for both individual files and, where applicable, directories or prefixes.
  4. Existence Checks: The ability to quickly verify if a file or path exists.

These primitives form the contract that Iceberg's higher-level logic relies on. But satisfying this contract across potentially dozens of different storage APIs is where the real challenge begins. Why bother with this complexity?

The Multi-Backend Reality: Why One Size Doesn't Fit All

Iceberg's design philosophy is to integrate with existing data ecosystems, not forcing migrations to a specific storage type. Data teams use a variety of storage for good reasons:

  • Cloud Object Stores (S3, GCS, Azure Blob): For scalability, durability, and cost-efficiency.
  • Distributed File Systems (HDFS): Common in established on-premises Hadoop clusters.
  • Local File Systems: Essential for development, testing, and smaller deployments.

Therefore, a universal storage layer must be a chameleon, adapting its behavior to each specific backend. The storage.rs code from the Iceberg Rust implementation hints at this with its Storage enum, which acts as a dispatcher based on the storage scheme.

// storage.rs
#[derive(Debug)]
pub(crate) enum Storage {
    #[cfg(feature = "storage-memory")]
    Memory(Operator), // OpenDAL Operator
    #[cfg(feature = "storage-fs")]
    LocalFs,
    #[cfg(feature = "storage-s3")]
    S3 { /* ... */ },
    // ... other backends via conditional compilation
}

This adaptability is powerful, but each backend has its own API, authentication model, and, critically, failure modes. How does a universal layer provide consistent reliability in such a heterogeneous and often network-dependent environment?

Robustness in a Distributed World: Handling Transient Failures

When your storage can be halfway across the world, accessed over a network, transient failures are not exceptional events—they are an operational reality. A network glitch, a temporary blip in a cloud service, or a rate-limiting response can all interrupt I/O. A storage layer for Iceberg must be resilient to these.

The primary mechanism for this is automated retries with backoff. An operation failing due to a detectable transient error shouldn't immediately propagate failure. Instead, it should be retried a configured number of times, ideally with an increasing delay (backoff) and some randomization (jitter) to avoid thundering herd problems.

In storage.rs (from the Iceberg Rust project), this is explicitly handled by applying an opendal::layers::RetryLayer to the Operator instances:

// storage.rs
// ...
    pub(crate) fn create_operator<'a>( /* ... */ ) -> crate::Result<(Operator, &'a str)> {
        // ... operator setup for different schemes ...
        let operator = operator.layer(RetryLayer::new()); // Ensures retries
        Ok((operator, relative_path))
    }
// ...

This neatly applies a consistent retry policy. But retries only solve one part of the robustness puzzle. The more significant architectural challenge is writing code that interacts with S3, HDFS, and a local filesystem using the same internal logic, without that logic becoming a labyrinth of backend-specific conditional branches.

The Abstraction Blueprint: A Uniform Contract for Diverse Backends

A strong abstraction layer is essential to avoid backend-specific logic from polluting the core Iceberg code. The goal is a uniform interface that all Iceberg components use for storage access, regardless of the backend.

This abstraction is primarily realized in the Iceberg Rust code through:

  • FileIOBuilder and FileIO (in file_io.rs): FileIO is the main entry point for Iceberg's interaction with storage. It's configured via FileIOBuilder, which takes a scheme (e.g., "s3", "file") and backend-specific properties.
  • Storage enum (in storage.rs): FileIO internally holds an instance of the Storage enum. This enum encapsulates the backend-specific configuration and the logic to create an opendal::Operator tailored for that backend.

The FileIO struct exposes methods like new_input(path) and new_output(path). These methods don't know if the path is an S3 URI or a local file path; they delegate to the Storage instance, which then uses the appropriate OpenDAL configurations to service the request.

// file_io.rs
pub struct FileIO {
    builder: FileIOBuilder,
    inner: Arc<Storage>, // The specific backend implementation
}

impl FileIO {
    pub fn new_input(&self, path: impl AsRef<str>) -> Result<InputFile> {
        let (op, relative_path) = self.inner.create_operator(&path)?; // Delegate to Storage
        // ... construct InputFile with the operator
    }
    // ...
}

This design provides the necessary separation. However, implementing and maintaining robust clients for S3, GCS, Azure, HDFS, etc., from scratch for the Storage enum variants would be a monumental engineering effort. Surely, there's a more leveraged approach?

OpenDAL: The Universal Connector Toolkit

This is precisely where OpenDAL (Open Data Access Layer) comes in. OpenDAL is a Rust library focused on providing a unified API to access many storage services without needing to handle the idiosyncrasies of each service's native SDK.

OpenDAL offers:

  • Pre-built Services: Connectors for S3, GCS, Azure Blob, HDFS, local FS, and many others.
  • A Consistent API: A standard set of operations (read, write, stat, delete, list) that behave consistently.
  • Composable Layers: Middleware for cross-cutting concerns like retries, logging, and caching.
  • Efficiency: Designed to be a zero-cost abstraction where possible.

In the storage.rs implementation within Iceberg Rust, each variant of the Storage enum that represents a remote or complex filesystem would internally use OpenDAL to configure and build an opendal::Operator. For example, when Storage::S3 needs to create an operator, it translates its S3Config into OpenDAL's S3 service builder, configures it (bucket, endpoint, credentials), and then calls .build() on the OpenDAL builder.

This means Iceberg's storage layer doesn't need to implement the intricate details of S3's multipart upload or GCS's authentication. It delegates that to OpenDAL, drastically reducing the amount of bespoke connector code. With OpenDAL providing the "physical" connection to diverse storage, how does Iceberg layer its logical file interaction model on top?

Layering Iceberg's File Access on the Universal Abstraction

With OpenDAL handling the backend specifics and Storage abstracting these into a common Operator, the FileIO struct provides the final, clean interface used throughout the Iceberg Rust codebase.

  • Table Operations: An Iceberg Table (from table.rs in the Rust project) is initialized with a FileIO instance. This FileIO object performs all its operations that require file access—reading metadata files, accessing manifest lists, or preparing to read data files for a scan.
// table.rs
#[derive(Debug, Clone)]
pub struct Table {
    file_io: FileIO, // Used for all file system interactions
    // ...
}
  • InputFile and OutputFile: When FileIO provides an InputFile or OutputFile, these structs encapsulate the opendal::Operator and the specific path. They then expose simple read() or write() methods, which internally use the operator's capabilities.
// file_io.rs
pub struct InputFile {
    op: Operator, // The OpenDAL operator for this file
    path: String,
    // ...
}
impl InputFile {
    pub async fn read(&self) -> crate::Result<Bytes> {
        // Uses op.read()
    }
}
  • Metadata Caching (ObjectCache): The ObjectCache (in object_cache.rs) fetches serialized manifest lists or manifests using FileIO before deserializing and caching them. This ensures even caching logic benefits from the universal and robust storage access.
  • Transactional Commits (Transaction): When a Transaction (in transaction/mod.rs) commits, it often involves writing new metadata or manifest files. These write operations are channeled through the table's FileIO instance, ensuring they land correctly on the configured storage backend.

This layered approach ensures that the core Iceberg logic for managing table state, planning queries, and executing DML is completely decoupled from the physical storage location and type.

Takeaway: A Blueprint for Universal Storage Access in Iceberg Rust

Building a universal storage layer for a system like Apache Iceberg, as seen in its Rust implementation, is a significant architectural undertaking. The key to its success lies in:

  1. Strict Abstraction: Defining a clear, backend-agnostic interface (FileIO) that the rest of the system programs against.
  2. Delegation of Specifics: Using a dedicated dispatcher (Storage enum) to handle backend-specific configurations and initialization.
  3. Leveraging a Universal Access Library: Employing a library like OpenDAL to manage the complexities of individual storage service APIs, authentication, and core I/O operations, including built-in robustness features like retries.
  4. Consistent Layering: Ensuring that all parts of the system, from table operations to caching, utilize the defined storage abstraction.

This approach provides Iceberg with its crucial "data anywhere" flexibility, allowing it to operate robustly across a diverse and evolving landscape of storage systems without compromising the clarity or maintainability of its core logic. It clearly demonstrates how thoughtful software architecture, particularly in a systems language like Rust, can effectively manage inherent complexities.

Read more