Artificial intelligence news.
Straight from the original announcements.

Fivetran explains how Apache Polaris catalogs managed Iceberg tables

November 18, 2026, 05:00By the AI News Bites Editorial TeamSource: Fivetran
Fivetran explains how Apache Polaris catalogs managed Iceberg tables
Image: Fivetran

Fivetran has published guidance for using Apache Polaris as the catalog behind its Managed Data Lake Service. The service uses Polaris to provide a REST-based interface for Iceberg tables across several cloud storage destinations.

Key points
  • Apache Polaris is the default catalog for Fivetran Iceberg tables.
  • Fivetran supports Amazon S3, ADLS Gen2, GCS, and Microsoft OneLake destinations.
  • Users should query catalog tables instead of directly reading Parquet files.
  • Setup requires a Polaris endpoint, catalog name, client ID, and client secret.
  • Configuration guidance covers Spark, DuckDB, Snowflake, and Trino and Starburst.

A REST catalog for Iceberg data

Apache Polaris is an open-source catalog for Apache Iceberg tables that implements the Iceberg REST API specification. Fivetran describes it as a metadata catalog service that tracks the current metadata pointer and the locations of metadata files, allowing query engines to find and access tables through a standardized REST interface.

Fivetran's Managed Data Lake Service writes data as Parquet files while maintaining Iceberg table metadata. It supports Amazon S3, Azure Data Lake Storage (ADLS Gen2), Google Cloud Storage (GCS), and Microsoft OneLake as destinations. The Fivetran Iceberg REST Catalog, built on Apache Polaris, is the default catalog for all Iceberg tables.

Each data lake destination has its own dedicated catalog instance, which is automatically configured using the destination's details. Fivetran says only it can write or update catalog metadata, an approach intended to preserve data integrity and prevent conflicts caused by external modifications.

Why engineers should query the catalog

Fivetran's central recommendation is to query Iceberg tables through the catalog rather than directly querying the underlying Parquet files. When raw files are queried, engines must inspect file metadata and perform list and open operations across potentially thousands of files. Iceberg's metadata index, made up of a manifest list, manifests, and data files, supports scan planning through remote calls that do not depend on table size in the same way.

Direct Parquet queries also lack the transaction guarantees provided by Iceberg. According to Fivetran, raw files can expose partial writes, allow concurrent writes to conflict, and provide no atomic commit across multiple files. Iceberg instead uses committed snapshots, atomic file additions, and optimistic concurrency with automatic retry.

The catalog also supports schema changes such as adding, dropping, renaming, and reordering columns without rewriting the underlying data. Iceberg tracks columns using unique IDs. Catalog-based queries can therefore continue using logical table and column names even if Fivetran changes the physical file layout. Readers can also apply delete files and see a consistent snapshot rather than rows that were logically deleted.

Credentials and supported query engines

Users can obtain four values from the Fivetran dashboard under Catalog integration > Base configuration: the Polaris server endpoint, the Polaris catalog name, a client ID, and a client secret. Fivetran warns that the client secret is displayed only once. It can be regenerated through the dashboard, but engine configurations then need to be updated.

The guidance includes OAuth2 configuration examples for Apache Spark, DuckDB, Snowflake, and Trino and Starburst. Spark requires Apache Iceberg 1.9.0 or later with Spark 3.4 or 3.5, while the DuckDB integration requires DuckDB 1.2.1 or later. Fivetran labels DuckDB's REST catalog integration as experimental, with storage currently limited to S3 and S3-compatible storage. GCS and ADLS support is pending.

Snowflake's REST catalog integration for external Iceberg tables is generally available for reads, while write support is in preview. Fivetran recommends enabling automatic refresh when creating Iceberg tables in Snowflake. The default refresh interval is 300 seconds, or five minutes, and the setting accepts values from 30 to 86400 seconds.

For users, the practical change is to refer to Fivetran-managed data with catalog.schema.table names rather than raw cloud storage paths. This lets the query engine discover the current snapshot, locate data and delete files, and plan a scan using the table's metadata.

Related news

Together AI launches free storage caching for GPU clusters
Products and toolsNovember 19, 2026

Together AI launches free storage caching for GPU clusters

Together AI has launched Together Storage Accelerator, a distributed caching layer for its GPU Clusters that serves repeated data reads from local NVMe storage. The…

LanceDB proposes multi-base paths for distributed Lance datasets
For developersNovember 16, 2026, 18:17

LanceDB proposes multi-base paths for distributed Lance datasets

LanceDB has introduced Lance’s multi-base path model, a design aimed at distributing one dataset across multiple storage locations without giving up the portability of…

Products and tools
Products and toolsNovember 16, 2026, 21:58

Snowflake Opens Iceberg Data Sharing to External Tools

Snowflake AI & Data Sharing now lets providers share supported Apache Iceberg tables with consumers using Iceberg REST Catalog compatible clients, including Apache…