The Iceberg Dream: load, transform, and serve Snowflake-managed Iceberg using zero Snowflake compute

TLDR

Snowflake Horizon's Iceberg REST Catalog lets you load and transform data as governed Iceberg tables using zero Snowflake compute. dlt writes raw data to Iceberg using an open-source pipeline, dbt Fusion with DuckDB transforms it, and both commit directly through Snowflake's catalog without ever starting a warehouse. The moment that data lands, it's fully governed by Snowflake: RBAC, masking, lineage, tags, Cortex. This became possible in late June 2026 once DuckDB 1.5.4, dbt Fusion preview.194+, and Snowflake Horizon's external-write support all shipped together.

At Snowflake Summit 2026 Snowflake claimed they have the broadest support for Iceberg V3, being the first fully open catalog to enable external reads and writes to Iceberg, including deletion vectors, row lineage, variant type, and default values. So, I had to put this to the test.

The “external writes” aspect, plus some recent developments from dbt and DuckDB, enables a version of the lakehouse I’ve wanted to build for my clients which was not possible until late June 2026. It goes like this: your tables live in your own cloud storage, in an open format, a catalog that has zero management, easy to stand up, simple to read and write to, readable by any engine, and writeable by other engines such as DuckDB. But these tables are still governed by Snowflake: one catalog, one set of grants, one place where masking and lineage and tags live. Open storage on the bottom, real governance + Snowflake AI on top.

Land raw data as Apache Iceberg in your own S3 bucket, with no Snowflake warehouse running.
Transform it into a new Iceberg table using dbt. Again: no Snowflake warehouse running.
Serve it to AI and BI with the full Snowflake governance stack: RBAC, masking policies, row-access policies, tags, lineage, Cortex.

The thing that makes this possible is Snowflake Horizon's Iceberg REST Catalog. External engines (dlt, DuckDB, dbt running on your laptop or a tiny container) write Snowflake-managed Iceberg tables directly over that API. Snowflake owns the catalog, vends scoped S3 credentials at write time, and your engine does the actual compute. What comes out is a first-class governed table. Not an external table Snowflake squints at. A real one. And the catalog is already there. You don't set it up. You don't turn it on. It came with Snowflake.

The transform step is the freshest part of this stack, as the rest of it was previously possible. It took three specific releases all shipping at once: DuckDB 1.5.4 added the write-compatibly options that let DuckDB commit Iceberg back through a REST catalog, dbt Fusion preview.194+ exposed those options in catalogs.yml v2, and Snowflake Horizon completed its external-engine write path via the Iceberg REST Catalog API. If you tried this prior June 30, 2026 (Fusion 194 release), the write simply failed.

What "Iceberg REST Catalog" actually means here

Quick level-set on vocabulary. Apache Iceberg is the open table format that lets you read and write parquet files like tables (select, insert, etc.), plus a metadata tree that gives you ACID commits, schema evolution, and time travel. The REST Catalog is the open API spec that tells an engine where the current metadata is and coordinates SQL commits.

Snowflake Horizon implements that REST Catalog API and points it at Snowflake-managed Iceberg tables. So when DuckDB or dlt speaks the Iceberg REST protocol, Snowflake answers, as the catalog. The endpoint looks like this:

1https://<your-account>.snowflakecomputing.com/polaris/api/catalog

Two things to internalize:

In REST-catalog speak, the warehouse is your Snowflake database (in our example below we call it ICE_RAW), not a virtual warehouse. No compute is implied by the word warehouse in this case.
Auth is an OAuth2 token exchange. You hand Horizon a Snowflake Programmatic Access Token (PAT) as the client_secret, with a scope like session:role:TRANSFORMER, and it hands you back a short-lived bearer token. Your role, and every grant attached to it, rides along on that token.

That second point is the whole governance story in one sentence: the catalog hands out access by Snowflake role. Hold that thought; it comes back in the "serve" section.

The architecture, on one screen

Every arrow from an engine into the catalog carries no Snowflake compute. The engines do the work; Horizon governs and brokers. The data sits in your bucket the whole time.

The one-time foundation (and why it's not the catalog)

The setup involved is surprisingly easy. Snowflake has removed almost all of the complexity around setting up the Iceberg Catalog. Here is what you need to set up.

Where the data lives. An S3 bucket and a Snowflake external volume that points at it, with an IAM role Snowflake assumes. This is the "your storage, your bucket" part of the open lakehouse.
Who can touch it. A couple of Snowflake roles and a service user with Programmatic Access Tokens. This is just Snowflake RBAC. You already know it.

The companion repo does all of this with Terraform + a little SQL, so you can read every line. Storage:

1# tf/snowflake: the external volume + Iceberg-by-default databases
2resource "snowflake_external_volume" "horizon" {
3  name = "HORIZON_EXT_VOL"
4  storage_location {
5    storage_location_name = "horizon-s3"
6    storage_provider      = "S3"
7    storage_base_url      = "s3://my-org-iceberg/horizon/"
8    storage_aws_role_arn  = aws_iam_role.snowflake.arn
9  }
10}
11
12resource "snowflake_database" "ice_raw" {
13  name   = "ICE_RAW"
14  external_volume = snowflake_external_volume.horizon.name
15  catalog  = "SNOWFLAKE"   # Snowflake-managed Iceberg

Access: plain RBAC, two roles, a service user that holds them.

1-- sql/horizon_access.sql (abridged)
2CREATE ROLE IF NOT EXISTS LOADER;       -- writes ICE_RAW
3CREATE ROLE IF NOT EXISTS TRANSFORMER;  -- reads ICE_RAW, writes ICE_TRANSFORMED
4
5GRANT USAGE ON DATABASE ICE_RAW TO ROLE TRANSFORMER;
6GRANT USAGE ON SCHEMA ICE_RAW.LANDING TO ROLE TRANSFORMER;
7GRANT SELECT ON ALL ICEBERG TABLES IN SCHEMA ICE_RAW.LANDING TO ROLE TRANSFORMER;
8GRANT SELECT ON FUTURE ICEBERG TABLES IN SCHEMA ICE_RAW.LANDING TO ROLE TRANSFORMER;
9
10GRANT USAGE ON DATABASE ICE_TRANSFORMED TO ROLE TRANSFORMER;
11GRANT CREATE SCHEMA ON DATABASE ICE_TRANSFORMED TO ROLE TRANSFORMER;
12
13CREATE USER IF NOT EXISTS HORIZON_SVC TYPE = SERVICE;
14GRANT ROLE LOADER, TRANSFORMER TO USER HORIZON_SVC;

Then you issue the tokens. Each engine gets a PAT scoped to exactly one role, least privilege baked in:

1snow sql -q "ALTER USER HORIZON_SVC ADD PAT HORIZON_LOAD_PAT DAYS_TO_EXPIRY=7 \
2  ROLE_RESTRICTION='LOADER'      COMMENT='dlt loader'"      --role ACCOUNTADMIN
3snow sql -q "ALTER USER HORIZON_SVC ADD PAT HORIZON_PAT      DAYS_TO_EXPIRY=7 \
4  ROLE_RESTRICTION='TRANSFORMER' COMMENT='dbt transformer'" --role ACCOUNTADMIN

That's it. That's the foundation. Storage you own, roles you grant. Nowhere in there did you build a catalog, because the catalog is Horizon, and Horizon is just… on.

💡One real gotcha worth saving you an hour: the hostname uses a hyphen, not an underscore. It's myorg-myacct.snowflakecomputing.com, and Python's ssl rejects the underscore form. curl tolerates it, so it'll look like it works until your pipeline doesn't.

Pillar 1: Load, with no Snowflake compute

The full code is available in the repo here. In this blog I'll pull out some relevant code for readability.

The loader is dlt using its open-source filesystem destination with table_format="iceberg", pointed at the Horizon REST catalog. No dlt+, no licensed connector. dlt writes the parquet and commits through the catalog with pyiceberg; Horizon vends the S3 creds.

The catalog connection lives in dlt's secrets, and the only Snowflake-specific trick is this: Snowflake-managed Iceberg assigns the table location itself (under your external volume), so it rejects a client-supplied location. We wrap dlt's create_table to omit it:

1import dlt
2from dlt.destinations import filesystem
3import dlt.common.libs.pyiceberg as _ice
4import pyarrow as _pa
5
6# Snowflake Horizon assigns the table location under the external volume and
7# rejects a client-supplied one, so drop `location` on create. That's the
8# intended external-write flow.
9def _create_no_location(catalog, table_id, table_location, schema, **kw):
10    if isinstance(schema, _pa.Schema):
11        schema = _ice.ensure_iceberg_compatible_arrow_schema(schema)
12    catalog.create_table(identifier=table_id, schema=schema,
13                         partition_spec=kw.get("partition_spec", _ice.UNPARTITIONED_PARTITION_SPEC),
14                         properties=kw.get("properties") or {})
15_ice.create_table = _create_no_location

Run it:

1uv run python hello_world_pipeline.py

And here's the moment. Go to Snowflake (now you can use a warehouse, because you're a human running a query) and the table is right there, governed, queryable, kind MANAGED:

1SELECT * FROM ICE_RAW.LANDING.HELLO_WORLD;

Rows, written by an open-source Python process, that never started a Snowflake warehouse. The bytes are in your bucket. The governance is Snowflake's. Sweet.

Pillar 1 Bonus:

For science, I decided to schedule the EL job on dltHub’s new SaaS platform, dltHub Pro. I realize how pointless it is to schedule a job that writes the words “hello” and “world” to Snowflake, but new shiny toys, I couldn’t resist.

The pipeline has now been running for two weeks, every 30 minutes during business hours. It has been executed about 200 times and has cost about $2 so far. (REST catalogs are notoriously slow so most of that money has been spent waiting.)

Pillar 2: Transform, with no Snowflake compute

This next part is most exciting because it was not possible 1 week ago.

The transformer is dbt with DuckDB doing the work. DuckDB attaches the Horizon REST catalog twice: ICE_RAW to read what dlt landed, ICE_TRANSFORMED to write the result. It reads, runs your SQL, and writes a brand-new Snowflake-managed Iceberg table straight back through the catalog. No handwaving or tricks.

How this was unlocked: DuckDB 1.5.4 shipped a set of Iceberg-REST write-compat options (duckdb-iceberg#1017), and dbt Fusion (preview.194+) exposes them in catalogs.yml v2. Four options teach DuckDB to commit in exactly the shape Horizon accepts. Here's the catalog config; the write target carries the magic:

1# dbt/catalogs.yml
2catalogs:
3  - name: ice_raw          # READ
4    type: iceberg_rest
5    table_format: iceberg
6    config:
7      duckdb:
8        endpoint: " env_var('HORIZON_CATALOG_URI') "
9        warehouse: ICE_RAW          # = the Snowflake database
10        secret: horizon
11        attach_as: ice_raw
12        access_delegation_mode: VENDED_CREDENTIALS
13
14  - name: ice_transformed  # WRITE
15    type: iceberg_rest

Turn the project on with the catalogs v2 flag, and bind your staging model to the write catalog:

1# dbt/dbt_project.yml
2flags:
3  use_catalogs_v2: true
4
5models:
6  horizon_iceberg:
7    staging:
8      +materialized: table
9      +catalog_name: ice_transformed
10      +schema: STAGING
11    marts:
12      +materialized: table
13      +catalog_name: ice_transformed
14      +schema: MARTS

Those +schema: values land each model in its own Iceberg namespace (STAGING, MARTS) inside ICE_TRANSFORMED. Like normal use of dbt, you don't pre-create schemas: dbt runs create schema if not exists before each model (just like normal), DuckDB-iceberg turns that into a namespace-create against the Horizon REST catalog, owned by the transformer role that created it.

💡Keep the schema names uppercase. Since duckdb handles the schema creation, you don’t get Snowflake unquoted identifier behavior.

In the repo you’ll find a basic staging model that just shows we can CTAS a new table in a different schema based on the ice_raw table. And a set of marts queries just to demo how we can use multiple schemas.

Here are the marts tables in Snowflake, owned by the transformer role:

If we want proof these dbt-written tables are real catalog commits and not a local trick, we can read it back from a completely different engine: pyiceberg or duckdb straight from the catalog. (The repo also includes a set of duckdb queries to read the tables from Horizon.)

1import os
2from pyiceberg.catalog.rest import RestCatalog
3cat = RestCatalog("h", uri=os.environ["HORIZON_CATALOG_URI"], warehouse="ICE_TRANSFORMED",
4    credential=os.environ["HORIZON_PAT"], scope="session:role:TRANSFORMER",
5    **{"header.X-Iceberg-Access-Delegation": "vended-credentials"})
6t = cat.load_table(("STAGING", "stg_hello_world")).scan().to_arrow()
7print(t.num_rows, "rows")

To check from Duckdb directly, launch Duckdb in the terminal then,

1INSTALL iceberg;
2
3LOAD iceberg;
4
5-- see repo, it will tell you how to generate the token
6CREATE OR REPLACE SECRET horizon (
7    TYPE ICEBERG,
8    TOKEN getenv('HORIZON_TOKEN')
9);
10
11-- 3. Attach the Snowflake-managed Iceberg catalog. warehouse = database name (ICE_RAW).
12ATTACH 'ICE_RAW' AS horizon (
13    TYPE ICEBERG,
14    ENDPOINT 'https://myorg-myacct.snowflakecomputing.com/polaris/api/catalog',
15    SECRET horizon

A model authored in dbt, executed by DuckDB, landed as governed Iceberg in Snowflake, and read back by pyiceberg. Three engines, one table, one catalog. That's the open lakehouse working as advertised.

💡If you can't run Fusion yet, the same native write works on stock dbt-core + dbt-duckdb with a tiny custom materialization. The repo ships it as a fallback in dbt/macros-duckdb/, with the two dbt-core-only gotchas documented (dbt-duckdb silently drops false boolean attach options; and the data-append needs an explicit commit). Fusion handles both for you, which is why it's the headline path.

Pillar 3: Serve it to AI and BI, governed

These are Snowflake-managed tables, so we can keep Pillar 3 very short. Everything you already do in Snowflake works: row-access policies, column masking, object tags, lineage, Cortex functions, semantic views, agents, BI tools. No special Iceberg configuration required.

Easier than DuckLake?

A few weeks ago a shared a well received demo about how easy DuckLake is. But dare I say Iceberg on Horizon is even easier?

DuckLake Horizon Iceberg Open storage (your bucket) ✅ ✅ Open table format ✅ (DuckLake format) ✅ (Apache Iceberg) Catalog you operate A database you stand up (often Postgres) None: it's Snowflake Horizon Governance (RBAC, masking, tags, lineage) Roll your own Already there AI + BI serving layer Bring it Already there (Cortex + any BI) Setup to first governed table Provision catalog DB + storage Storage + grants; no catalog

The catalog row is the whole story. In DuckLake you provision and babysit the metadata database. Here you don't, because the metadata database is Horizon, and you didn't build Horizon; you bought Snowflake, and it came on.

That's the "easiest thing since sliced bread" feeling people keep describing. It's not that the lakehouse got more powerful. It's that the hardest piece (a trustworthy, governed catalog) turned out to be the piece you don't have to make.

Try it

The full, reproducible repo is here. The fast path per the readme:

1# 1. Foundation: your storage + Snowflake grants (one time)
2cd tf/aws && terraform init && terraform apply
3cd ../snowflake && terraform init && terraform apply
4snow sql -f sql/horizon_access.sql --role ACCOUNTADMIN
5
6# 2. Load: no Snowflake compute
7cd dlt && uv run python hello_world_pipeline.py
8
9# 3. Transform: no Snowflake compute
10cd ../dbt && dbt system update && dbt run   # dbt Fusion >= preview.194
11
12# 4. Serve: governed, from Snowflake
13snow sql -q "SELECT * FROM ICE_TRANSFORMED.STAGING.STG_HELLO_WORLD"

Wrap up

Open data in your bucket. A catalog you didn't have to build. Loads and transforms that don't burn warehouse time. And the second that data exists, it's wearing all of Snowflake's governance (the RBAC, the masking, the tags, the lineage, the AI) because to Snowflake it was never "external" in the first place.

dbt is embracing Open Data Infrastructure (yes, a new buzz phrase for you) and making moves to make dbt more compatible with this vision.

Now go try it and let me know if Iceberg no longer feels complicated and intimidating!

FAQ

Do I need to run a Snowflake warehouse to use this architecture?

No, not for loading or transforming. Both dlt and dbt (via DuckDB) write directly to Snowflake-managed Iceberg tables through the Horizon REST Catalog API. Snowflake vends scoped S3 credentials at write time, but the actual compute happens on your engine, not a Snowflake warehouse. You only need a warehouse when a human (or Cortex, or a BI tool) queries the data afterward.

What's the difference between this and just using an external table in Snowflake?

An external table is something Snowflake reads but doesn't fully own or govern the same way. Tables written through the Horizon REST Catalog are first-class Snowflake-managed Iceberg tables. They show up with kind MANAGED, and every governance feature, RBAC, masking policies, row-access policies, tags, lineage, applies to them exactly as it would to a native Snowflake table. Snowflake was never treating this data as external in the first place.

Why wasn't this possible before late June 2026?

Three separate releases had to land together. DuckDB 1.5.4 added the write-compatibility options needed to commit Iceberg tables back through a REST catalog. dbt Fusion preview.194+ exposed those options in catalogs.yml v2. And Snowflake Horizon had to complete its external-engine write path via the Iceberg REST Catalog API. Before dbt Fusion 194 shipped on June 30, 2026, attempting the write step simply failed.