Configuration¶

pg2iceberg takes its settings from environment variables and an optional YAML file. Two are required — the source database and the Iceberg catalog — and the rest have defaults or are inferred:

export POSTGRES_URL=postgres://user:password@db.example.com:5432/app
export ICEBERG_CATALOG_URL=https://catalog.example.com
export ICEBERG_WAREHOUSE=s3://my-bucket/warehouse/
pg2iceberg run

pg2iceberg init inspects the database and writes a config file to start from — see below.

Where settings come from¶

defaults < config file < environment variables < flags

The config file is --config <path>, else PG2ICEBERG_CONFIG, else pg2iceberg.yaml in the current directory if it exists. Every section of it is optional.

In the file's string values, ${VAR} is replaced by the environment variable VAR (an error if it isn't set), and $${ is a literal ${. Comments aren't read, so a commented-out line may name a variable that isn't set. This keeps secrets out of the file:

sink:
  catalog_uri: https://catalog.example.com
  catalog_token: ${CATALOG_TOKEN}

Environment variables¶

AWS's own variables are read as AWS tools read them, and give way to the config file: AWS_REGION / AWS_DEFAULT_REGION for sink.s3_region, AWS_ENDPOINT_URL_S3 / AWS_ENDPOINT_URL for sink.s3_endpoint. Credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN) are read by the AWS default credential chain, with credential_mode: iam.

Inferred settings¶

Generating a config¶

POSTGRES_URL=postgres://... pg2iceberg init

init reads the environment (not an existing config file), and writes pg2iceberg.yaml (--output <path>, or --output - to print; --force to overwrite):

• every table it would replicate, with its estimated size; the tables it would skip are listed, commented out, with the reason;
• the settings the environment gives; secrets (POSTGRES_URL, catalog tokens and client secrets) as ${VAR} references, so they stay in the environment.

It also checks what replication needs of the database, and says how to fix what's missing: wal_level = logical, the replication privilege, a free replication slot, a place for pg2iceberg's state schema, and ownership of the tables (creating the publication takes it).

Listing the tables pins them: a table created later isn't replicated until it's added. Delete tables: to replicate every table with a primary key again.

Full reference¶

# Replication source
source:
  mode: logical                  # the only mode (default); may be omitted
  postgres_url: ""               # or a connection string, in place of `postgres`
  postgres:
    host: ""                     # required without postgres_url
    port: 5432
    database: ""                 # required without postgres_url
    user: ""                     # required without postgres_url
    password: ""
    sslmode: disable             # "disable" | "require" | "verify-ca" | "verify-full"
                                 # — currently anything non-disable enables
                                 #   webpki-roots verification; finer-grained
                                 #   modes (mTLS, custom CA) are follow-ons.

  # Logical replication settings
  logical:
    publication_name: pg2iceberg_pub
    slot_name: pg2iceberg_slot
    standby_interval: 10s        # how often the standby_status ack is sent
    snapshot_concurrency: 4      # tables the initial snapshot reads at once

# Tables to replicate. Leave out for every table with a primary key.
tables:
  - name: public.orders          # fully-qualified PostgreSQL table name, or schema.*
    iceberg:
      partition:                 # partition transforms — six are supported:
        - "day(created_at)"      #   year/month/day/hour
        - "bucket[16](id)"       #   murmur3_x86_32 % N
        - "region"               #   identity
        - "truncate[4](name)"    #   string truncation / int floor

    # Optional — overrides discovered PK
    # primary_key: [id]

    # Optional — overrides discovery from information_schema/pg_index
    # columns:
    #   - { name: id, pg_type: int4 }
    #   - { name: qty, pg_type: int8, nullable: true }

    # Skip the initial snapshot for this table (already populated by some
    # other process). Default: false.
    # skip_snapshot: false

# Iceberg sink
sink:
  catalog_uri: ""                # Iceberg REST catalog URL (required)
  catalog_auth: ""               # "none" | "bearer" | "oauth2" | "sigv4"; default: inferred
  catalog_token: ""              # bearer token
  catalog_client_id: ""          # OAuth2 client ID
  catalog_client_secret: ""      # OAuth2 client secret

  credential_mode: ""            # "static" | "iam" | "vended"; default: inferred
  warehouse: ""                  # s3://bucket/prefix (required for static/iam)
  namespace: ""                  # one Iceberg namespace for every table; default: each table's PG schema
  s3_endpoint: ""                # S3-compatible storage (MinIO, R2); default: AWS
  s3_access_key: ""              # static credentials
  s3_secret_key: ""
  s3_region: ""                  # default: inferred

  # Flush thresholds — flush when any threshold is reached
  flush_rows: 10000              # also caps change events held in memory
  materializer_batch_rows: 50000 # change events per materializer step
  materializer_concurrency: 4    # tables the materializer applies at once
  pk_index_memory: 536870912     # 512 MiB: memory for each table's primary keys
  flush_interval: 10s            # how often changes are staged

  # How often the materializer commits staged changes to Iceberg
  # (`run` and `materializer-only`)
  materializer_interval: 10s

  # Compaction. Runs as part of every materializer cycle, gated by these
  # thresholds. `target_file_size: 0` disables compaction entirely.
  compaction_data_files: 8       # merge small files once there are this many data files
  compaction_delete_files: 32    # rewrite files with deleted rows once there are this many delete files,
  compaction_delete_ratio: 0.1   # or once deleted rows are this share of those files' rows
  target_file_size: 134217728    # 128 MiB

  # Who maintains the tables: `pg2iceberg` (compaction above, and
  # `pg2iceberg maintain`) or `managed`: their catalog — S3 Tables, R2
  # Data Catalog, Glue's table optimizers. See Table Maintenance.
  maintenance: pg2iceberg

  # `pg2iceberg maintain` (snapshot expiry + orphan cleanup)
  maintenance_retention: 168h    # 7 days. Snapshots older than this are dropped.
  maintenance_grace: 30m         # Orphan files younger than this are protected.
                                 # Cleanup only scans each table's own directory,
                                 # <warehouse>/materialized/<namespace>.<table>/.

  # Free-form REST catalog props passthrough. Layered on top of the
  # built-in props (uri, warehouse, auth, S3, access-delegation header).
  # catalog_props:
  #   "header.X-Custom-Header": "value"

  # Control-plane meta tables. When set, pg2iceberg writes operational
  # telemetry (commits, compactions, maintenance ops, blue-green markers)
  # to Iceberg tables under this namespace.
  # meta_namespace: _pg2iceberg_meta

# Coordinator state
state:
  # Optional dedicated PG for the _pg2iceberg coordinator schema.
  # When empty, the source PG hosts it.
  # postgres_url: postgres://coord_user:secret@coord-db.example.com/coord?sslmode=require
  coordinator_schema: _pg2iceberg
  group: default                 # consumer group name (distributed mode)

metrics_addr: ":9090"            # /metrics, /healthz, /readyz; `off` serves nothing
liveness_timeout: 5m             # /healthz fails once nothing completes for this long
snapshot_only: false             # legacy field; prefer the `snapshot` subcommand

Notes on individual fields¶

• source.logical.snapshot_concurrency — the most tables the initial snapshot reads at once (default 4; 1 reads them one after another). Each is read over a Postgres connection of its own, all in the same snapshot, and written on its own, so the snapshot uses that many cores and connections, and holds up to two chunks of flush_rows rows for each table. One large table is still read by one connection.
• source.logical.snapshot_chunk_pages / snapshot_target_file_size — not yet exposed; the snapshot reads tables in chunks of flush_rows rows.
• sink.flush_rows — the most change events pg2iceberg holds in memory before staging them. A transaction bigger than this is staged in chunks of this size as it streams in, and claimed in one step when it commits, so readers never see part of it. The initial snapshot reads tables in chunks of this many rows, reading the next while it writes one, so it holds up to twice as many for each table it reads at once (snapshot_concurrency). Larger values mean fewer, larger staged files.
• sink.materializer_batch_rows — the most change events the materializer folds into one Iceberg snapshot, which bounds its memory, along with the up to eight staged files it reads ahead (each at most flush_rows events). A transaction larger than this is written as several snapshots committed in one atomic catalog update, so readers of the table never see part of it; catching up, the materializer commits up to four snapshots at a time anyway.
• sink.materializer_concurrency — the most tables the materializer applies at once (default 4; 1 applies them one after another). A table's work is mostly waiting on the object store and catalog, so more at once catch up faster on many tables; each holds its own materializer_batch_rows events and staged files read ahead, and commits on its own, so memory and the catalog's request rate grow with it. With meta tables on (meta_namespace), tables go one at a time.
• sink.pk_index_memory — the most memory, in bytes, the materializer holds of a table's primary keys: it maps each key to the data file holding its row (file index). Within it — about 7 million keys at the default 512 MiB, leaving room for the map to grow and a quarter for the filter of evicted keys — nothing changes. Past it, the oldest files' keys are evicted, and changes to their rows cost more: an insert that may be of one writes an equality delete, and an update leaving TOAST columns out, a delete on a partitioned table, a TRUNCATE or a column added with a default reads the files that may hold them. Each table has its own budget.
• sink.flush_interval / sink.materializer_interval — the micro-batch cadence: changes are staged every flush_interval (or sooner, once flush_rows are buffered) and committed to Iceberg every materializer_interval (or back to back, while catching up on a backlog). A change typically reaches Iceberg within about the two added together. Shorter intervals mean fresher tables, and more, smaller Iceberg snapshots and files for compaction to merge.
• sink.compaction_delete_files / sink.compaction_delete_ratio — when compaction drops deleted rows. An update or delete leaves the old row in its data file, hidden by an equality delete that every reader applies; dropping it means rewriting the files holding such rows, and every older one, before the delete file can go. Under updates spread across a large table that's most of the table, so pg2iceberg waits until the deleted rows are compaction_delete_ratio of the rows those files hold (rewriting then reads at most 10 rows per row deleted, at 0.1), or until there are compaction_delete_files delete files for readers to open. Lower values keep reads cheaper; higher ones rewrite less.
• sink.flush_bytes — not yet exposed; flush threshold is flush_rows + flush_interval only.
• sink.materializer_target_file_size — not yet exposed; uses target_file_size.
• sink.materializer_worker_id — replaced by the --worker-id flag on pg2iceberg materializer-only.
• sink.meta_enabled — inferred from meta_namespace: setting it enables meta-table writes.
• metrics_addr / liveness_timeout — where run, stream-only, materializer-only and snapshot serve /metrics, /healthz and /readyz, and how long nothing may complete before /healthz fails. See Observability.
• state.path — file-based checkpoint store. Not implemented; coord is always Postgres-backed.
• snapshot_only — use the pg2iceberg snapshot subcommand instead. The field still parses for backward compat.