Observability¶

pg2iceberg has four observability surfaces:

1. Prometheus metrics and health checks on an HTTP endpoint, served by every long-running subcommand.
2. Traces, exported over OTLP when you point it at a collector: where the time in a flush or a materializer cycle goes, down to each request.
3. Structured logs on stdout, as text or JSON.
4. Iceberg control-plane meta tables, when sink.meta_namespace is set. See metadata-tables.md.

Use the metrics for live dashboards and alerts, traces to see why one cycle was slow or failed, and the meta tables for history, such as a per-commit audit trail you can query with SQL.

The endpoint¶

run, stream-only, materializer-only and snapshot serve three paths:

The one-shot subcommands (compact, maintain, verify, cleanup, ...) don't serve the endpoint.

A metrics_addr you set must be free, or pg2iceberg won't start. If you leave it at the default and port 9090 is already taken, for example by a second pg2iceberg on the same host, pg2iceberg logs a warning and runs without the endpoint.

curl -s localhost:9090/metrics | grep '^pg2iceberg_replication_lag_bytes'

Health checks¶

pg2iceberg records a metric only when it finishes something: a request to the catalog, object store or coordinator, a flush, an ack, a materializer cycle, or a watcher tick. Nothing records on a timer of its own.

So if nothing has been recorded for liveness_timeout, the process is stuck, not idle. Typical causes are a request that never returns or a deadlock. /healthz then answers 503, and restarting the process is the fix. Replication and the materializer work side by side in run, so a request that never returns counts even while the other keeps recording: /healthz fails once one has been in flight for liveness_timeout.

A process that is merely slow keeps passing. For example, a materializer cycle can take many minutes against a distant catalog, but its requests keep completing in the meantime. An idle run or stream-only still records every 30 seconds through its watcher.

materializer-only completes nothing between cycles, so its liveness timeout is at least three sink.materializer_intervals.

Kubernetes probes:

livenessProbe:
  httpGet: { path: /healthz, port: 9090 }
  periodSeconds: 30
  failureThreshold: 2
readinessProbe:
  httpGet: { path: /readyz, port: 9090 }
  periodSeconds: 10

/readyz reports ready during the initial snapshot, which can take hours. Rollouts don't need to wait for it to finish.

Metrics¶

Every metric pg2iceberg emits is listed below. The table label holds the Iceberg table, qualified with its namespace (public.orders). Every duration is a histogram in seconds, with buckets from 5 ms to 5 min.

Process¶

These come next to the standard process_cpu_seconds_total, process_resident_memory_bytes, process_virtual_memory_bytes, process_open_fds, process_max_fds, process_threads and process_start_time_seconds. On anything but Linux, only the start time is reported.

Replication¶

Read from pg_replication_slots every watcher tick (30 s).

Staging¶

Materializing¶

Compaction¶

Requests¶

Every request to the Iceberg catalog, the object store and the coordinator.

The materializer reads staged files ahead of the one it's applying, up to eight at once, while it uploads and commits what it has read: its gets overlap its other requests. A response that arrives while it's folding or encoding waits until it next awaits a request, so a get's duration and span can include that time.

Invariants¶

What to alert on¶

example/single/monitoring/alerts.yml has Prometheus rules for these. The essentials:

How stale a table is only means something while it has a backlog. A table nobody writes to keeps its last source commit time but is fully up to date, which is why the stale-table alert also checks the backlog.

The replication lag counts every write on the source, including writes to tables pg2iceberg doesn't replicate. pg2iceberg acks those writes along with its own.

Traces¶

pg2iceberg exports its spans as OpenTelemetry traces over OTLP/HTTP (protobuf) once OTEL_EXPORTER_OTLP_ENDPOINT says where: an OpenTelemetry Collector, Grafana Tempo, Jaeger, or a vendor's OTLP endpoint. Unset, nothing is exported.

OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318 pg2iceberg run

Each unit of work is a trace of its own, and every request it makes to the Iceberg catalog, the object store or the coordinator is a span within it, with its store, operation, duration and, if it failed, the error:

Requests are exported as <store>.<operation>, e.g. catalog.commit_snapshots, object_store.put, coordinator.claim_offsets.

The standard OpenTelemetry variables apply:

pg2iceberg exports over HTTP only — a collector's port 4318. gRPC (OTEL_EXPORTER_OTLP_PROTOCOL=grpc, port 4317) is refused at startup. An idle run makes a few traces a minute (acks, materializer cycles, watcher checks); under load, one per flush and per cycle. An export that fails is logged.

Try it¶

example/single runs Prometheus with these rules, Tempo for the traces, and Grafana with a pg2iceberg dashboard:

cd example/single
OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo:4318 docker compose --profile monitoring up -d --wait
docker compose --profile workload up -d workload

Grafana is at http://localhost:3000 (no login), Prometheus at http://localhost:9090, and pg2iceberg's own endpoint at http://localhost:9091/metrics. The dashboard's Traces row lists recent materializer cycles, failed traces and slow catalog requests; open one for its waterfall, or search them in Explore with TraceQL, e.g. { name = "materializer.commit" && span.table = "rideshare.rides" }.

Logs¶

The binary logs through tracing to stdout, as plain text, or one JSON object a line with PG2ICEBERG_LOG_FORMAT=json — for a log collector to parse. A message logged during a unit of work names it and its fields, e.g. materializer.cycle:materializer.table{table=public.orders}: .... Tune verbosity with RUST_LOG:

# Default
RUST_LOG=info,pg2iceberg=debug pg2iceberg run --config config.yaml

# Quiet
RUST_LOG=warn pg2iceberg run --config config.yaml

# Maximum verbosity for one crate
RUST_LOG=info,pg2iceberg_logical=trace pg2iceberg run --config config.yaml

Useful log targets¶

Iceberg control-plane meta tables¶

When sink.meta_namespace is set, pg2iceberg writes operational telemetry to Iceberg tables you can query from any Iceberg-compatible engine: one row per commit, compaction and maintenance run. See metadata-tables.md for the schema and example queries.