BeziSync

Destinations

Each destination states what it needs from a pipeline, and the wizard only offers that: the names it accepts (with a live preview), its event formats, the event shape it requires, its own pipeline settings and the smallest connector size it runs in. Preflight checks the rest.

#Apache Iceberg

Changes land in Iceberg tables, one per source table, through an Iceberg REST catalog or AWS Glue. Presets: Iceberg REST catalog (Polaris, Lakekeeper, Gravitino, Nessie), Amazon S3 Tables (SigV4, service s3tables, warehouse = table bucket ARN), AWS Glue Data Catalog (warehouse = an s3:// location), Snowflake Open Catalog (OAuth2, scope PRINCIPAL_ROLE:ALL, vended credentials), Databricks Unity Catalog (token, vended credentials) and RustFS S3 Tables (preview: SigV4 service s3, warehouse = the table bucket, path-style S3; change logs only, as it refuses the first upsert commit of a new table). Data files are written with the object-storage keys of the connection, the runtime's IAM role, or credentials the catalog vends. Snowflake, BigQuery (BigLake), Databricks, Trino, StarRocks and ClickHouse read the tables through the same catalogs.

  • Tables are created by their first change in the connection's namespace (BeziSync creates the namespace on deploy): columns from the source, the primary key (or the key columns chosen in the Transform step) as row identity, lower-case names with dots as _ (naming {schema}_{table}). New source columns are added; other schema changes stop the pipeline.
  • Rows (Destination step): latest row, deletes kept with __deleted = true (default), latest row, deletes removed, or every change appended (a change log). Tables without a key are always appended (a preflight warning says which).
  • Commits at least every minute by default (5 or 15 minutes for fewer, larger files; "as soon as possible" commits every batch). Health counts that interval as expected delay, not lag.
  • Partitioning: upserted tables are bucketed by their first key column (bucket(id, 16); 4 or 64 buckets in the Destination step), change logs by the day of the change (day(__source_ts_ms)). The wizard records each table's key column in sink.options.partitions (set it there through YAML or the API); existing tables keep their partitioning. Expert mode can turn it off.
  • Event shape: Iceberg needs the row after change with __op and __source_ts_ms; BeziSync adds that step (or completes yours) and sends timestamps as ISO-8601 text (Kafka Connect types for SQL Server). Outbox steps and tombstone deletes are not available.
  • Preflight signs in to the catalog, checks the namespace, writes a one-row test table, reads it back and drops it, then checks keys and tables that already exist. The Events tab and the delivery canary read the newest rows of each table.
  • Image: pipelines run ghcr.io/memiiso/debezium-server-iceberg (see Debezium versions), amd64 only. It serves health on port 9000, which BeziSync probes; agent runtimes need an agent of this BeziSync version or newer (preflight says so). Deleting a pipeline with "drop destinations" drops its tables (purging files where the catalog allows; RustFS keeps them).

#Database (JDBC)

Changes land in PostgreSQL, MySQL, MariaDB or SQL Server tables, one per source table, written by Debezium's JDBC sink connector (the official images ship the drivers).

  • Tables are created in the connection's schema (PostgreSQL), the login's default schema (SQL Server, dbo unless set otherwise) or the database (MySQL, MariaDB) with the source's key as primary key, and get new columns as the source changes (naming {schema}_{table}; dots become _). "Use existing tables only" turns that off.
  • Rows (Destination step): upsert by key (default: the table mirrors the source) or insert only (every change is a new row; needs no key). Deletes delete the row, or are ignored. With the "row after change" shape, BeziSync turns deletes into what this setting needs.
  • Keys: upserts need a key on every table; preflight blocks tables without one until you pick key columns (Transform step) or insert only.
  • Preflight reaches the database, signs in, creates and drops a test table, and checks keys and name lengths (63 characters in PostgreSQL, 64 in MySQL, 128 in SQL Server). The Events tab and the delivery canary read rows of the tables.
  • Deleting a pipeline with "drop destinations" drops its tables.
  • Across databases, Debezium copies column defaults into the tables it creates: a PostgreSQL timestamptz DEFAULT now() column fails to create in MySQL or MariaDB ("Invalid default value"). Exclude the column (Tables step) or create the table yourself and choose "Use existing tables only".

#Amazon SQS and Amazon SNS

  • SQS: every table of the pipeline goes to the connection's queue, one message per change, so it suits low-volume tables; consumers tell tables apart by source.table (or __table with the "row after change" shape). A FIFO queue needs content-based deduplication (Debezium sends no deduplication ID). The Events tab peeks at messages still waiting (they stay available to your consumers, but each look counts as a receive, so it is off when the queue's dead-letter redrive allows fewer than 10 receives); there is no delivery canary, as consumers may take it first. BeziSync keeps the connector from logging message bodies.
  • SNS: one topic per table (cdc_{schema}_{table}; names ending in .fifo make FIFO topics, created with content-based deduplication and ordered per row key). Topics are created on deploy unless turned off. SNS keeps nothing to read back, so BeziSync subscribes a small queue of its own to each topic (bz-preview-<topic>, 15-minute retention) for the Events tab and the delivery canary; deleting the pipeline removes them (and the topics with "drop destinations").
  • Both: keys or the runtime's IAM role; JSON or CloudEvents (SQS and SNS take text); deletes as __deleted rows or dropped (they refuse empty "tombstone" messages, so BeziSync turns those off); messages over 256 KiB (SNS, and SQS by default) stop the pipeline.

#Azure Event Hubs

Every table of the pipeline goes to the connection's event hub (it must exist), partitioned by the row key so each row keeps its order; consumers tell tables apart by source.table (or __table). Sign in with a namespace connection string (shared access policy with Send), without EntityPath. The Events tab and the delivery canary read the last events of each partition, which needs Listen: add a second connection string for reading when the first cannot listen (preflight warns). Events over the tier's limit (1 MB on Standard) stop the pipeline. Entra ID sign-in is not available: Debezium's sink supports connection strings only.

#Apache Pulsar

One persistent topic per table in the connection's tenant and namespace, keyed by the row key. The connector produces to the service URL (pulsar://…:6650, or pulsar+ssl://…:6651 with an optional private CA); BeziSync uses the admin URL (http(s)://…:8080) to check the namespace, create missing topics on deploy, and read the newest messages for the Events tab and the delivery canary (examinemessage: no subscription, your consumers are untouched). Sign in with a token (JWT, or a StreamNative API key). A send that does not complete within 30 seconds fails the batch, which Debezium retries, rather than being lost.

Edit this page on GitHub