Sink

Sink connectors in Nexus come with several core features, and each sink connector offers varying levels of support for these features.

• Exactly-once: When data enters a distributed system, and each data piece is processed only once with accurate results throughout the entire workflow, the system is considered to meet exactly-once consistency.

  For a sink connector, it supports exactly-once if each data piece is written to the target only once. There are typically two methods to achieve this:

  1. The target database supports key deduplication. For example: MySQL, Kudu.

  2. The target supports XA Transactions (transactions that can be applied across sessions, even if the original session ends; a new session can continue by knowing the last transaction’s ID to either resubmit or roll back the transaction). By using a Two-phase Commit, exactly-once can be ensured. For example: File, MySQL.

• CDC (Change Data Capture): A sink connector is considered to support CDC if it can handle row kinds (INSERT, UPDATE_BEFORE, UPDATE_AFTER, DELETE) based on a primary key.

• Support for writing multiple tables: Nexus allows writing to multiple tables in a single job. Users can dynamically specify the table's identifier by configuring placeholders.