Kafka Source

Step 4 - Kafka Connection Settings

Kafka Bootstrap Servers

Indicates a list of Kafka servers the Kafka source should connect to. This list should be provided as comma-separated values

e.g. localhost:9092, localhost:9093

Topic List

Indicates the list of topics the source should subscribe to. This list should be provided as comma-separated values.

e.g. topic_one, topic_two

Kafka Source Group ID

Identifies the Kafka source group through an ID. The Group ID ensures that sources having the same topic and partition within the same group don’t receive duplicate events.

Step 5 - Threading and Partition Management

Threading Options

Specifies if the Kafka source should operate on a single thread or multiple threads. Possible values are

• Single Thread: o run the Kafka source on a single thread.

• Topic-wise: To use a separate thread for each topic.

• Partition-wise: To use a separate thread for each partition.

Partition Numbers

Indicates a list of the partition numbers for the chosen topic. This list should be provided as comma-separated values.

e.g. 0, 2, 4

Step 6 - Event Handling

Sequence Tracking

When set to ON, it takes into account the sequence of events received through the source. Each event must include a sequence number to indicate its order.

Binary Message Reception

Must be set to ON to enable the Kafka source to receive binary events.

Topic Offset Mapping

Defines the reading offsets for each topic and partition in key value pairs. When an offset is set, the Kafka source skips reading messages before that offset.

e.g. reading2=20, temperature=500 reads from the 21th message of the reading2 topic, and from the 501th message of the temperature topic.

Step 7 - Offset Commit Settings

Enable Offset Commiting

Auto Commit Interval

Enable Asynchronous Commiting

Step 8 - Additional Settings

Additional Configurations

Contains any other configurations for the consumer in key-value format. Some supported configurations can be exemplified as

e.g. Keystore Type: 'ssl.keystore.type' as key and 'JKS' as value.

e.g. Batch Size: 'batch.size' as key and '200' as value.

Step 9 - Preview

In Preview Step, you're provided with a concise summary of all the changes you've made to the Kafka Source Node. This step is pivotal for reviewing and ensuring that your configurations are as intended before completing node setup.

• Viewing Configurations: Preview Step presents a consolidated view of your node setup.

• Saving and Exiting: Use the Complete button to save your changes and exit the node and return back to Canvas.

• Revisions: Use the Back button to return to any Step of modify node setup.

The Preview Step offers a user-friendly summary to manage and finalize node settings in Cortex.