Sls

Sls source connector

Key Features​

• batch

• stream

• exactly-once

• column projection

• parallelism

• support user-defined split

Description​

Source connector for Aliyun Sls.

Source Options​

Name	Type	Required	Default	Description
project	String	Yes	-	Aliyun Sls Project
logstore	String	Yes	-	Aliyun Sls Logstore
endpoint	String	Yes	-	Aliyun Access Endpoint
access_key_id	String	Yes	-	Aliyun AccessKey ID
access_key_secret	String	Yes	-	Aliyun AccessKey Secret
start_mode	StartMode[earliest],[group_cursor],[latest]	No	group_cursor	The initial consumption pattern of consumers.
consumer_group	String	No	Nexus-Consumer-Group	Sls consumer group id, used to distinguish different consumer groups.
auto_cursor_reset	CursorMode[begin],[end]	No	end	When there is no cursor in the consumer group, cursor initialization occurs
batch_size	Int	No	1000	The amount of data pulled from SLS each time
partition-discovery.interval-millis	Long	No	-1	The interval for dynamically discovering topics and partitions.

Task Example​

Simple​

This example reads the data of sls's logstore1 and prints it to the client.

Create RAM user and authorization,Please ensure thr ram user have sufficient rights to perform, reference RAM Custom Authorization Example

# Defining the runtime environment
env {
  parallelism = 2
  job.mode = "STREAMING"
  checkpoint.interval = 30000
}

source {
  Sls {
    endpoint = "cn-hangzhou-intranet.log.aliyuncs.com"
    project = "project1"
    logstore = "logstore1"
    access_key_id = "xxxxxxxxxxxxxxxxxxxxxxxx"
    access_key_secret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    schema = {
      fields = {
            id = "int"
            name = "string"
            description = "string"
            weight = "string"
      }
    }
  }
}

sink {
  Console {
  }
}