# OssFile > Oss file source connector ### Key features[​](https://seatunnel.apache.org/docs/2.3.7/connector-v2/source/OssFile#key-features) * [x] batch * [ ] stream * [x] exactly-once Read all the data in a split in a pollNext call. What splits are read will be saved in snapshot. * [x] column projection * [x] parallelism * [ ] support user-defined split * [x] file format type * [x] text * [x] csv * [x] parquet * [x] orc * [x] json * [x] excel * [x] xml * [x] binary ### Data Type Mapping[​](https://seatunnel.apache.org/docs/2.3.7/connector-v2/source/OssFile#data-type-mapping) Data type mapping is related to the type of file being read, We supported as the following file types: `text` `csv` `parquet` `orc` `json` `excel` `xml` #### JSON File Type[​](https://seatunnel.apache.org/docs/2.3.7/connector-v2/source/OssFile#json-file-type) If you assign file type to `json`, you should also assign schema option to tell connector how to parse data to the row you want. For example: upstream data is the following: ``` {"code": 200, "data": "get success", "success": true} ``` You can also save multiple pieces of data in one file and split them by newline: ``` {"code": 200, "data": "get success", "success": true} {"code": 300, "data": "get failed", "success": false} ``` you should assign schema as the following: ``` schema { fields { code = int data = string success = boolean } } ``` connector will generate data as the following: | code | data | success | | ---- | ----------- | ------- | | 200 | get success | true | #### Text Or CSV File Type[​](https://seatunnel.apache.org/docs/2.3.7/connector-v2/source/OssFile#text-or-csv-file-type) If you assign file type to `text` `csv`, you can choose to specify the schema information or not. For example, upstream data is the following: ``` tyrantlucifer#26#male ``` If you do not assign data schema connector will treat the upstream data as the following: | content | | --------------------- | | tyrantlucifer#26#male | If you assign data schema, you should also assign the option `field_delimiter` too except CSV file type you should assign schema and delimiter as the following: ``` field_delimiter = "#" schema { fields { name = string age = int gender = string } } ``` connector will generate data as the following: | name | age | gender | | ------------- | --- | ------ | | tyrantlucifer | 26 | male | #### Orc File Type[​](https://seatunnel.apache.org/docs/2.3.7/connector-v2/source/OssFile#orc-file-type) If you assign file type to `parquet` `orc`, schema option not required, connector can find the schema of upstream data automatically. | Orc Data type | Nexus Data type | | ------------------------------------ | ---------------------------------------------------------- | | BOOLEAN | BOOLEAN | | INT | INT | | BYTE | BYTE | | SHORT | SHORT | | LONG | LONG | | FLOAT | FLOAT | | DOUBLE | DOUBLE | | BINARY | BINARY | |

STRING
VARCHAR
CHAR

| STRING | | DATE | LOCAL\_DATE\_TYPE | | TIMESTAMP | LOCAL\_DATE\_TIME\_TYPE | | DECIMAL | DECIMAL | | LIST(STRING) | STRING\_ARRAY\_TYPE | | LIST(BOOLEAN) | BOOLEAN\_ARRAY\_TYPE | | LIST(TINYINT) | BYTE\_ARRAY\_TYPE | | LIST(SMALLINT) | SHORT\_ARRAY\_TYPE | | LIST(INT) | INT\_ARRAY\_TYPE | | LIST(BIGINT) | LONG\_ARRAY\_TYPE | | LIST(FLOAT) | FLOAT\_ARRAY\_TYPE | | LIST(DOUBLE) | DOUBLE\_ARRAY\_TYPE | | Map\ | MapType, This type of K and V will transform to Nexus type | | STRUCT | NexusRowType | #### Parquet File Type[​](https://seatunnel.apache.org/docs/2.3.7/connector-v2/source/OssFile#parquet-file-type) If you assign file type to `parquet` `orc`, schema option not required, connector can find the schema of upstream data automatically. | Orc Data type | Nexus Data type | | ----------------------- | ---------------------------------------------------------- | | INT\_8 | BYTE | | INT\_16 | SHORT | | DATE | DATE | | TIMESTAMP\_MILLIS | TIMESTAMP | | INT64 | LONG | | INT96 | TIMESTAMP | | BINARY | BYTES | | FLOAT | FLOAT | | DOUBLE | DOUBLE | | BOOLEAN | BOOLEAN | | FIXED\_LEN\_BYTE\_ARRAY |

TIMESTAMP
DECIMAL

| | DECIMAL | DECIMAL | | LIST(STRING) | STRING\_ARRAY\_TYPE | | LIST(BOOLEAN) | BOOLEAN\_ARRAY\_TYPE | | LIST(TINYINT) | BYTE\_ARRAY\_TYPE | | LIST(SMALLINT) | SHORT\_ARRAY\_TYPE | | LIST(INT) | INT\_ARRAY\_TYPE | | LIST(BIGINT) | LONG\_ARRAY\_TYPE | | LIST(FLOAT) | FLOAT\_ARRAY\_TYPE | | LIST(DOUBLE) | DOUBLE\_ARRAY\_TYPE | | Map\ | MapType, This type of K and V will transform to Nexus type | | STRUCT | NexusRowType | ### Options[​](https://seatunnel.apache.org/docs/2.3.7/connector-v2/source/OssFile#options) | name | type | required | default value | Description | | ---------------------------- | ------- | -------- | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | path | string | yes | - | The Oss path that needs to be read can have sub paths, but the sub paths need to meet certain format requirements. Specific requirements can be referred to "parse\_partition\_from\_path" option | | file\_format\_type | string | yes | - | File type, supported as the following file types: `text` `csv` `parquet` `orc` `json` `excel` `xml` `binary` | | bucket | string | yes | - | The bucket address of oss file system, for example: `oss://nexus-test`. | | endpoint | string | yes | - | fs oss endpoint | | read\_columns | list | no | - | The read column list of the data source, user can use it to implement field projection. The file type supported column projection as the following shown: `text` `csv` `parquet` `orc` `json` `excel` `xml` . If the user wants to use this feature when reading `text` `json` `csv` files, the "schema" option must be configured. | | access\_key | string | no | - | | | access\_secret | string | no | - | | | delimiter | string | no | \001 | Field delimiter, used to tell connector how to slice and dice fields when reading text files. Default `\001`, the same as hive's default delimiter. | | parse\_partition\_from\_path | boolean | no | true | Control whether parse the partition keys and values from file path. For example if you read a file from path `oss://hadoop-cluster/tmp/nexus/parquet/name=tyrantlucifer/age=26`. Every record data from file will be added these two fields: name="tyrantlucifer", age=16 | | date\_format | string | no | yyyy-MM-dd | Date type format, used to tell connector how to convert string to date, supported as the following formats:`yyyy-MM-dd` `yyyy.MM.dd` `yyyy/MM/dd`. default `yyyy-MM-dd` | | datetime\_format | string | no | yyyy-MM-dd HH:mm:ss | Datetime type format, used to tell connector how to convert string to datetime, supported as the following formats:`yyyy-MM-dd HH:mm:ss` `yyyy.MM.dd HH:mm:ss` `yyyy/MM/dd HH:mm:ss` `yyyyMMddHHmmss` | | time\_format | string | no | HH:mm:ss | Time type format, used to tell connector how to convert string to time, supported as the following formats:`HH:mm:ss` `HH:mm:ss.SSS` | | skip\_header\_row\_number | long | no | 0 | Skip the first few lines, but only for the txt and csv. For example, set like following:`skip_header_row_number = 2`. Then Nexus will skip the first 2 lines from source files | | schema | config | no | - | The schema of upstream data. | | sheet\_name | string | no | - | Reader the sheet of the workbook,Only used when file\_format is excel. | | xml\_row\_tag | string | no | - | Specifies the tag name of the data rows within the XML file, only used when file\_format is xml. | | xml\_use\_attr\_format | boolean | no | - | Specifies whether to process data using the tag attribute format, only used when file\_format is xml. | | compress\_codec | string | no | none | Which compress codec the files used. | | encoding | string | no | UTF-8 | | | file\_filter\_pattern | string | no | | `*.txt` means you only need read the files end with `.txt` | | common-options | config | no | - | Source plugin common parameters, please refer to [Source Common Options](/data-integration-with-nexus/nexus-elements/connectors/source/source-common-options.md) for details. | #### compress\_codec \[string][​](https://seatunnel.apache.org/docs/2.3.7/connector-v2/source/OssFile#compress_codec-string) The compress codec of files and the details that supported as the following shown: * txt: `lzo` `none` * json: `lzo` `none` * csv: `lzo` `none` * orc/parquet:\ automatically recognizes the compression type, no additional settings required. #### encoding \[string][​](https://seatunnel.apache.org/docs/2.3.7/connector-v2/source/OssFile#encoding-string) Only used when file\_format\_type is json,text,csv,xml. The encoding of the file to read. This param will be parsed by `Charset.forName(encoding)`. #### file\_filter\_pattern \[string][​](https://seatunnel.apache.org/docs/2.3.7/connector-v2/source/OssFile#file_filter_pattern-string) Filter pattern, which used for filtering files. #### schema \[config][​](https://seatunnel.apache.org/docs/2.3.7/connector-v2/source/OssFile#schema-config) Only need to be configured when the file\_format\_type are text, json, excel, xml or csv ( Or other format we can't read the schema from metadata). **fields \[Config]**[**​**](https://seatunnel.apache.org/docs/2.3.7/connector-v2/source/OssFile#fields-config) The schema of upstream data. ### How to Create a Oss Data Synchronization Jobs[​](https://seatunnel.apache.org/docs/2.3.7/connector-v2/source/OssFile#how-to-create-a-oss-data-synchronization-jobs) The following example demonstrates how to create a data synchronization job that reads data from Oss and prints it on the local client: ``` # Set the basic configuration of the task to be performed env { parallelism = 1 job.mode = "BATCH" } # Create a source to connect to Oss source { OssFile { path = "/nexus/orc" bucket = "oss://tyrantlucifer-image-bed" access_key = "xxxxxxxxxxxxxxxxx" access_secret = "xxxxxxxxxxxxxxxxxxxxxx" endpoint = "oss-cn-beijing.aliyuncs.com" file_format_type = "orc" } } # Console printing of the read Oss data sink { Console { } } ``` ``` # Set the basic configuration of the task to be performed env { parallelism = 1 job.mode = "BATCH" } # Create a source to connect to Oss source { OssFile { path = "/nexus/json" bucket = "oss://tyrantlucifer-image-bed" access_key = "xxxxxxxxxxxxxxxxx" access_secret = "xxxxxxxxxxxxxxxxxxxxxx" endpoint = "oss-cn-beijing.aliyuncs.com" file_format_type = "json" schema { fields { id = int name = string } } } } # Console printing of the read Oss data sink { Console { } } ``` #### Multiple Table[​](https://seatunnel.apache.org/docs/2.3.7/connector-v2/source/OssFile#multiple-table) No need to config schema file type, eg: `orc`. ``` env { parallelism = 1 spark.app.name = "Nexus" spark.executor.instances = 2 spark.executor.cores = 1 spark.executor.memory = "1g" spark.master = local job.mode = "BATCH" } source { OssFile { tables_configs = [ { schema = { table = "fake01" } bucket = "oss://whale-ops" access_key = "xxxxxxxxxxxxxxxxxxx" access_secret = "xxxxxxxxxxxxxxxxxxx" endpoint = "https://oss-accelerate.aliyuncs.com" path = "/test/nexus/read/orc" file_format_type = "orc" }, { schema = { table = "fake02" } bucket = "oss://whale-ops" access_key = "xxxxxxxxxxxxxxxxxxx" access_secret = "xxxxxxxxxxxxxxxxxxx" endpoint = "https://oss-accelerate.aliyuncs.com" path = "/test/nexus/read/orc" file_format_type = "orc" } ] result_table_name = "fake" } } sink { Assert { rules { table-names = ["fake01", "fake02"] } } } ``` Need config schema file type, eg: `json` ``` env { execution.parallelism = 1 spark.app.name = "Nexus" spark.executor.instances = 2 spark.executor.cores = 1 spark.executor.memory = "1g" spark.master = local job.mode = "BATCH" } source { OssFile { tables_configs = [ { bucket = "oss://whale-ops" access_key = "xxxxxxxxxxxxxxxxxxx" access_secret = "xxxxxxxxxxxxxxxxxxx" endpoint = "https://oss-accelerate.aliyuncs.com" path = "/test/nexus/read/json" file_format_type = "json" schema = { table = "fake01" fields { c_map = "map" c_array = "array" c_string = string c_boolean = boolean c_tinyint = tinyint c_smallint = smallint c_int = int c_bigint = bigint c_float = float c_double = double c_bytes = bytes c_date = date c_decimal = "decimal(38, 18)" c_timestamp = timestamp c_row = { C_MAP = "map" C_ARRAY = "array" C_STRING = string C_BOOLEAN = boolean C_TINYINT = tinyint C_SMALLINT = smallint C_INT = int C_BIGINT = bigint C_FLOAT = float C_DOUBLE = double C_BYTES = bytes C_DATE = date C_DECIMAL = "decimal(38, 18)" C_TIMESTAMP = timestamp } } } }, { bucket = "oss://whale-ops" access_key = "xxxxxxxxxxxxxxxxxxx" access_secret = "xxxxxxxxxxxxxxxxxxx" endpoint = "https://oss-accelerate.aliyuncs.com" path = "/test/nexus/read/json" file_format_type = "json" schema = { table = "fake02" fields { c_map = "map" c_array = "array" c_string = string c_boolean = boolean c_tinyint = tinyint c_smallint = smallint c_int = int c_bigint = bigint c_float = float c_double = double c_bytes = bytes c_date = date c_decimal = "decimal(38, 18)" c_timestamp = timestamp c_row = { C_MAP = "map" C_ARRAY = "array" C_STRING = string C_BOOLEAN = boolean C_TINYINT = tinyint C_SMALLINT = smallint C_INT = int C_BIGINT = bigint C_FLOAT = float C_DOUBLE = double C_BYTES = bytes C_DATE = date C_DECIMAL = "decimal(38, 18)" C_TIMESTAMP = timestamp } } } } ] result_table_name = "fake" } } sink { Assert { rules { table-names = ["fake01", "fake02"] } } } ``` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.selfuel.digital/data-integration-with-nexus/nexus-elements/connectors/source/ossfile.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.