BROKERS: kafka_broker_host:broker_port

The host and port identifying the Kafka broker.

TOPIC: kafka_topic

The name of the Kafka topic from which to load data. The topic must exist.

PARTITIONS: (partition_numbers)

A single, a comma-separated list, and/or a range of partition numbers from which GPSS reads messages from the Kafka topic. A range that you specify with the M...N syntax includes both the range start and end values. By default, GPSS reads messages from all partitions of the Kafka topic.

Note: Ensure that you do not configure multiple jobs that specify overlapping partition numbers in the same topic; GPSS behavior is undefined.

FALLBACK_OFFSET: { earliest | latest }

Specifies the behaviour of GPSS when it detects a Kafka message offset gap. When set to earliest, GPSS automatically resumes a load operation from the earliest available published message. When set to latest, GPSS loads only new messages to the Kafka topic. If this property is not set, GPSS returns an error.

avro

When you specify the avro data format, you must define only a single json type column in COLUMNS. If the Kafka message key or value schema is registered in a Confluent Schema Registry, you must also provide the AVRO_OPTION.

binary

When you specify the binary data format, you must define only a single bytea type column in COLUMNS.

csv

When you specify the csv data format, the message content cannot contain line ending characters (CR and LF).

You must not provide a VALUE block when you specify csv format for a KEY block. Similarly, you must not provide a KEY block when you specify csv format for a VALUE block.

custom

When you specify the custom data format, you must provide a CUSTOM_OPTION.

delimited

When you specify the delimited data format, you must provide a DELIMITED_OPTION.

json

When you specify the json data format, you must define only a single json type column in COLUMNS.

SCHEMA_REGISTRY_ADDR: schemareg_host:schemareg_port

When you specify FORMAT: avro and the Avro schema of the JSON data you want to load is registered in the Confluent Schema Registry, you must identify the host name and port number of each Confluent Schema Registry server in your Kafka cluster. You may specify more than one address, and at least one of the addresses must be legal.

SCHEMA_CA_ON_GPDB: sr_ca_file_path

The file system path to the CA certificate that GPSS uses to verify the peer. This file must reside in sr_ca_file_path on all Greenplum Database segment hosts.

SCHEMA_CERT_ON_GPDB: sr_cert_file_path

The file system path to the client certificate that GPSS uses to connect to the HTTPS schema registry. This file must reside in sr_cert_file_path on all Greenplum Database segment hosts.

SCHEMA_KEY_ON_GPDB: sr_key_file_path

The file system path to the private key file that GPSS uses to connect to the HTTPS schema registry. This file must reside in sr_key_file_path on all Greenplum Database segment hosts.

SCHEMA_MIN_TLS_VERSION: minimum_version

The minimum transport layer security (TLS) version that GPSS requests on the connection to the schema registry. Supported versions are 1.0, 1.1, 1.2, or 1.3. The default minimum TLS version is 1.0.

SCHEMA_PATH_ON_GPDB: path_to_file

When you specify the avro format and the Avro schema of the JSON key or value data that you want to load is specified in a separate .avsc file, you must identify the file system location in path_to_file, and the file must reside in this location on every Greenplum Database segment host.

Note: GPSS does not cache the schema. GPSS must reload the schema for every batch of Kafka data. Also, GPSS supports providing the schema for either the key or the value, but not both.

BYTES_TO_BASE64: boolean

When true, GPSS converts Avro bytes fields into base64-encoded strings. The default value is false, GPSS does not perform the conversion.

NAME: udf_name

The name of the custom formatter user-defined function.

PARAMSTR: udf_parameter_string

A string specifying the comma-separated list of arguments to pass to the custom formatter user-defined function.

• topic - text
• partition - int
• offset - bigint

• columns are listed in UPDATE_COLUMNS,
• the MATCH_COLUMNS target table column values are equal to the input data, and
• an optional UPDATE_CONDITION is specified and met.

• the MATCH_COLUMNS target table column values are equal to the input data, and
• an optional DELETE_CONDITION is specified and met.

match_column_name

Specifies the column(s) to use as the join condition for the update. The attribute value in the specified target column(s) must be equal to that of the corresponding source data column(s) in order for the row to be updated in the target table.

order_column_name

Specify the column(s) by which GPSS sorts the rows. When multiple matching rows exist in a batch, ORDER_COLUMNS is used with MATCH_COLUMNS to determine the input row with the largest value; GPSS uses that row to write/update the target.

update_column_name

Specifies the column(s) to update for the rows that meet the MATCH_COLUMNS criteria and the optional UPDATE_CONDITION.

NAME: target_column_name

Specifies the target Greenplum Database table column name.

EXPRESSION: { source_column_name | expression }

Specifies a Kafka COLUMNS:NAME (source_column_name) or an expression. When you specify an expression, you may provide a value expression that you would specify in the SELECT list of a query, such as a constant value, a column reference, an operator invocation, a built-in or user-defined function call, and so on.

target_column_name: { source_column_name | expression }

When you use this MAPPING syntax, specify the target_column_name and {source_column_name | expression} as described above.

SAVE_FAILING_BATCH: boolean

Determines whether or not GPSS saves data into a backup table before it writes the data to Greenplum Database. Saving the data in this manner aids recovery when GPSS encounters errors during the evaluation of expressions. The default is false; GPSS does not use a backup table, and returns immediately when it encounters an expression error. When you set this property to true, GPSS writes both the good and the bad data in the batch to a backup table named gpssbackup_jobhash, and continues to process incoming Kafka messages. You must then manually load the good data from the backup table into Greenplum or set RECOVER_FAILING_BATCH (Beta) to true to have GPSS automatically reload the good data.

Note: Using a backup table to hedge against mapping errors may impact performance, especially when the data that you are loading has not been cleaned.

RECOVER_FAILING_BATCH: boolean (Beta)

When set to true and SAVE_FAILING_BATCH is also true, GPSS automatically reloads the good data in the batch and retains only the error data in the backup table. The default value is false; GPSS does not process the backup table.

Note: Enabling this property requires that GPSS has the Greenplum Database privileges to create a function.

MAX_ROW: number_of_rows

The number of rows to batch before triggering an INSERT operation on the Greenplum Database table. The default value of MAX_ROW is 0, which instructs gpkafka to ignore this commit trigger condition.

MINIMAL_INTERVAL: wait_time

The minimum amount of time to wait (milliseconds) between each INSERT operation on the table. The default value is 0, wait forever.

CONSISTENCY: consistency_value

Specify how GPSS should manage message offsets when it acts as a high-level consumer. Valid values are strong, at-least, at-most, and none. The default value is strong. Refer to Understanding Kafka Message Offset Management for more detailed information.

IDLE_DURATION: idle_time

The maximum amount of time to wait (milliseconds) for the first batch of Kafka data. When you use this property to enable lazy load, GPSS waits until Kafka data is available before locking the target Greenplum table. You can specify:

• 0 (lazy load is disabled)
• -1 (lazy load is enabled, the job never stops), or
• a positive value (lazy load is enabled, the job stops after idle_time duration of no data in the Kafka topic)

The default value is 0.

Controls the polling time period and batch size when reading Kafka data.

BATCHSIZE: num_records

The number of Kafka records in a batch. BATCHSIZE should be smaller than COMMIT:MAX_ROW. The default batch size is 200.

TIMEOUT: poll_time

The maximum time, in milliseconds, to wait in a polling cycle if Kafka data is not available. You must specify a TIMEOUT greater than 100 milliseconds and less than COMMIT:MINIMAL_INTERVAL. The default poll timeout is 1000 milliseconds.

POST_BATCH_SQL: udf_or_sql_to_run

The user-defined function or SQL command(s) that you want to run after the specified number of batches are read from Kafka. The default is null.

BATCH_INTERVAL: num_batches

The number of batches to read before executing udf_or_sql_to_run. The default batch interval is 0.

kafka_property_name

The name of a Kafka property.

kafka_property_value

The Kafka property value.

RETRY_INTERVAL: retry_time

The period of time that gpkafka waits before retrying the job. You can specify the time interval in day (d), hour (h), minute (m), second (s), or millisecond (ms) integer units; do not mix units. The default retry interval is 5m (5 minutes).

MAX_RETRIES: num_retries

The maximum number of times gpkafka attempts to retry the job. The default is 0, do not retry. If you specify a negative value, gpkafka retries the job indefinitely.