Kafka Destination Plugin

This destination plugin lets you sync data from a CloudQuery source to Kafka in various formats such as CSV, JSON. Each table will be pushed to a separate topic.

Example

This example configures connects to a local Kafka destination with no authentication and pushes messages in JSON format. Note that the plugin only supports append write-mode.

The (top level) spec section is described in the Destination Spec Reference.

kind: destination
spec:
  name: "kafka"
  path: "cloudquery/kafka"
  version: "v2.0.0"
  write_mode: "append" # only supports 'append'

  spec:
    brokers: ["<broker-host>:<broker-port>"]
    format: "json"

Plugin Spec

This is the (nested) plugin spec

• brokers (list(string)) (required)

  list of brokers to connect to.

• sasl_username (string) (optional)

  If connecting via SASL/PLAIN, the username to use.

• sasl_password (string) (optional)

  If connecting via SASL/PLAIN, the password to use.

• verbose (bool) (optional)

  If true, the plugin will log all underlying Kafka client messages to the log.

• format (string) (required)

  Format of the output file. json and csv are supported.

• format_spec (map format_spec) (optional) Optional parameters to change the format of the file

format_spec

• delimiter (string) (optional) (default: ,)

  Character that will be used as want to use as the delimiter if the format type is csv

• skip_header (bool) (optional) (default: false)

  Specifies if the first line of a file should be the headers (when format is csv).