Skip to main content

Strimzi Kafka Operator on Openshift

The KOBIL Shift-Operator makes use of the Strimizi Kafka Operator to create Kafka-services in the targeted KOBIL Shift namespace (or OpenShift project) - at time of deployment for the KOBIL Shift. Please check for the KOBIL Shift ReadMe file for release/version compatibility for the Strimzi Operator release. The Kafka settings are part of the KOBIL Shift deployment metaconfiguration file in the section "strimzi".

Note: When running a cluster-environment do not share Kafka-services for multiple KOBIL Shift deployments. You have to provide single Kafka services per targeted KOBIL Shift namespace.

Once Strimzi Kafka Operator is running and initial KOBIL Shift deployment is successfully done, you will find kafka-cluster-entity-operator, kafka-cluster-zookeeper and at least one kafka service worker pod in the KOBIL Shift namespace or project. Also the kafkas and kafkatopics are created during deployment. In addition find a "kafka" KOBIL CRD "servicegroup" object for the Kafka services configuration via KOBIL Shift charts and Operator logic.

When using OpenShift platform you may use the RedHat OpenShift Strimzi Operator. The Strimzi Operator is then instrumented by the KOBIL Shift Operator to create the Kafka srevice in the KOBIL Shift namespace or project.

Once the KOBIL Shift-Operator had processed the KOBIL Shift deployment (helm install shift ..) you will find in the created project the kafka resources. To verify this run:

kubectl get kafkas
kubectl get kafkatopics

Find section "strimzi" in KOBIL Shift metaconfiguration "values.yaml" - here to configure for the available profiles "basic" and "tuned" (also having the option to define by custom profile "custom" an individual setup):

KOBIL Shift metaconfig section "strimzi"

# Configuration of the Kafka custom resources. Requires [Strimzi Kafka operator](https://strimzi.io/)
strimzi:
  # -- Enable/disable deployment of Kafka custom resources.
  enabled: true

  # -- Storage configuration for Kafka and Zookeeper.
  storage:
    # -- Size of persistent volumes for Kafka and Zookeeper.
    size:
      kafka: 20Gi
      zookeeper: 5Gi

  # -- Storage class to use for the persistent volumes. Default ~ uses the default k8ss storage class.
  class:
    kafka: ~
    zookeeper: ~

  sizing:
  # -- Configure seizing for Kafka cluster. Supported values are 'basic', 'tuned',
  # and 'custom'. When using mode 'custom', values `.custom.kafka` and `custom.zookeeper`
  # must be provided. See README.md for details.
  # Note: Changing the sizing mode after deployment is highly discouraged, as it effects the topic replica
  # count and partition assignment to nodes. It can even lead to data loss.
  mode: "basic"
  custom:
    kafka: ~
    zookeeper: ~

# -- Configuration of additional Kafkatopic resource, c.f. https://strimzi.io/docs/operators/latest/configuring.html#ref-operator-topic-str

additionalTopics:

  # -- Name of the topic. Used for both the internal topic name and the Kafkatopic resource name.
  # Must comply with kubernetes resource naming conventions (DNS subdomain names, RFC 1123)
  # https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#dns-subdomain-names
  # and Kafka topic naming conventions
  # https://strimzi.io/docs/operators/latest/configuring.html#kafka_topic_naming_conventions.
  # The name must
  # * not be longer than 249 characters.
  # * contain only lowercase alphanumeric characters, \'-\' or \'.\'
  # * start and end with an alphanumeric character
  #topicname:
    # -- Configure partitions for this topic. When omitted,
    # partitions are set to the default value (\'num.partitions\')
    # configured for the broker: 1.
    #partitions: ~

    # -- Configure replicas for this topic. When omitted,
    # replicas are set to the default value 'default.replication.factor'
    # configured for the broker: 1 for performance mode 'basic' and 3 for
    # performance mode tuned.
    #replicas: ~

    # -- Advanced topic configuration. When omitted,
    # the default configuration of the broker is used, which corresponds to
    # 'retention.ms: 604800000' and 'segment.bytes: 1073741824'.
    # See https://docs.confluent.io/platform/current/installation/configuration/topic-configs.html# for valid options
    #config: {}

Find from the KOBIL Chart Shift README.MD more details for the "Kafka Sizing"

Sizing of the Kafka cluster deployed using Strimzi Kafka Operator can be configured using value strimzi.sizing.mode. Supported values are basic, tuned, custom.

When using mode custom, values strimzi.sizing.custom.kafka and strimzi.sizing.custom.zookeeper must be specified. See also documentation on sizing and configuration

Note: Changing the sizing mode after deployment is highly discouraged, as it effects the topic replica count and partition assignment to nodes. It can even lead to data loss.

  • Performance mode basic corresponds to configuration
strimzi:
  sizing:
    mode: "basic"
    custom:
      kafka:
        replicas: 1
        resources:
          requests:
            memory: 2Gi
            cpu: "100m"
          limits:
            memory: 2Gi
        jvmOptions:
          -Xms: 1024m
          -Xmx: 1024m
        config:
          auto.create.topics.enable: "false"
          delete.topic.enable: "false"
          log.message.format.version: 2.8
          inter.broker.protocol.version: 2.8
          default.replication.factor: 1
          min.insync.replicas: 1
          offsets.topic.replication.factor: 1
          transaction.state.log.replication.factor: 1
          transaction.state.log.min.isr: 1
      zookeeper:
        replicas: 1
        resources:
          requests:
            memory: 768Mi
            cpu: "50m"
          limits:
            memory: 768Mi
       jvmOptions:
         -Xms: 512m
         -Xmx: 512m
  • Performance mode tuned corresponds to configuration
strimzi:
  sizing:
    mode: "tuned"
    custom:
      kafka:
        replicas: 3
        resources:
          requests:
            memory: 8Gi
            cpu: "2"
          limits:
            memory: 8Gi
        jvmOptions:
          -Xms: 4096m
          -Xmx: 4096m
        config:
          auto.create.topics.enable: "false"
          delete.topic.enable: "false"
          log.message.format.version: 2.8
          inter.broker.protocol.version: 2.8
          default.replication.factor: 3
          min.insync.replicas: 2
          offsets.topic.replication.factor: 3
          transaction.state.log.replication.factor: 3
          transaction.state.log.min.isr: 2
      zookeeper:
        replicas: 3
        resources:
          requests:
            memory: 1536Mi
            cpu: "1"
          limits:
            memory: 1536Mi
          jvmOptions:
            -Xms: 1024m
           -Xmx: 1024m

Related commands:

  oc get servicegroups
  oc describte servicegroup <kafka-servicegroup-name>
  oc get kafka
  oc get kafkatopics