Configuring Shift for an external Kafka cluster

The file external-kafka-values.yaml contains the required values to configure Shift against an external Kafka cluster. Sample matching to KOBIL Shift release 0.145.0 is here: external-kafka-values.yaml

Use this file together with your custom-values.yaml when installing or upgrading shift. Provide this file last (rightmost) in the helm command to ensure it has the highest priority.

helm install shift kobil/shift -f custom-values.yaml -f external-kafka-values.yaml

In addition, several topics must be created in the external Kafka cluster. Below is the list of the topics and the required number of partitions.

Some topics require a dedicated config for retention.ms and segment.ms. If these values are not provided, the Kafka default of 604800000 (7 days) can be used.

We generally recommend to configure topics with replication factor 3 and set min.insync.replicas to 2. This requires a Kafka cluster with at least 3 brokers.

  com.kobil.audit:
    partitions: 10

  com.kobil.audit.admin:
    partitions: 10

  com.kobil.ast.architecture.info:
    partitions: 10
    config:
      retention.ms: -1

  com.kobil.ast.version.info:
    partitions: 10
    config:
      retention.ms: -1

  com.kobil.ast.leader.election:
    partitions: 1

  com.kobil.ast.health-check:
    partitions: 1
    config:
      retention.ms: 60000
      segment.ms: 60000

  com.kobil.ast.ca.signer-created:
    partitions: 10

  com.kobil.ast.ca.published:
    partitions: 10

  com.kobil.ast.ca.info:
    partitions: 10

  com.kobil.ast.stream.event:
    partitions: 10
    config:
      retention.ms: 900000
      segment.ms: 900000

  com.kobil.ast.stream.sse:
    partitions: 10
    config:
      retention.ms: 900000
      segment.ms: 900000

  com.kobil.ast.stream.client.status:
    partitions: 10
    config:
      retention.ms: 7200000
      segment.ms: 7200000

  com.kobil.ast.login.events:
    partitions: 10

  com.kobil.ast.management.events:
    partitions: 10

  com.kobil.ast.app.info:
    partitions: 10

  com.kobil.ast.localization.updates:
    partitions: 10

  com.kobil.ast.tms.status:
    partitions: 10

  com.kobil.scp.notifier.push-messages:
    partitions: 10

  com.kobil.scp.notifier.processing-notifications:
    partitions: 10

  com.kobil.scp.notifier.notification-success:
    partitions: 10

  com.kobil.scp.notifier.notification-errors:
    partitions: 10

  com.kobil.scp.notifier.app-errors:
    partitions: 10

  com.kobil.scp.notifier.provider-errors:
    partitions: 10

  com.kobil.payment.create-tenant:
    partitions: 10