Kafka¶

By enabling this toolset, HolmesGPT will be able to fetch metadata from Kafka. This provides Holmes the ability to introspect into Kafka by listing consumers and topics or finding lagging consumer groups.

This toolset uses the AdminClient of the confluent-kafka python library. Kafka's Java API is also a good source of documentation.

Configuration¶

Holmes CLI Holmes Helm Chart Robusta Helm Chart

Add the following to ~/.holmes/config.yaml. Create the file if it doesn't exist:

toolsets:
    kafka/admin:
        enabled: true
        config:
            clusters:
                - name: aks-prod-kafka
                  broker: kafka-1.aks-prod-kafka-brokers.kafka.svc:9095
                  username: kafka-plaintext-user
                  password: ******
                  sasl_mechanism: SCRAM-SHA-512
                  security_protocol: SASL_PLAINTEXT
                - name: gke-stg-kafka
                  broker: gke-kafka.gke-stg-kafka-brokers.kafka.svc:9095
                  username: kafka-plaintext-user
                  password: ****
                  sasl_mechanism: SCRAM-SHA-512
                  security_protocol: SASL_PLAINTEXT

When using the standalone Holmes Helm Chart, update your values.yaml:

toolsets:
    kafka/admin:
        enabled: true
        config:
            clusters:
                - name: aks-prod-kafka
                  broker: kafka-1.aks-prod-kafka-brokers.kafka.svc:9095
                  username: kafka-plaintext-user
                  password: ******
                  sasl_mechanism: SCRAM-SHA-512
                  security_protocol: SASL_PLAINTEXT
                - name: gke-stg-kafka
                  broker: gke-kafka.gke-stg-kafka-brokers.kafka.svc:9095
                  username: kafka-plaintext-user
                  password: ****
                  sasl_mechanism: SCRAM-SHA-512
                  security_protocol: SASL_PLAINTEXT

Apply the configuration:

helm upgrade holmes holmes/holmes --values=values.yaml

When using the Robusta Helm Chart (which includes HolmesGPT), update your generated_values.yaml:

holmes:
  toolsets:
      kafka/admin:
          enabled: true
          config:
              clusters:
                  - name: aks-prod-kafka
                    broker: kafka-1.aks-prod-kafka-brokers.kafka.svc:9095
                    username: kafka-plaintext-user
                    password: ******
                    sasl_mechanism: SCRAM-SHA-512
                    security_protocol: SASL_PLAINTEXT
                  - name: gke-stg-kafka
                    broker: gke-kafka.gke-stg-kafka-brokers.kafka.svc:9095
                    username: kafka-plaintext-user
                    password: ****
                    sasl_mechanism: SCRAM-SHA-512
                    security_protocol: SASL_PLAINTEXT

Apply the configuration:

helm upgrade robusta robusta/robusta --values=generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

Below is a description of the configuration field for each cluster:

Config key	Description
name	Give a meaningful name to your cluster. Holmes will use it to decide what cluster to look into. Names must be unique across all clusters.
broker	List of host/port pairs to use for establishing the initial connection to the Kafka cluster
username	Username for SASL authentication
password	Password for SASL authentication
sasl_mechanism	SASL mechanism (e.g., SCRAM-SHA-512)
security_protocol	Security protocol (e.g., SASL_PLAINTEXT)

Capabilities¶

The table below describes the specific capabilities provided by this toolset. HolmesGPT can decide to invoke any of these capabilities when answering questions or investigating issues.

Tool Name	Description
kafka_list_topics	List all Kafka topics
kafka_describe_topic	Get detailed information about a specific topic
kafka_list_consumers	List all consumer groups
kafka_describe_consumer	Get detailed information about a consumer group
kafka_consumer_lag	Check consumer lag for a consumer group