Kafka¶

By enabling this toolset, HolmesGPT will be able to fetch metadata from Kafka. This provides Holmes the ability to introspect into Kafka by listing consumers and topics or finding lagging consumer groups.

This toolset uses the AdminClient of the confluent-kafka python library. Kafka's Java API is also a good source of documentation.

Configuration¶

SASL authentication¶

SASL_PLAINTEXT encrypts nothing between Holmes and the broker: the username and all data travel in the clear, and a captured SCRAM exchange is open to an offline dictionary attack on the password (with PLAIN, the password itself is sent). Use it only on a network you trust; otherwise use SASL_SSL, as in SASL+TLS.

Holmes CLIHolmes Helm ChartRobusta Helm Chart

Set the environment variables:

export KAFKA_USERNAME=kafka-plaintext-user
export KAFKA_PASSWORD=your-kafka-password

Add the following to ~/.holmes/config.yaml. Create the file if it doesn't exist:

toolsets:
  kafka/admin:
    enabled: true
    config:
      clusters:
        - name: prod-kafka
          broker: kafka.prod.example.com:9095
          username: "{{ env.KAFKA_USERNAME }}"
          password: "{{ env.KAFKA_PASSWORD }}"
          sasl_mechanism: SCRAM-SHA-512
          security_protocol: SASL_PLAINTEXT

After making changes to your configuration, run:

holmes toolset refresh

Create a Kubernetes secret in the namespace Holmes runs in:

kubectl create secret generic holmes-kafka \
  --from-literal=KAFKA_USERNAME=kafka-plaintext-user \
  --from-literal=KAFKA_PASSWORD=your-kafka-password \
  -n <namespace>

When using the standalone Holmes Helm Chart, merge this into your values.yaml, appending list entries (such as additionalEnvVars) to the lists already there:

extraEnvVarsSecrets:
  - holmes-kafka

toolsets:
  kafka/admin:
    enabled: true
    config:
      clusters:
        - name: prod-kafka
          broker: kafka.prod.example.com:9095
          username: "{{ env.KAFKA_USERNAME }}"
          password: "{{ env.KAFKA_PASSWORD }}"
          sasl_mechanism: SCRAM-SHA-512
          security_protocol: SASL_PLAINTEXT

Apply the configuration:

helm upgrade holmes robusta/holmes -f values.yaml

Create a Kubernetes secret in the namespace Holmes runs in:

kubectl create secret generic holmes-kafka \
  --from-literal=KAFKA_USERNAME=kafka-plaintext-user \
  --from-literal=KAFKA_PASSWORD=your-kafka-password \
  -n <namespace>

When using the Robusta Helm Chart (which includes HolmesGPT), merge the keys under holmes: into the holmes: key of your generated_values.yaml (add one if it has none), appending list entries (such as additionalEnvVars) to the lists already there:

holmes:
  extraEnvVarsSecrets:
    - holmes-kafka

  toolsets:
    kafka/admin:
      enabled: true
      config:
        clusters:
          - name: prod-kafka
            broker: kafka.prod.example.com:9095
            username: "{{ env.KAFKA_USERNAME }}"
            password: "{{ env.KAFKA_PASSWORD }}"
            sasl_mechanism: SCRAM-SHA-512
            security_protocol: SASL_PLAINTEXT

Apply the configuration:

helm upgrade robusta robusta/robusta -f generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

mTLS — certificate files (Kubernetes mounted secrets)¶

Use this approach when the certificates are files: on your machine with the CLI, or in Kubernetes, mounted into the Holmes pod from a secret. With the CLI, the /path/to/ paths in the config are paths on the machine that runs the CLI. With the Helm charts, the config paths are inside the Holmes pod (the secret's mount path), and the /path/to/ paths in kubectl create secret are on the machine where you run kubectl.

Holmes CLIHolmes Helm ChartRobusta Helm Chart

Add the following to ~/.holmes/config.yaml. Create the file if it doesn't exist:

toolsets:
  kafka/admin:
    enabled: true
    config:
      clusters:
        - name: prod-kafka
          broker: kafka.prod.example.com:9093
          security_protocol: SSL
          ssl_ca_cert_path: /path/to/ca.crt
          ssl_client_cert_path: /path/to/client.pem
          ssl_client_key_path: /path/to/client.key

After making changes to your configuration, run:

holmes toolset refresh

Create a Kubernetes secret in the namespace Holmes runs in:

kubectl create secret generic holmes-kafka-tls \
  --from-file=ca.crt=/path/to/ca.crt \
  --from-file=client.pem=/path/to/client.pem \
  --from-file=client.key=/path/to/client.key \
  -n <namespace>

When using the standalone Holmes Helm Chart, merge this into your values.yaml, appending list entries (such as additionalEnvVars) to the lists already there:

additionalVolumes:
  - name: kafka-tls
    secret:
      secretName: holmes-kafka-tls

additionalVolumeMounts:
  - name: kafka-tls
    mountPath: /etc/kafka-tls
    readOnly: true

toolsets:
  kafka/admin:
    enabled: true
    config:
      clusters:
        - name: prod-kafka
          broker: kafka.prod.example.com:9093
          security_protocol: SSL
          ssl_ca_cert_path: /etc/kafka-tls/ca.crt
          ssl_client_cert_path: /etc/kafka-tls/client.pem
          ssl_client_key_path: /etc/kafka-tls/client.key

Apply the configuration:

helm upgrade holmes robusta/holmes -f values.yaml

Create a Kubernetes secret in the namespace Holmes runs in:

kubectl create secret generic holmes-kafka-tls \
  --from-file=ca.crt=/path/to/ca.crt \
  --from-file=client.pem=/path/to/client.pem \
  --from-file=client.key=/path/to/client.key \
  -n <namespace>

When using the Robusta Helm Chart (which includes HolmesGPT), merge the keys under holmes: into the holmes: key of your generated_values.yaml (add one if it has none), appending list entries (such as additionalEnvVars) to the lists already there:

holmes:
  additionalVolumes:
    - name: kafka-tls
      secret:
        secretName: holmes-kafka-tls

  additionalVolumeMounts:
    - name: kafka-tls
      mountPath: /etc/kafka-tls
      readOnly: true

  toolsets:
    kafka/admin:
      enabled: true
      config:
        clusters:
          - name: prod-kafka
            broker: kafka.prod.example.com:9093
            security_protocol: SSL
            ssl_ca_cert_path: /etc/kafka-tls/ca.crt
            ssl_client_cert_path: /etc/kafka-tls/client.pem
            ssl_client_key_path: /etc/kafka-tls/client.key

Apply the configuration:

helm upgrade robusta robusta/robusta -f generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

SASL+TLS (`SASL_SSL`)¶

For SASL+TLS (SASL_SSL) add SASL credentials alongside the cert paths:

In Kubernetes, this reuses the holmes-kafka secret created in the SASL authentication section above.

In Kubernetes, this also reuses the holmes-kafka-tls secret created in the mTLS — certificate files (Kubernetes mounted secrets) section above.

Holmes CLIHolmes Helm ChartRobusta Helm Chart

Set the environment variables:

export KAFKA_USERNAME=kafka-plaintext-user
export KAFKA_PASSWORD=your-kafka-password

Add the following to ~/.holmes/config.yaml. Create the file if it doesn't exist:

toolsets:
  kafka/admin:
    enabled: true
    config:
      clusters:
        - name: prod-kafka
          broker: kafka.prod.example.com:9093
          security_protocol: SASL_SSL
          sasl_mechanism: SCRAM-SHA-512
          username: "{{ env.KAFKA_USERNAME }}"
          password: "{{ env.KAFKA_PASSWORD }}"
          ssl_ca_cert_path: /path/to/ca.crt
          ssl_client_cert_path: /path/to/client.pem
          ssl_client_key_path: /path/to/client.key

After making changes to your configuration, run:

holmes toolset refresh

When using the standalone Holmes Helm Chart, merge this into your values.yaml, appending list entries (such as additionalEnvVars) to the lists already there:

extraEnvVarsSecrets:
  - holmes-kafka

additionalVolumes:
  - name: kafka-tls
    secret:
      secretName: holmes-kafka-tls

additionalVolumeMounts:
  - name: kafka-tls
    mountPath: /etc/kafka-tls
    readOnly: true

toolsets:
  kafka/admin:
    enabled: true
    config:
      clusters:
        - name: prod-kafka
          broker: kafka.prod.example.com:9093
          security_protocol: SASL_SSL
          sasl_mechanism: SCRAM-SHA-512
          username: "{{ env.KAFKA_USERNAME }}"
          password: "{{ env.KAFKA_PASSWORD }}"
          ssl_ca_cert_path: /etc/kafka-tls/ca.crt
          ssl_client_cert_path: /etc/kafka-tls/client.pem
          ssl_client_key_path: /etc/kafka-tls/client.key

Apply the configuration:

helm upgrade holmes robusta/holmes -f values.yaml

When using the Robusta Helm Chart (which includes HolmesGPT), merge the keys under holmes: into the holmes: key of your generated_values.yaml (add one if it has none), appending list entries (such as additionalEnvVars) to the lists already there:

holmes:
  extraEnvVarsSecrets:
    - holmes-kafka

  additionalVolumes:
    - name: kafka-tls
      secret:
        secretName: holmes-kafka-tls

  additionalVolumeMounts:
    - name: kafka-tls
      mountPath: /etc/kafka-tls
      readOnly: true

  toolsets:
    kafka/admin:
      enabled: true
      config:
        clusters:
          - name: prod-kafka
            broker: kafka.prod.example.com:9093
            security_protocol: SASL_SSL
            sasl_mechanism: SCRAM-SHA-512
            username: "{{ env.KAFKA_USERNAME }}"
            password: "{{ env.KAFKA_PASSWORD }}"
            ssl_ca_cert_path: /etc/kafka-tls/ca.crt
            ssl_client_cert_path: /etc/kafka-tls/client.pem
            ssl_client_key_path: /etc/kafka-tls/client.key

Apply the configuration:

helm upgrade robusta robusta/robusta -f generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

mTLS — base64-encoded inline certificates¶

Use this approach when certificates are passed as environment variables (e.g., from a secret manager or Kubernetes secret).

Holmes CLIHolmes Helm ChartRobusta Helm Chart

Set the environment variables:

export KAFKA_CA_CERT_BASE64="$(base64 < /path/to/ca.crt | tr -d '\n')"
export KAFKA_CLIENT_CERT_BASE64="$(base64 < /path/to/client.pem | tr -d '\n')"
export KAFKA_CLIENT_KEY_BASE64="$(base64 < /path/to/client.key | tr -d '\n')"

Add the following to ~/.holmes/config.yaml. Create the file if it doesn't exist:

toolsets:
  kafka/admin:
    enabled: true
    config:
      clusters:
        - name: prod-kafka
          broker: kafka.prod.example.com:9093
          security_protocol: SSL
          ssl_ca_cert: "{{ env.KAFKA_CA_CERT_BASE64 }}"
          ssl_client_cert: "{{ env.KAFKA_CLIENT_CERT_BASE64 }}"
          ssl_client_key: "{{ env.KAFKA_CLIENT_KEY_BASE64 }}"

After making changes to your configuration, run:

holmes toolset refresh

Create a Kubernetes secret in the namespace Holmes runs in:

kubectl create secret generic holmes-kafka-tls-base64 \
  --from-literal=KAFKA_CA_CERT_BASE64="$(base64 < /path/to/ca.crt | tr -d '\n')" \
  --from-literal=KAFKA_CLIENT_CERT_BASE64="$(base64 < /path/to/client.pem | tr -d '\n')" \
  --from-literal=KAFKA_CLIENT_KEY_BASE64="$(base64 < /path/to/client.key | tr -d '\n')" \
  -n <namespace>

When using the standalone Holmes Helm Chart, merge this into your values.yaml, appending list entries (such as additionalEnvVars) to the lists already there:

extraEnvVarsSecrets:
  - holmes-kafka-tls-base64

toolsets:
  kafka/admin:
    enabled: true
    config:
      clusters:
        - name: prod-kafka
          broker: kafka.prod.example.com:9093
          security_protocol: SSL
          ssl_ca_cert: "{{ env.KAFKA_CA_CERT_BASE64 }}"
          ssl_client_cert: "{{ env.KAFKA_CLIENT_CERT_BASE64 }}"
          ssl_client_key: "{{ env.KAFKA_CLIENT_KEY_BASE64 }}"

Apply the configuration:

helm upgrade holmes robusta/holmes -f values.yaml

Create a Kubernetes secret in the namespace Holmes runs in:

kubectl create secret generic holmes-kafka-tls-base64 \
  --from-literal=KAFKA_CA_CERT_BASE64="$(base64 < /path/to/ca.crt | tr -d '\n')" \
  --from-literal=KAFKA_CLIENT_CERT_BASE64="$(base64 < /path/to/client.pem | tr -d '\n')" \
  --from-literal=KAFKA_CLIENT_KEY_BASE64="$(base64 < /path/to/client.key | tr -d '\n')" \
  -n <namespace>

When using the Robusta Helm Chart (which includes HolmesGPT), merge the keys under holmes: into the holmes: key of your generated_values.yaml (add one if it has none), appending list entries (such as additionalEnvVars) to the lists already there:

holmes:
  extraEnvVarsSecrets:
    - holmes-kafka-tls-base64

  toolsets:
    kafka/admin:
      enabled: true
      config:
        clusters:
          - name: prod-kafka
            broker: kafka.prod.example.com:9093
            security_protocol: SSL
            ssl_ca_cert: "{{ env.KAFKA_CA_CERT_BASE64 }}"
            ssl_client_cert: "{{ env.KAFKA_CLIENT_CERT_BASE64 }}"
            ssl_client_key: "{{ env.KAFKA_CLIENT_KEY_BASE64 }}"

Apply the configuration:

helm upgrade robusta robusta/robusta -f generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

Configuration fields¶

Below is a description of the configuration fields for each cluster entry:

Config key	Required	Description
`name`	Yes	Unique name for this cluster. Holmes uses it to decide which cluster to query.
`broker`	Yes	Comma-separated list of `host:port` pairs for the initial broker connection.
`security_protocol`	No	Security protocol: `PLAINTEXT`, `SSL`, `SASL_PLAINTEXT`, or `SASL_SSL`.
`sasl_mechanism`	No	SASL mechanism: `PLAIN`, `SCRAM-SHA-256`, or `SCRAM-SHA-512`.
`username`	No	Username for SASL authentication.
`password`	No	Password for SASL authentication.
`client_id`	No	Kafka client ID (default: `holmes-kafka-client`).
`ssl_ca_cert_path`	No	Path to the CA certificate file (PEM). Use when certs are mounted as Kubernetes secrets.
`ssl_client_cert_path`	No	Path to the client certificate file (PEM) for mTLS.
`ssl_client_key_path`	No	Path to the client private key file (PEM) for mTLS.
`ssl_ca_cert`	No	Base64-encoded CA certificate (PEM). Alternative to `ssl_ca_cert_path`.
`ssl_client_cert`	No	Base64-encoded client certificate (PEM) for mTLS. Alternative to `ssl_client_cert_path`.
`ssl_client_key`	No	Base64-encoded client private key (PEM) for mTLS. Alternative to `ssl_client_key_path`.

When both a path field and its inline base64 counterpart are set, the path field takes precedence.

Capabilities¶

The table below describes the specific capabilities provided by this toolset. HolmesGPT can decide to invoke any of these capabilities when answering questions or investigating issues.

Tool Name	Description
kafka_list_topics	List all Kafka topics
kafka_describe_topic	Get detailed information about a specific topic
kafka_list_consumers	List all consumer groups
kafka_describe_consumer	Get detailed information about a consumer group
kafka_consumer_lag	Check consumer lag for a consumer group

Kafka¶

Configuration¶

SASL authentication¶

mTLS — certificate files (Kubernetes mounted secrets)¶

SASL+TLS (SASL_SSL)¶

mTLS — base64-encoded inline certificates¶

Configuration fields¶

Capabilities¶

SASL+TLS (`SASL_SSL`)¶