Grafana Dashboards¶
Connect HolmesGPT to Grafana for dashboard analysis, visual rendering, query extraction, and understanding your monitoring setup. When the Grafana Image Renderer is installed, HolmesGPT can visually render dashboards and panels to detect anomalies like spikes, trends, and outliers.
Prerequisites¶
A Grafana service account token with the following permissions:
- Basic role → Viewer
For visual rendering, the Grafana Image Renderer plugin must be installed on your Grafana instance and enable_rendering: true must be set in the config. HolmesGPT auto-detects the renderer — if it's not installed, visual rendering tools are simply not registered and everything else works normally.
Configuration¶
Add the following to ~/.holmes/config.yaml. Create the file if it doesn't exist:
toolsets:
grafana/dashboards:
enabled: true
config:
api_key: <your grafana service account token>
api_url: <your grafana url> # e.g. https://acme-corp.grafana.net or http://localhost:3000
# Optional: Additional headers for all requests
# additional_headers:
# X-Custom-Header: "custom-value"
After making changes to your configuration, run:
To test, run:
First, create a Kubernetes secret with your Grafana service account token:
kubectl create secret generic grafana-api-key \
--from-literal=api-key=your-grafana-service-account-token \
-n holmes
Namespace must match Holmes' deployment
Create the secret in the same namespace where Holmes runs. The -n holmes flag in the Holmes Helm tab and -n default in the Robusta Helm tab match each chart's documented defaults — adjust if you installed Holmes/Robusta into a different namespace. A secret in the wrong namespace silently resolves to an empty env var and authentication will fail with no clear error.
Then add to your Holmes Helm values:
additionalEnvVars:
- name: GRAFANA_API_KEY
valueFrom:
secretKeyRef:
name: grafana-api-key
key: api-key
toolsets:
grafana/dashboards:
enabled: true
config:
api_key: "{{ env.GRAFANA_API_KEY }}"
api_url: <your grafana url> # e.g. https://acme-corp.grafana.net
# Optional: Additional headers for all requests
# additional_headers:
# X-Custom-Header: "custom-value"
First, create a Kubernetes secret with your Grafana service account token:
kubectl create secret generic grafana-api-key \
--from-literal=api-key=your-grafana-service-account-token \
-n default
Namespace must match Holmes' deployment
Create the secret in the same namespace where Holmes runs. The -n holmes flag in the Holmes Helm tab and -n default in the Robusta Helm tab match each chart's documented defaults — adjust if you installed Holmes/Robusta into a different namespace. A secret in the wrong namespace silently resolves to an empty env var and authentication will fail with no clear error.
Then add to your Robusta Helm values:
holmes:
additionalEnvVars:
- name: GRAFANA_API_KEY
valueFrom:
secretKeyRef:
name: grafana-api-key
key: api-key
toolsets:
grafana/dashboards:
enabled: true
config:
api_key: "{{ env.GRAFANA_API_KEY }}"
api_url: <your grafana url> # e.g. https://acme-corp.grafana.net
# Optional: Additional headers for all requests
# additional_headers:
# X-Custom-Header: "custom-value"
Update your Helm values and run a Helm upgrade:
Multiple Grafana Instances¶
If you run a Grafana instance in each Kubernetes cluster (or want a single Holmes deployment to query several Grafana servers), set instances instead of the single-instance api_url/api_key fields. The LLM picks the right instance per tool call via the grafana_instance parameter (run grafana_list_instances to see configured names).
Top-level credentials act as global defaults. Per-instance settings override them. Use either api_key (Bearer) or username + password (HTTP Basic) — they're mutually exclusive per instance.
toolsets:
grafana/dashboards:
enabled: true
config:
# Global defaults — applied to every instance unless overridden
username: holmes
password: "{{ env.GRAFANA_PASSWORD }}"
timeout_seconds: 30
instances:
- name: prod-eu
api_url: https://grafana.eu-west-1.internal
- name: prod-us
api_url: https://grafana.us-east-1.internal
- name: staging
api_url: https://grafana.staging.internal
# Per-instance override (uses an API key instead of the global basic auth)
api_key: "{{ env.STAGING_GRAFANA_API_KEY }}"
additionalEnvVars:
- name: GRAFANA_PASSWORD
valueFrom:
secretKeyRef:
name: grafana-credentials
key: password
- name: STAGING_GRAFANA_API_KEY
valueFrom:
secretKeyRef:
name: grafana-credentials
key: staging-api-key
toolsets:
grafana/dashboards:
enabled: true
config:
username: holmes
password: "{{ env.GRAFANA_PASSWORD }}"
instances:
- name: prod-eu
api_url: https://grafana.eu-west-1.internal
- name: prod-us
api_url: https://grafana.us-east-1.internal
- name: staging
api_url: https://grafana.staging.internal
api_key: "{{ env.STAGING_GRAFANA_API_KEY }}"
holmes:
additionalEnvVars:
- name: GRAFANA_PASSWORD
valueFrom:
secretKeyRef:
name: grafana-credentials
key: password
toolsets:
grafana/dashboards:
enabled: true
config:
username: holmes
password: "{{ env.GRAFANA_PASSWORD }}"
instances:
- name: prod-eu
api_url: https://grafana.eu-west-1.internal
- name: prod-us
api_url: https://grafana.us-east-1.internal
Health check is tolerant: if some instances are unreachable at startup, the toolset still loads with the healthy ones. The unreachable instances are listed in the toolset status string and via grafana_list_instances.
Username/password authentication (single instance)¶
If you just want HTTP Basic auth without multi-instance, set username/password at the top level:
toolsets:
grafana/dashboards:
enabled: true
config:
api_url: https://grafana.internal
username: holmes
password: "{{ env.GRAFANA_PASSWORD }}"
Visual Rendering¶
When the Grafana Image Renderer is available, HolmesGPT can take screenshots of dashboards and panels and analyze them using the LLM's vision capabilities. This is useful for:
- Spotting anomalous spikes or patterns across many panels at once
- Analyzing visual dashboard layouts without parsing raw query data
- Investigating dashboards that use complex visualizations (heatmaps, gauges, etc.)
The LLM controls all rendering parameters — time range, dimensions, theme, timezone, and template variables — so it can zoom in on specific time windows or adjust the view as needed during investigation.
Rendering is disabled by default. To enable it, add enable_rendering: true to your config:
Reuses the grafana-api-key Kubernetes secret created in the Configuration section above.
Reuses the grafana-api-key Kubernetes secret created in the Configuration section above.
When rendering a full dashboard, HolmesGPT captures the entire page (all rows) so that panels at the bottom are not cropped.
Advanced Configuration¶
SSL Verification¶
For self-signed certificates, you can disable SSL verification:
Reuses the grafana-api-key Kubernetes secret created in the Configuration section above.
Reuses the grafana-api-key Kubernetes secret created in the Configuration section above.
External URL¶
If HolmesGPT accesses Grafana through an internal URL but you want clickable links in results to use a different URL:
Reuses the grafana-api-key Kubernetes secret created in the Configuration section above.
Reuses the grafana-api-key Kubernetes secret created in the Configuration section above.
Common Use Cases¶
holmes ask "Get the CPU usage queries from the Kubernetes cluster dashboard and check if any nodes are throttling"