Kubernetes Remediation (MCP)¶

Which Kubernetes toolset should I use?

Holmes has three Kubernetes integrations. Most users only need the first:

Kubernetes (built-in) — default for most users. Read-only access to cluster resources via kubectl, authenticated with the pod's ServiceAccount in-cluster or your local kubeconfig for CLI. No extra deployment.
Kubernetes (MCP) — use when you need OAuth/OIDC authentication (e.g. AKS with Microsoft Entra ID, or per-user RBAC enforced by your identity provider). Replaces the built-in toolset.
Kubernetes Remediation (MCP) — add on top of either of the above when you want Holmes to perform write actions (restart, scale, drain, patch, etc.). Complements the read-only toolsets rather than replacing them.

The Kubernetes Remediation MCP server provides safe kubectl command execution with layered security controls. It enables Holmes to not only diagnose Kubernetes issues but also remediate them — restarting pods, scaling deployments, draining nodes, and more.

This toolset is additive: keep your existing read-only Kubernetes toolset (built-in or MCP) enabled for diagnosis, and enable this one alongside it for write actions.

Write operations

Unlike the built-in read-only Kubernetes toolset, this MCP server can execute write operations (edit, patch, delete, scale, drain, etc.). Configure the allowedCommands setting carefully to match your security requirements.

Prerequisites¶

For CLI deployments, you'll need to create the RBAC resources manually. For Helm deployments, the chart creates them automatically.

Configuration¶

Holmes CLIHolmes Helm ChartRobusta Helm Chart

For CLI usage, you need to deploy the Kubernetes Remediation MCP server with appropriate RBAC.

Step 1: Create RBAC Resources

Create a file named k8s-remediation-rbac.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: holmes-mcp
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: k8s-remediation-mcp-sa
  namespace: holmes-mcp
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: k8s-remediation-mcp
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin  # Use a more restrictive role in production
subjects:
- kind: ServiceAccount
  name: k8s-remediation-mcp-sa
  namespace: holmes-mcp

kubectl apply -f k8s-remediation-rbac.yaml

Step 2: Deploy the MCP Server

Create a file named k8s-remediation-mcp-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-remediation-mcp-server
  namespace: holmes-mcp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: k8s-remediation-mcp-server
  template:
    metadata:
      labels:
        app: k8s-remediation-mcp-server
    spec:
      serviceAccountName: k8s-remediation-mcp-sa
      containers:
      - name: k8s-remediation-mcp
        image: us-central1-docker.pkg.dev/genuine-flight-317411/mcp/kubernetes-remediation-mcp:1.0.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8000
          name: http
        env:
        - name: KUBECTL_ALLOWED_COMMANDS
          value: "edit,patch,delete,scale,rollout,cordon,uncordon,drain,taint,label,annotate"
        - name: KUBECTL_TIMEOUT
          value: "60"
        resources:
          requests:
            memory: "64Mi"
            cpu: "50m"
          limits:
            memory: "128Mi"
        securityContext:
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
          allowPrivilegeEscalation: false
        readinessProbe:
          tcpSocket:
            port: 8000
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          tcpSocket:
            port: 8000
          initialDelaySeconds: 10
          periodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
  name: k8s-remediation-mcp-server
  namespace: holmes-mcp
spec:
  selector:
    app: k8s-remediation-mcp-server
  ports:
  - port: 8000
    targetPort: 8000
    protocol: TCP
    name: http

kubectl apply -f k8s-remediation-mcp-deployment.yaml

Step 3: Configure Holmes CLI

Add the MCP server configuration to ~/.holmes/config.yaml:

mcp_servers:
  kubernetes_remediation:
    description: "Kubernetes remediation - execute kubectl commands"
    config:
      url: "http://k8s-remediation-mcp-server.holmes-mcp.svc.cluster.local:8000/mcp"
      mode: streamable-http
    restricted_tools:
      - "*"
    approval_required_tools:
      - "*"

restricted_tools: ["*"] means all tools from this MCP server can only be called during a runbook invocation (prevents ad-hoc write operations). approval_required_tools: ["*"] means all tools require user confirmation before execution.

After making changes to your configuration, run:

holmes toolset refresh

Add the following to your values.yaml:

mcpAddons:
  kubernetesRemediation:
    enabled: true
    # Tools that can only be called after a runbook invocation
    # Use ["*"] to restrict all tools, or specify tool names like ["kubectl", "run_image"]
    restrictedTools:
      - "*"
    # Tools that require user confirmation before execution
    # Use ["*"] to require approval for all tools, or specify tool names
    approvalRequiredTools:
      - "*"

Then deploy or upgrade your Holmes installation:

helm upgrade --install holmes robusta/holmes -f values.yaml

Add the following to your generated_values.yaml:

holmes:
  mcpAddons:
    kubernetesRemediation:
      enabled: true
      # Tools that can only be called after a runbook invocation
      # Use ["*"] to restrict all tools, or specify tool names like ["kubectl", "run_image"]
      restrictedTools:
        - "*"
      # Tools that require user confirmation before execution
      # Use ["*"] to require approval for all tools, or specify tool names
      approvalRequiredTools:
        - "*"

Then deploy or upgrade your Robusta installation:

helm upgrade --install robusta robusta/robusta -f generated_values.yaml --set clusterName=YOUR_CLUSTER_NAME

Security Controls¶

The MCP server implements multiple security layers:

Control	Description
Restricted tools	By default, all tools require a runbook invocation to be called — prevents ad-hoc write operations
Approval required	By default, all tools require user confirmation before execution
Command allowlist	Only explicitly allowed kubectl subcommands can execute
Flag blocklist	Flags like `--kubeconfig`, `--context`, `--token` are always blocked
Shell injection protection	Shell metacharacters are rejected
Image allowlist	The `run_image` tool only allows pre-approved container images
RBAC enforcement	Kubernetes RBAC restricts which resources can be accessed
Command timeout	Commands are killed after a configurable timeout (default: 60s)

Available Tools¶

Tool	Description
`kubectl`	Execute a validated kubectl command. Args are passed as a list (e.g., `["get", "pods", "-n", "production"]`)
`run_image`	Run a temporary pod with a pre-approved image (disabled by default)
`get_config`	Get the current MCP server configuration for debugging

Common Use Cases¶

holmes ask "Restart the payment-service deployment in the production namespace"

holmes ask "Scale the web-frontend deployment to 5 replicas"

holmes ask "Cordon the problematic node and drain it safely"

holmes ask "The checkout-api pods are crashlooping - investigate and fix"

Additional Resources¶

Kubernetes Remediation MCP Server setup guide