⚡ October 28, 2026¶
Generated: 2026-10-28 10:09 UTC
Total Duration: 1h 0m 29s
Iterations: 1
Judge (classifier) model: gpt-6.1-sol-medium
Fast Benchmark
Markers: regression or benchmark
Schedule: Weekly (Sunday 2 AM UTC)
Purpose: Quick regression tests to catch breaking changes
Adjusted result
259_wrong_cluster_logs_confusion had an eval bug in this run, fixed in #2679: it shared its cluster names with another eval on the same Elasticsearch, and its rubric checked for wording the system prompt doesn't ask for. Its 8 failures here are counted as passes. Re-run with the fix on all 9 models, 2 iterations each: 17 of 18 passed.
HolmesGPT is continuously evaluated against real-world Kubernetes and cloud troubleshooting scenarios.
If you find scenarios that HolmesGPT does not perform well on, please consider adding them as evals to the benchmark.
Model Accuracy Comparison¶
| Model | Pass | Fail | Skip/Error | Total | Success Rate |
|---|---|---|---|---|---|
| deepseek-v4-pro | 19 | 3 | 0 | 22 | 🟡 86% (19/22) |
| gemini-3.1-pro-preview | 18 | 4 | 0 | 22 | 🟡 82% (18/22) |
| glm-5.3 | 17 | 5 | 0 | 22 | 🟡 77% (17/22) |
| gpt-5.5 | 21 | 1 | 0 | 22 | 🟡 95% (21/22) |
| gpt-6.1-sol-high | 21 | 1 | 0 | 22 | 🟡 95% (21/22) |
| haiku-4.5 | 15 | 7 | 0 | 22 | 🟡 68% (15/22) |
| kimi-k3 | 21 | 1 | 0 | 22 | 🟡 95% (21/22) |
| opus-5.5-medium | 22 | 0 | 0 | 22 | 🟢 100% (22/22) |
| sonnet-5.5-medium | 19 | 3 | 0 | 22 | 🟡 86% (19/22) |
Skill-Suggestion Check Failures¶
These runs failed a deterministic skill-suggestion check, whatever the judge decided about the answer. They count as failures in the success rates above.
| Eval | Model | Check | Judge | Suggested skill(s) | |
|---|---|---|---|---|---|
| 🧩 | 243_pod_names_contain_service | gemini-3.1-pro-preview | suggested a skill, expected none | correct | Hardcoded readiness probe failure |
| 🧩 | 227_count_configmaps_per_namespace[0] | glm-5.3 | suggested a skill, expected none | correct | Kubernetes query tool quirks in this environment |
| 🧩 | 51_logs_summarize_errors | glm-5.3 | suggested a skill, expected none | correct | Kubernetes workload naming and pod-log lookup conventions in this environment |
| 🧩 | 61_exact_match_counting | glm-5.3 | suggested a skill, expected none | correct | Kubernetes query tool quirks in this environment |
| 🧩 | 09_crashpod | gpt-6.1-sol-high | suggested a skill, expected none | correct | Kubernetes resource lookup conventions |
| 🧩 | 271_skill_suggestion_elasticsearch | haiku-4.5 | suggested no skill, expected one | wrong | — |
| 🧩 | 273_skill_suggestion_prometheus | haiku-4.5 | suggested no skill, expected one | wrong | — |
| 🧩 | 271_skill_suggestion_elasticsearch | sonnet-5.5-medium | suggested no skill, expected one | wrong | — |
| 🧩 | 273_skill_suggestion_prometheus | sonnet-5.5-medium | suggested no skill, expected one | wrong | — |
Model Cost Comparison¶
| Model | Tests | Avg Cost | Min Cost | Max Cost | Total Cost |
|---|---|---|---|---|---|
| deepseek-v4-pro | 22 | $0.11 | $0.00 | $0.60 | $2.45 |
| gemini-3.1-pro-preview | 22 | $0.11 | $0.03 | $0.31 | $2.43 |
| glm-5.3 | 22 | $0.09 | $0.00 | $0.23 | $1.88 |
| gpt-5.5 | 22 | $0.20 | $0.01 | $0.61 | $4.38 |
| gpt-6.1-sol-high | 22 | $0.05 | $0.00 | $0.13 | $1.19 |
| haiku-4.5 | 22 | $0.04 | $0.02 | $0.08 | $0.90 |
| kimi-k3 | 22 | $0.11 | $0.04 | $0.22 | $2.33 |
| opus-5.5-medium | 22 | $0.20 | $0.10 | $0.45 | $4.39 |
| sonnet-5.5-medium | 22 | $0.08 | $0.04 | $0.24 | $1.75 |
Model Latency Comparison¶
| Model | Avg (s) | Min (s) | Max (s) | P50 (s) | P95 (s) |
|---|---|---|---|---|---|
| deepseek-v4-pro | 102.1 | 14.5 | 535.9 | 72.1 | 209.1 |
| gemini-3.1-pro-preview | 56.1 | 21.0 | 129.7 | 41.6 | 117.5 |
| glm-5.3 | 165.4 | 18.4 | 393.8 | 149.2 | 343.0 |
| gpt-5.5 | 50.0 | 14.0 | 108.9 | 42.4 | 95.4 |
| gpt-6.1-sol-high | 75.2 | 14.9 | 157.0 | 75.8 | 144.4 |
| haiku-4.5 | 25.4 | 14.2 | 39.4 | 24.4 | 35.8 |
| kimi-k3 | 85.7 | 19.5 | 191.8 | 79.6 | 151.5 |
| opus-5.5-medium | 34.2 | 11.8 | 70.3 | 32.0 | 58.7 |
| sonnet-5.5-medium | 25.4 | 13.3 | 47.4 | 23.4 | 44.8 |
Performance by Tag¶
Success rate by test category and model:
| Tag | deepseek-v4-pro | gemini-3.1-pro-preview | glm-5.3 | gpt-5.5 | gpt-6.1-sol-high | haiku-4.5 | kimi-k3 | opus-5.5-medium | sonnet-5.5-medium | Warnings |
|---|---|---|---|---|---|---|---|---|---|---|
| advanced | 🟡 33% (⅓) | 🟡 67% (⅔) | 🟡 33% (⅓) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🟡 33% (⅓) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟡 33% (⅓) | |
| benchmark | 🟡 75% (6/8) | 🟡 75% (6/8) | 🟡 75% (6/8) | 🟡 88% (⅞) | 🟢 100% (8/8) | 🟡 38% (⅜) | 🟢 100% (8/8) | 🟢 100% (8/8) | 🟡 62% (⅝) | |
| context_window | 🟢 100% (2/2) | 🟡 50% (½) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🔴 0% (0/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | |
| counting | 🟢 100% (2/2) | 🟢 100% (2/2) | 🔴 0% (0/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | |
| datetime | 🟢 100% (3/3) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟡 33% (⅓) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | |
| easy | 🟢 100% (7/7) | 🟢 100% (7/7) | 🟡 71% (5/7) | 🟢 100% (7/7) | 🟡 86% (6/7) | 🟡 86% (6/7) | 🟢 100% (7/7) | 🟢 100% (7/7) | 🟢 100% (7/7) | |
| elasticsearch | 🟡 75% (¾) | 🟡 75% (¾) | 🟢 100% (4/4) | 🟢 100% (4/4) | 🟢 100% (4/4) | 🟡 50% (2/4) | 🟡 75% (¾) | 🟢 100% (4/4) | 🟡 75% (¾) | |
| fast | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | |
| grafana | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | |
| hard | 🟡 50% (½) | 🟡 50% (½) | 🟡 50% (½) | 🟡 50% (½) | 🟢 100% (2/2) | 🟡 50% (½) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟡 50% (½) | |
| kubernetes | 🟡 90% (9/10) | 🟡 90% (9/10) | 🟡 80% (8/10) | 🟢 100% (10/10) | 🟡 90% (9/10) | 🟡 90% (9/10) | 🟢 100% (10/10) | 🟢 100% (10/10) | 🟡 90% (9/10) | |
| logs | 🟡 86% (6/7) | 🟡 71% (5/7) | 🟡 71% (5/7) | 🟡 86% (6/7) | 🟢 100% (7/7) | 🟡 29% (2/7) | 🟢 100% (7/7) | 🟢 100% (7/7) | 🟡 71% (5/7) | |
| loki | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | |
| medium | 🟡 82% (9/11) | 🟡 73% (8/11) | 🟡 91% (10/11) | 🟢 100% (11/11) | 🟢 100% (11/11) | 🟡 55% (6/11) | 🟡 91% (10/11) | 🟢 100% (11/11) | 🟡 82% (9/11) | |
| metrics | 🟡 50% (½) | 🟢 100% (2/2) | 🟡 50% (½) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟡 50% (½) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟡 50% (½) | |
| multi-cluster | 🟡 67% (⅔) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟡 67% (⅔) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🟢 100% (3/3) | |
| one-test | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | |
| port-forward | 🟡 75% (¾) | 🟢 100% (4/4) | 🟡 75% (¾) | 🟢 100% (4/4) | 🟢 100% (4/4) | 🟡 75% (¾) | 🟢 100% (4/4) | 🟢 100% (4/4) | 🟡 75% (¾) | |
| prometheus | 🔴 0% (0/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | |
| question-answer | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | |
| regression | 🟡 93% (13/14) | 🟡 86% (12/14) | 🟡 79% (11/14) | 🟢 100% (14/14) | 🟡 93% (13/14) | 🟡 86% (12/14) | 🟡 93% (13/14) | 🟢 100% (14/14) | 🟢 100% (14/14) | |
| skills | 🟡 92% (12/13) | 🟡 92% (12/13) | 🟡 69% (9/13) | 🟢 100% (13/13) | 🟡 92% (12/13) | 🟡 77% (10/13) | 🟢 100% (13/13) | 🟢 100% (13/13) | 🟡 85% (11/13) | |
| transparency | 🟡 67% (⅔) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟡 67% (⅔) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🟢 100% (3/3) | |
| Overall | 🟡 86% (19/22) | 🟡 82% (18/22) | 🟡 77% (17/22) | 🟡 95% (21/22) | 🟡 95% (21/22) | 🟡 68% (15/22) | 🟡 95% (21/22) | 🟢 100% (22/22) | 🟡 86% (19/22) |
Raw Results¶
Status of all evaluations across models. Color coding:
- 🟢 Passing 100% (stable)
- 🟡 Passing 1-99%
- 🔴 Passing 0% (failing)
- 🔧 Mock data failure (missing or invalid test data)
- ⚠️ Setup or model-provider failure (environment/infrastructure issue; the model never answered, so the run is left out of success rates)
- ⏱️ Timeout or rate limit error (left out of success rates)
- ⏭️ Test skipped (e.g., known issue or precondition not met)
- 🧩 Failed the skill-suggestion check (counted as a failure; see Skill-Suggestion Check Failures for the skill and the judge's verdict)
Detailed Raw Results¶
| Eval ID | deepseek-v4-pro | gemini-3.1-pro-preview | glm-5.3 | gpt-5.5 | gpt-6.1-sol-high | haiku-4.5 | kimi-k3 | opus-5.5-medium | sonnet-5.5-medium |
|---|---|---|---|---|---|---|---|---|---|
| 09_crashpod 🔗 | 🟢 100% (1/1) / ⏱️ 63.7s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 36.8s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 123.9s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 42.4s / 💰 $0.16 | 🔴🧩 0% (0/1) / ⏱️ 78.8s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 23.7s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 53.2s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 25.6s / 💰 $0.16 | 🟢 100% (1/1) / ⏱️ 23.6s / 💰 $0.06 |
| 100a_loki_historical_logs 🔗 | 🟢 100% (1/1) / ⏱️ 206.3s / 💰 $0.29 | 🟢 100% (1/1) / ⏱️ 37.7s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 343.0s / 💰 $0.17 | 🟢 100% (1/1) / ⏱️ 95.4s / 💰 $0.42 | 🟢 100% (1/1) / ⏱️ 117.3s / 💰 $0.13 | 🟢 100% (1/1) / ⏱️ 24.4s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 191.8s / 💰 $0.22 | 🟢 100% (1/1) / ⏱️ 56.2s / 💰 $0.45 | 🟢 100% (1/1) / ⏱️ 30.2s / 💰 $0.14 |
| 101_loki_historical_logs_pod_deleted 🔗 | 🟢 100% (1/1) / ⏱️ 209.1s / 💰 $0.21 | 🟢 100% (1/1) / ⏱️ 34.0s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 242.8s / 💰 $0.20 | 🟢 100% (1/1) / ⏱️ 65.2s / 💰 $0.32 | 🟢 100% (1/1) / ⏱️ 76.3s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 33.1s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 93.0s / 💰 $0.13 | 🟢 100% (1/1) / ⏱️ 32.0s / 💰 $0.19 | 🟢 100% (1/1) / ⏱️ 21.8s / 💰 $0.06 |
| 108_logs_nearby_lines 🔗 | 🔴 0% (0/1) / ⏱️ 92.9s / 💰 $0.08 | 🔴 0% (0/1) / ⏱️ 117.5s / 💰 $0.31 | 🔴 0% (0/1) / ⏱️ 152.0s / 💰 $0.11 | 🔴 0% (0/1) / ⏱️ 77.1s / 💰 $0.26 | 🟢 100% (1/1) / ⏱️ 85.0s / 💰 $0.06 | 🔴 0% (0/1) / ⏱️ 30.6s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 151.5s / 💰 $0.19 | 🟢 100% (1/1) / ⏱️ 36.1s / 💰 $0.24 | 🔴 0% (0/1) / ⏱️ 30.6s / 💰 $0.09 |
| 112_find_pvcs_by_uuid 🔗 | 🟢 100% (1/1) / ⏱️ 61.0s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 31.7s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 125.1s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 31.1s / 💰 $0.09 | 🟢 100% (1/1) / ⏱️ 39.2s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 19.3s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 59.7s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 22.9s / 💰 $0.14 | 🟢 100% (1/1) / ⏱️ 17.8s / 💰 $0.06 |
| 12_job_crashing 🔗 | 🟢 100% (1/1) / ⏱️ 49.2s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 30.8s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 81.3s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 40.3s / 💰 $0.15 | 🟢 100% (1/1) / ⏱️ 40.5s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 22.7s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 42.2s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 23.8s / 💰 $0.13 | 🟢 100% (1/1) / ⏱️ 17.1s / 💰 $0.06 |
| 179_grafana_big_dashboard_query 🔗 | 🟢 100% (1/1) / ⏱️ 26.2s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 33.4s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 242.7s / 💰 $0.14 | 🟢 100% (1/1) / ⏱️ 18.6s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 44.6s / 💰 $0.09 | 🟢 100% (1/1) / ⏱️ 20.3s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 32.3s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 21.8s / 💰 $0.26 | 🟢 100% (1/1) / ⏱️ 17.2s / 💰 $0.07 |
| 227_count_configmaps_per_namespace[0] 🔗 | 🟢 100% (1/1) / ⏱️ 42.3s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 29.5s / 💰 $0.05 | 🔴🧩 0% (0/1) / ⏱️ 96.4s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 36.2s / 💰 $0.12 | 🟢 100% (1/1) / ⏱️ 32.8s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 17.9s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 30.7s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 21.4s / 💰 $0.13 | 🟢 100% (1/1) / ⏱️ 23.4s / 💰 $0.09 |
| 243_pod_names_contain_service 🔗 | 🟢 100% (1/1) / ⏱️ 43.4s / 💰 $0.05 | 🔴🧩 0% (0/1) / ⏱️ 37.9s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 162.8s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 42.1s / 💰 $0.14 | 🟢 100% (1/1) / ⏱️ 62.2s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 20.2s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 58.7s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 24.3s / 💰 $0.15 | 🟢 100% (1/1) / ⏱️ 23.1s / 💰 $0.06 |
| 24_misconfigured_pvc 🔗 | 🟢 100% (1/1) / ⏱️ 56.6s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 52.5s / 💰 $0.11 | 🟢 100% (1/1) / ⏱️ 74.3s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 46.5s / 💰 $0.21 | 🟢 100% (1/1) / ⏱️ 58.4s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 22.9s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 58.8s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 26.4s / 💰 $0.15 | 🟢 100% (1/1) / ⏱️ 23.0s / 💰 $0.06 |
| 254_elasticsearch_dr_test_log_check 🔗 | 🔴 0% (0/1) / ⏱️ 111.0s / 💰 $0.08 | 🔴 0% (0/1) / ⏱️ 87.9s / 💰 $0.14 | 🟢 100% (1/1) / ⏱️ 117.7s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 53.2s / 💰 $0.20 | 🟢 100% (1/1) / ⏱️ 70.8s / 💰 $0.05 | 🔴 0% (0/1) / ⏱️ 29.4s / 💰 $0.04 | 🔴 0% (0/1) / ⏱️ 114.8s / 💰 $0.12 | 🟢 100% (1/1) / ⏱️ 44.8s / 💰 $0.19 | 🟢 100% (1/1) / ⏱️ 44.8s / 💰 $0.10 |
| 259_wrong_cluster_logs_confusion 🔗 | 🟢 100% (1/1) / ⏱️ 160.5s / 💰 $0.11 | 🟢 100% (1/1) / ⏱️ 96.1s / 💰 $0.25 | 🟢 100% (1/1) / ⏱️ 268.6s / 💰 $0.11 | 🟢 100% (1/1) / ⏱️ 61.7s / 💰 $0.27 | 🟢 100% (1/1) / ⏱️ 62.3s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 31.3s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 126.1s / 💰 $0.13 | 🟢 100% (1/1) / ⏱️ 40.6s / 💰 $0.16 | 🟢 100% (1/1) / ⏱️ 32.5s / 💰 $0.07 |
| 260_global_es_remote_cluster_logs 🔗 | 🟢 100% (1/1) / ⏱️ 77.7s / 💰 $0.10 | 🟢 100% (1/1) / ⏱️ 62.1s / 💰 $0.12 | 🟢 100% (1/1) / ⏱️ 113.7s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 50.9s / 💰 $0.24 | 🟢 100% (1/1) / ⏱️ 82.9s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 28.3s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 135.7s / 💰 $0.13 | 🟢 100% (1/1) / ⏱️ 38.5s / 💰 $0.17 | 🟢 100% (1/1) / ⏱️ 28.3s / 💰 $0.07 |
| 271_skill_suggestion_elasticsearch 🔗 | 🟢 100% (1/1) / ⏱️ 114.0s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 88.2s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 150.2s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 67.0s / 💰 $0.13 | 🟢 100% (1/1) / ⏱️ 99.9s / 💰 $0.02 | 🔴🧩 0% (0/1) / ⏱️ 35.8s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 134.4s / 💰 $0.11 | 🟢 100% (1/1) / ⏱️ 58.7s / 💰 $0.14 | 🔴🧩 0% (0/1) / ⏱️ 24.8s / 💰 $0.06 |
| 273_skill_suggestion_prometheus 🔗 | 🔴 0% (0/1) / ⏱️ 72.1s / 💰 $0.14 | 🟢 100% (1/1) / ⏱️ 88.8s / 💰 $0.12 | 🔴 0% (0/1) / ⏱️ 259.6s / 💰 $0.10 | 🟢 100% (1/1) / ⏱️ 84.6s / 💰 $0.20 | 🟢 100% (1/1) / ⏱️ 157.0s / 💰 $0.06 | 🔴🧩 0% (0/1) / ⏱️ 29.3s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 150.1s / 💰 $0.12 | 🟢 100% (1/1) / ⏱️ 70.3s / 💰 $0.21 | 🔴🧩 0% (0/1) / ⏱️ 26.6s / 💰 $0.07 |
| 283_todowrite_multistep_audit 🔗 | 🟢 100% (1/1) / ⏱️ 40.7s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 41.6s / 💰 $0.09 | 🟢 100% (1/1) / ⏱️ 110.8s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 31.0s / 💰 $0.14 | 🟢 100% (1/1) / ⏱️ 73.4s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 25.4s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 47.0s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 21.6s / 💰 $0.14 | 🟢 100% (1/1) / ⏱️ 21.4s / 💰 $0.06 |
| 43_current_datetime_from_prompt 🔗 | 🟢 100% (1/1) / ⏱️ 14.5s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 21.0s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 18.4s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 18.2s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 14.9s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 14.2s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 19.5s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 11.8s / 💰 $0.10 | 🟢 100% (1/1) / ⏱️ 13.3s / 💰 $0.04 |
| 51_logs_summarize_errors 🔗 | 🟢 100% (1/1) / ⏱️ 36.8s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 28.9s / 💰 $0.05 | 🔴🧩 0% (0/1) / ⏱️ 149.2s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 30.7s / 💰 $0.13 | 🟢 100% (1/1) / ⏱️ 75.8s / 💰 $0.04 | 🔴 0% (0/1) / ⏱️ 15.3s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 42.8s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 25.4s / 💰 $0.16 | 🟢 100% (1/1) / ⏱️ 20.9s / 💰 $0.07 |
| 61_exact_match_counting 🔗 | 🟢 100% (1/1) / ⏱️ 16.0s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 21.1s / 💰 $0.03 | 🔴🧩 0% (0/1) / ⏱️ 65.4s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 14.0s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 15.5s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 14.8s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 19.5s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 15.2s / 💰 $0.11 | 🟢 100% (1/1) / ⏱️ 14.9s / 💰 $0.05 |
| 73a_time_window_anomaly 🔗 | 🟢 100% (1/1) / ⏱️ 108.2s / 💰 $0.12 | 🔴 0% (0/1) / ⏱️ 53.1s / 💰 $0.12 | 🟢 100% (1/1) / ⏱️ 239.0s / 💰 $0.10 | 🟢 100% (1/1) / ⏱️ 42.2s / 💰 $0.19 | 🟢 100% (1/1) / ⏱️ 126.2s / 💰 $0.11 | 🔴 0% (0/1) / ⏱️ 24.1s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 79.6s / 💰 $0.13 | 🟢 100% (1/1) / ⏱️ 37.9s / 💰 $0.31 | 🟢 100% (1/1) / ⏱️ 22.1s / 💰 $0.08 |
| 73b_time_window_anomaly 🔗 | 🟢 100% (1/1) / ⏱️ 107.6s / 💰 $0.18 | 🟢 100% (1/1) / ⏱️ 73.6s / 💰 $0.16 | 🟢 100% (1/1) / ⏱️ 108.6s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 42.2s / 💰 $0.28 | 🟢 100% (1/1) / ⏱️ 96.7s / 💰 $0.12 | 🔴 0% (0/1) / ⏱️ 39.4s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 94.9s / 💰 $0.14 | 🟢 100% (1/1) / ⏱️ 58.0s / 💰 $0.43 | 🟢 100% (1/1) / ⏱️ 47.4s / 💰 $0.24 |
| 96_no_matching_skill 🔗 | 🟢 100% (1/1) / ⏱️ 535.9s / 💰 $0.60 | 🟢 100% (1/1) / ⏱️ 129.7s / 💰 $0.31 | 🟢 100% (1/1) / ⏱️ 393.8s / 💰 $0.23 | 🟢 100% (1/1) / ⏱️ 108.9s / 💰 $0.61 | 🟢 100% (1/1) / ⏱️ 144.4s / 💰 $0.11 | 🟢 100% (1/1) / ⏱️ 35.6s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 149.5s / 💰 $0.18 | 🟢 100% (1/1) / ⏱️ 38.7s / 💰 $0.27 | 🟢 100% (1/1) / ⏱️ 33.6s / 💰 $0.09 |
Results are automatically generated and updated weekly. View full traces and detailed analysis in Braintrust experiment: ci-benchmark-40661516238.