⚡ July 12, 2026¶
Generated: 2026-07-12 05:58 UTC
Total Duration: 2h 45m 41s
Iterations: 1
Judge (classifier) model: gpt-4.1
Fast Benchmark
Markers: regression or benchmark
Schedule: Weekly (Sunday 2 AM UTC)
Purpose: Quick regression tests to catch breaking changes
HolmesGPT is continuously evaluated against real-world Kubernetes and cloud troubleshooting scenarios.
If you find scenarios that HolmesGPT does not perform well on, please consider adding them as evals to the benchmark.
Model Accuracy Comparison¶
| Model | Pass | Fail | Skip/Error | Total | Success Rate |
|---|---|---|---|---|---|
| deepseek-r1-reasoner | 15 | 5 | 0 | 20 | 🟡 75% (15/20) |
| deepseek-v3.2-chat | 15 | 5 | 0 | 20 | 🟡 75% (15/20) |
| gemini-3.1-pro-preview | 18 | 2 | 0 | 20 | 🟡 90% (18/20) |
| gpt-5.3-codex | 14 | 6 | 0 | 20 | 🟡 70% (14/20) |
| gpt-5.4 | 17 | 3 | 0 | 20 | 🟡 85% (17/20) |
| gpt-5.5 | 17 | 3 | 0 | 20 | 🟡 85% (17/20) |
| haiku-4.5 | 15 | 5 | 0 | 20 | 🟡 75% (15/20) |
| opus-4.6 | 18 | 2 | 0 | 20 | 🟡 90% (18/20) |
| opus-4.7 | 18 | 2 | 0 | 20 | 🟡 90% (18/20) |
| opus-4.8 | 17 | 3 | 0 | 20 | 🟡 85% (17/20) |
| qwen-next-80B-instruct | 7 | 13 | 0 | 20 | 🟡 35% (7/20) |
| qwen-next-80B-thinking | 0 | 20 | 0 | 20 | 🔴 0% (0/20) |
| sonnet-4.6 | 11 | 9 | 0 | 20 | 🟡 55% (11/20) |
Model Cost Comparison¶
| Model | Tests | Avg Cost | Min Cost | Max Cost | Total Cost |
|---|---|---|---|---|---|
| deepseek-r1-reasoner | 17 | $0.00 | $0.00 | $0.00 | $0.04 |
| deepseek-v3.2-chat | 17 | $0.00 | $0.00 | $0.01 | $0.03 |
| gemini-3.1-pro-preview | 20 | $0.86 | $0.03 | $15.33 | $17.30 |
| gpt-5.3-codex | 20 | $0.02 | $0.00 | $0.04 | $0.44 |
| gpt-5.4 | 20 | $0.09 | $0.00 | $0.22 | $1.77 |
| gpt-5.5 | 20 | $0.27 | $0.01 | $0.96 | $5.48 |
| haiku-4.5 | 20 | $0.06 | $0.02 | $0.16 | $1.12 |
| opus-4.6 | 20 | $0.41 | $0.10 | $3.23 | $8.14 |
| opus-4.7 | 20 | $0.26 | $0.14 | $0.52 | $5.12 |
| opus-4.8 | 20 | $0.28 | $0.14 | $0.49 | $5.69 |
| qwen-next-80B-instruct | 20 | $0.01 | $0.00 | $0.07 | $0.28 |
| sonnet-4.6 | 20 | $0.18 | $0.06 | $0.88 | $3.56 |
Model Latency Comparison¶
| Model | Avg (s) | Min (s) | Max (s) | P50 (s) | P95 (s) |
|---|---|---|---|---|---|
| deepseek-r1-reasoner | 26.8 | 1.2 | 104.7 | 23.5 | 104.7 |
| deepseek-v3.2-chat | 22.5 | 1.2 | 86.7 | 19.2 | 86.7 |
| gemini-3.1-pro-preview | 138.8 | 5.8 | 2214.9 | 21.2 | 2214.9 |
| gpt-5.3-codex | 10.0 | 3.4 | 17.1 | 11.1 | 17.1 |
| gpt-5.4 | 20.4 | 4.5 | 44.9 | 20.6 | 44.9 |
| gpt-5.5 | 46.5 | 4.3 | 173.0 | 34.9 | 173.0 |
| haiku-4.5 | 26.9 | 4.1 | 100.0 | 18.1 | 100.0 |
| opus-4.6 | 66.4 | 5.5 | 566.8 | 40.1 | 566.8 |
| opus-4.7 | 24.1 | 4.0 | 59.4 | 20.8 | 59.4 |
| opus-4.8 | 36.1 | 4.2 | 90.3 | 26.5 | 90.3 |
| qwen-next-80B-instruct | 18.1 | 4.3 | 47.9 | 15.9 | 47.9 |
| sonnet-4.6 | 42.9 | 4.1 | 226.5 | 32.5 | 226.5 |
⚠️ Note: 21 test(s) excluded from latency calculations due to throttling/timeout errors (opus-4.7: 1, qwen-next-80B-thinking: 20)
Performance by Tag¶
Success rate by test category and model:
| Tag | deepseek-r1-reasoner | deepseek-v3.2-chat | gemini-3.1-pro-preview | gpt-5.3-codex | gpt-5.4 | gpt-5.5 | haiku-4.5 | opus-4.6 | opus-4.7 | opus-4.8 | qwen-next-80B-instruct | qwen-next-80B-thinking | sonnet-4.6 | Warnings |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| benchmark | 🟡 83% (⅚) | 🟡 83% (⅚) | 🟡 83% (⅚) | 🟡 50% (3/6) | 🟡 67% (4/6) | 🟡 83% (⅚) | 🟡 67% (4/6) | 🟢 100% (6/6) | 🟡 83% (⅚) | 🟡 83% (⅚) | 🟡 33% (2/6) | 🔴 0% (0/6) | 🟡 83% (⅚) | |
| context_window | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟡 50% (½) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟡 50% (½) | 🔴 0% (0/2) | 🟢 100% (2/2) | |
| counting | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🟡 50% (½) | |
| datetime | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟡 67% (⅔) | 🔴 0% (0/3) | 🟢 100% (3/3) | |
| easy | 🟡 88% (⅞) | 🟡 88% (⅞) | 🟡 88% (⅞) | 🟡 75% (6/8) | 🟡 88% (⅞) | 🟡 75% (6/8) | 🟡 75% (6/8) | 🟡 75% (6/8) | 🟡 88% (⅞) | 🟡 75% (6/8) | 🟡 38% (⅜) | 🔴 0% (0/8) | 🟡 25% (2/8) | |
| elasticsearch | 🔴 0% (0/3) | 🔴 0% (0/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟡 67% (⅔) | 🔴 0% (0/3) | 🟢 100% (3/3) | |
| grafana | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🟢 100% (1/1) | |
| hard | 🟡 50% (½) | 🟡 50% (½) | 🟡 50% (½) | 🟡 50% (½) | 🟡 50% (½) | 🟡 50% (½) | 🟡 50% (½) | 🟢 100% (2/2) | 🟡 50% (½) | 🟡 50% (½) | 🟡 50% (½) | 🔴 0% (0/2) | 🟡 50% (½) | |
| kubernetes | 🟡 89% (8/9) | 🟡 89% (8/9) | 🟡 89% (8/9) | 🟡 56% (5/9) | 🟡 78% (7/9) | 🟡 78% (7/9) | 🟡 67% (6/9) | 🟡 78% (7/9) | 🟡 89% (8/9) | 🟡 78% (7/9) | 🟡 11% (1/9) | 🔴 0% (0/9) | 🟡 33% (3/9) | |
| logs | 🟡 67% (4/6) | 🟡 67% (4/6) | 🟡 67% (4/6) | 🟡 33% (2/6) | 🟡 50% (3/6) | 🟡 67% (4/6) | 🟡 50% (3/6) | 🟡 83% (⅚) | 🟡 67% (4/6) | 🟡 67% (4/6) | 🟡 33% (2/6) | 🔴 0% (0/6) | 🟡 50% (3/6) | |
| loki | 🟡 50% (½) | 🟡 50% (½) | 🟡 50% (½) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🟡 50% (½) | 🔴 0% (0/2) | 🟡 50% (½) | 🟡 50% (½) | 🟡 50% (½) | 🔴 0% (0/2) | 🔴 0% (0/2) | 🟡 50% (½) | |
| medium | 🟡 67% (6/9) | 🟡 67% (6/9) | 🟢 100% (9/9) | 🟡 67% (6/9) | 🟡 89% (8/9) | 🟢 100% (9/9) | 🟡 78% (7/9) | 🟢 100% (9/9) | 🟢 100% (9/9) | 🟢 100% (9/9) | 🟡 33% (3/9) | 🔴 0% (0/9) | 🟡 89% (8/9) | |
| metrics | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🟢 100% (1/1) | |
| multi-cluster | 🔴 0% (0/3) | 🔴 0% (0/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟡 67% (⅔) | 🔴 0% (0/3) | 🟢 100% (3/3) | |
| network | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | |
| one-test | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | |
| port-forward | 🟡 67% (⅔) | 🟡 67% (⅔) | 🟡 67% (⅔) | 🟡 33% (⅓) | 🟡 33% (⅓) | 🟡 67% (⅔) | 🟡 33% (⅓) | 🟡 67% (⅔) | 🟡 67% (⅔) | 🟡 67% (⅔) | 🟡 33% (⅓) | 🔴 0% (0/3) | 🟡 67% (⅔) | |
| question-answer | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🟢 100% (1/1) | |
| regression | 🟡 71% (10/14) | 🟡 71% (10/14) | 🟡 93% (13/14) | 🟡 79% (11/14) | 🟡 93% (13/14) | 🟡 86% (12/14) | 🟡 79% (11/14) | 🟡 86% (12/14) | 🟡 93% (13/14) | 🟡 86% (12/14) | 🟡 36% (5/14) | 🔴 0% (0/14) | 🟡 43% (6/14) | |
| skills | 🟡 92% (11/12) | 🟡 92% (11/12) | 🟡 92% (11/12) | 🟡 75% (9/12) | 🟡 92% (11/12) | 🟡 83% (10/12) | 🟡 83% (10/12) | 🟡 83% (10/12) | 🟡 92% (11/12) | 🟡 83% (10/12) | 🟡 25% (3/12) | 🔴 0% (0/12) | 🟡 33% (4/12) | |
| transparency | 🔴 0% (0/3) | 🔴 0% (0/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟡 67% (⅔) | 🔴 0% (0/3) | 🟢 100% (3/3) | |
| Overall | 🟡 75% (15/20) | 🟡 75% (15/20) | 🟡 90% (18/20) | 🟡 70% (14/20) | 🟡 85% (17/20) | 🟡 85% (17/20) | 🟡 75% (15/20) | 🟡 90% (18/20) | 🟡 90% (18/20) | 🟡 85% (17/20) | 🟡 35% (7/20) | 🔴 0% (0/20) | 🟡 55% (11/20) |
Raw Results¶
Status of all evaluations across models. Color coding:
- 🟢 Passing 100% (stable)
- 🟡 Passing 1-99%
- 🔴 Passing 0% (failing)
- 🔧 Mock data failure (missing or invalid test data)
- ⚠️ Setup failure (environment/infrastructure issue)
- ⏱️ Timeout or rate limit error
- ⏭️ Test skipped (e.g., known issue or precondition not met)
Detailed Raw Results¶
| Eval ID | deepseek-r1-reasoner | deepseek-v3.2-chat | gemini-3.1-pro-preview | gpt-5.3-codex | gpt-5.4 | gpt-5.5 | haiku-4.5 | opus-4.6 | opus-4.7 | opus-4.8 | qwen-next-80B-instruct | qwen-next-80B-thinking | sonnet-4.6 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 09_crashpod 🔗 | 🟢 100% (1/1) / ⏱️ 23.5s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 20.9s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 18.1s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 13.0s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 26.9s / 💰 $0.10 | 🟢 100% (1/1) / ⏱️ 25.4s / 💰 $0.11 | 🟢 100% (1/1) / ⏱️ 18.1s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 24.9s / 💰 $0.19 | 🟢 100% (1/1) / ⏱️ 14.9s / 💰 $0.21 | 🟢 100% (1/1) / ⏱️ 15.4s / 💰 $0.19 | 🔴 0% (0/1) / ⏱️ 17.5s / 💰 $0.01 | ⏱️ 0% (0/1) / ⏱️ 1271.4s | 🔴 0% (0/1) / ⏱️ 25.3s / 💰 $0.12 |
| 100a_loki_historical_logs 🔗 | 🟢 100% (1/1) / ⏱️ 39.9s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 86.7s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 28.3s / 💰 $0.10 | 🔴 0% (0/1) / ⏱️ 3.6s / 💰 $0.02 | 🔴 0% (0/1) / ⏱️ 37.2s / 💰 $0.17 | 🟢 100% (1/1) / ⏱️ 70.6s / 💰 $0.56 | 🔴 0% (0/1) / ⏱️ 15.5s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 103.9s / 💰 $0.70 | 🟢 100% (1/1) / ⏱️ 24.9s / 💰 $0.25 | 🟢 100% (1/1) / ⏱️ 56.1s / 💰 $0.37 | 🔴 0% (0/1) / ⏱️ 22.7s / 💰 $0.02 | ⏱️ 0% (0/1) / ⏱️ 1270.9s | 🟢 100% (1/1) / ⏱️ 39.8s / 💰 $0.15 |
| 101_loki_historical_logs_pod_deleted 🔗 | 🔴 0% (0/1) / ⏱️ 104.7s / 💰 $0.00 | 🔴 0% (0/1) / ⏱️ 51.7s / 💰 $0.00 | 🔴 0% (0/1) / ⏱️ 2214.9s / 💰 $15.33 | 🔴 0% (0/1) / ⏱️ 11.1s / 💰 $0.01 | 🔴 0% (0/1) / ⏱️ 26.7s / 💰 $0.12 | 🔴 0% (0/1) / ⏱️ 76.5s / 💰 $0.41 | 🔴 0% (0/1) / ⏱️ 30.9s / 💰 $0.05 | 🔴 0% (0/1) / ⏱️ 566.8s / 💰 $3.23 | 🔴 0% (0/1) / ⏱️ 37.2s / 💰 $0.30 | 🔴 0% (0/1) / ⏱️ 66.8s / 💰 $0.36 | 🔴 0% (0/1) / ⏱️ 9.9s / 💰 $0.00 | ⏱️ 0% (0/1) / ⏱️ 1272.3s | 🔴 0% (0/1) / ⏱️ 226.5s / 💰 $0.88 |
| 108_logs_nearby_lines 🔗 | 🔴 0% (0/1) / ⏱️ 74.7s / 💰 $0.00 | 🔴 0% (0/1) / ⏱️ 38.4s / 💰 $0.00 | 🔴 0% (0/1) / ⏱️ 35.4s / 💰 $0.12 | 🔴 0% (0/1) / ⏱️ 17.1s / 💰 $0.04 | 🔴 0% (0/1) / ⏱️ 44.9s / 💰 $0.22 | 🔴 0% (0/1) / ⏱️ 49.4s / 💰 $0.39 | 🔴 0% (0/1) / ⏱️ 33.0s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 57.6s / 💰 $0.40 | 🔴 0% (0/1) / ⏱️ 20.8s / 💰 $0.24 | 🔴 0% (0/1) / ⏱️ 48.2s / 💰 $0.36 | 🔴 0% (0/1) / ⏱️ 47.9s / 💰 $0.07 | ⏱️ 0% (0/1) / ⏱️ 1271.2s | 🔴 0% (0/1) / ⏱️ 34.7s / 💰 $0.16 |
| 112_find_pvcs_by_uuid 🔗 | 🟢 100% (1/1) / ⏱️ 8.3s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 6.9s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 17.2s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 10.4s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 10.1s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 17.9s / 💰 $0.13 | 🟢 100% (1/1) / ⏱️ 12.1s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 23.7s / 💰 $0.17 | 🟢 100% (1/1) / ⏱️ 15.5s / 💰 $0.24 | 🟢 100% (1/1) / ⏱️ 17.1s / 💰 $0.20 | 🔴 0% (0/1) / ⏱️ 6.3s / 💰 $0.00 | ⏱️ 0% (0/1) / ⏱️ 1271.3s | 🟢 100% (1/1) / ⏱️ 15.4s / 💰 $0.09 |
| 12_job_crashing 🔗 | 🟢 100% (1/1) / ⏱️ 32.0s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 33.8s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 17.1s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 16.5s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 25.0s / 💰 $0.12 | 🟢 100% (1/1) / ⏱️ 17.3s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 16.6s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 25.0s / 💰 $0.17 | 🟢 100% (1/1) / ⏱️ 13.4s / 💰 $0.20 | 🟢 100% (1/1) / ⏱️ 21.8s / 💰 $0.23 | 🟢 100% (1/1) / ⏱️ 18.0s / 💰 $0.02 | ⏱️ 0% (0/1) / ⏱️ 1271.2s | 🔴 0% (0/1) / ⏱️ 26.2s / 💰 $0.13 |
| 176_network_policy_blocking_traffic_no_skills 🔗 | 🟢 100% (1/1) / ⏱️ 28.3s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 40.7s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 15.1s / 💰 $0.06 | 🔴 0% (0/1) / ⏱️ 12.8s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 20.6s / 💰 $0.10 | 🟢 100% (1/1) / ⏱️ 22.0s / 💰 $0.18 | 🔴 0% (0/1) / ⏱️ 64.7s / 💰 $0.12 | 🔴 0% (0/1) / ⏱️ 40.1s / 💰 $0.26 | 🟢 100% (1/1) / ⏱️ 22.3s / 💰 $0.26 | 🔴 0% (0/1) / ⏱️ 78.7s / 💰 $0.48 | 🔴 0% (0/1) / ⏱️ 24.7s / 💰 $0.02 | ⏱️ 0% (0/1) / ⏱️ 1271.2s | 🔴 0% (0/1) / ⏱️ 32.5s / 💰 $0.14 |
| 179_grafana_big_dashboard_query 🔗 | 🟢 100% (1/1) / ⏱️ 13.1s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 8.6s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 21.2s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 8.2s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 8.3s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 10.1s / 💰 $0.11 | 🟢 100% (1/1) / ⏱️ 11.7s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 15.6s / 💰 $0.17 | 🟢 100% (1/1) / ⏱️ 12.6s / 💰 $0.24 | 🟢 100% (1/1) / ⏱️ 15.0s / 💰 $0.23 | 🟢 100% (1/1) / ⏱️ 12.9s / 💰 $0.01 | ⏱️ 0% (0/1) / ⏱️ 1271.0s | 🟢 100% (1/1) / ⏱️ 13.9s / 💰 $0.09 |
| 227_count_configmaps_per_namespace[0] 🔗 | 🟢 100% (1/1) / ⏱️ 17.9s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 9.5s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 54.1s / 💰 $0.19 | 🟢 100% (1/1) / ⏱️ 10.9s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 9.6s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 21.5s / 💰 $0.16 | 🟢 100% (1/1) / ⏱️ 20.5s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 20.2s / 💰 $0.15 | 🟢 100% (1/1) / ⏱️ 25.4s / 💰 $0.23 | 🟢 100% (1/1) / ⏱️ 17.2s / 💰 $0.20 | 🔴 0% (0/1) / ⏱️ 14.9s / 💰 $0.01 | ⏱️ 0% (0/1) / ⏱️ 1271.6s | 🔴 0% (0/1) / ⏱️ 17.4s / 💰 $0.17 |
| 243_pod_names_contain_service 🔗 | 🟢 100% (1/1) / ⏱️ 31.2s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 22.7s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 19.3s / 💰 $0.08 | 🔴 0% (0/1) / ⏱️ 3.4s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 23.6s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 24.9s / 💰 $0.16 | 🟢 100% (1/1) / ⏱️ 15.8s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 26.8s / 💰 $0.19 | 🟢 100% (1/1) / ⏱️ 127.3s / 💰 $0.19 | 🟢 100% (1/1) / ⏱️ 17.9s / 💰 $0.19 | 🔴 0% (0/1) / ⏱️ 5.2s / 💰 $0.00 | ⏱️ 0% (0/1) / ⏱️ 1271.2s | 🔴 0% (0/1) / ⏱️ 28.9s / 💰 $0.12 |
| 24_misconfigured_pvc 🔗 | 🟢 100% (1/1) / ⏱️ 23.8s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 17.1s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 20.7s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 13.0s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 17.5s / 💰 $0.06 | 🔴 0% (0/1) / ⏱️ 34.9s / 💰 $0.15 | 🟢 100% (1/1) / ⏱️ 17.3s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 25.3s / 💰 $0.21 | 🟢 100% (1/1) / ⏱️ 19.7s / 💰 $0.24 | 🟢 100% (1/1) / ⏱️ 24.6s / 💰 $0.23 | 🔴 0% (0/1) / ⏱️ 7.2s / 💰 $0.00 | ⏱️ 0% (0/1) / ⏱️ 1270.9s | 🔴 0% (0/1) / ⏱️ 31.2s / 💰 $0.14 |
| 254_elasticsearch_dr_test_log_check 🔗 | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.2s | 🟢 100% (1/1) / ⏱️ 33.3s / 💰 $0.10 | 🟢 100% (1/1) / ⏱️ 11.1s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 15.6s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 90.1s / 💰 $0.54 | 🟢 100% (1/1) / ⏱️ 44.2s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 66.1s / 💰 $0.31 | 🟢 100% (1/1) / ⏱️ 53.5s / 💰 $0.36 | 🟢 100% (1/1) / ⏱️ 48.7s / 💰 $0.28 | 🔴 0% (0/1) / ⏱️ 15.5s / 💰 $0.01 | ⏱️ 0% (0/1) / ⏱️ 1272.0s | 🟢 100% (1/1) / ⏱️ 56.1s / 💰 $0.18 |
| 259_wrong_cluster_logs_confusion 🔗 | 🔴 0% (0/1) / ⏱️ 1.2s | 🔴 0% (0/1) / ⏱️ 1.3s | 🟢 100% (1/1) / ⏱️ 61.2s / 💰 $0.21 | 🟢 100% (1/1) / ⏱️ 12.3s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 15.9s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 173.0s / 💰 $0.96 | 🟢 100% (1/1) / ⏱️ 100.0s / 💰 $0.16 | 🟢 100% (1/1) / ⏱️ 72.5s / 💰 $0.33 | 🟢 100% (1/1) / ⏱️ 25.4s / 💰 $0.21 | 🟢 100% (1/1) / ⏱️ 90.3s / 💰 $0.49 | 🟢 100% (1/1) / ⏱️ 16.1s / 💰 $0.01 | ⏱️ 0% (0/1) / ⏱️ 1271.7s | 🟢 100% (1/1) / ⏱️ 69.7s / 💰 $0.21 |
| 260_global_es_remote_cluster_logs 🔗 | 🔴 0% (0/1) / ⏱️ 1.3s | 🔴 0% (0/1) / ⏱️ 1.3s | 🟢 100% (1/1) / ⏱️ 38.8s / 💰 $0.13 | 🟢 100% (1/1) / ⏱️ 14.3s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 33.7s / 💰 $0.13 | 🟢 100% (1/1) / ⏱️ 102.9s / 💰 $0.51 | 🔴 0% (0/1) / ⏱️ 42.8s / 💰 $0.11 | 🟢 100% (1/1) / ⏱️ 71.7s / 💰 $0.34 | 🟢 100% (1/1) / ⏱️ 42.2s / 💰 $0.30 | 🟢 100% (1/1) / ⏱️ 47.1s / 💰 $0.28 | 🟢 100% (1/1) / ⏱️ 15.9s / 💰 $0.01 | ⏱️ 0% (0/1) / ⏱️ 1271.2s | 🟢 100% (1/1) / ⏱️ 60.3s / 💰 $0.19 |
| 43_current_datetime_from_prompt 🔗 | 🟢 100% (1/1) / ⏱️ 4.4s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 3.5s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 5.8s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 3.4s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 5.0s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 4.3s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 4.1s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 5.5s / 💰 $0.10 | 🟢 100% (1/1) / ⏱️ 4.0s / 💰 $0.14 | 🟢 100% (1/1) / ⏱️ 4.2s / 💰 $0.14 | 🟢 100% (1/1) / ⏱️ 4.3s / 💰 $0.00 | ⏱️ 0% (0/1) / ⏱️ 1271.1s | 🟢 100% (1/1) / ⏱️ 4.1s / 💰 $0.06 |
| 51_logs_summarize_errors 🔗 | 🟢 100% (1/1) / ⏱️ 14.3s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 10.6s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 15.6s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 10.0s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 9.8s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 13.3s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 12.1s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 19.5s / 💰 $0.15 | 🟢 100% (1/1) / ⏱️ 16.8s / 💰 $0.22 | 🟢 100% (1/1) / ⏱️ 18.6s / 💰 $0.21 | 🟢 100% (1/1) / ⏱️ 10.9s / 💰 $0.01 | ⏱️ 0% (0/1) / ⏱️ 1270.6s | 🔴 0% (0/1) / ⏱️ 27.1s / 💰 $0.12 |
| 61_exact_match_counting 🔗 | 🟢 100% (1/1) / ⏱️ 5.2s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 5.2s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 8.7s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 4.6s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 4.5s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 5.5s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 8.1s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 8.3s / 💰 $0.12 | 🟢 100% (1/1) / ⏱️ 7.0s / 💰 $0.29 | 🟢 100% (1/1) / ⏱️ 7.7s / 💰 $0.28 | 🔴 0% (0/1) / ⏱️ 5.5s / 💰 $0.00 | ⏱️ 0% (0/1) / ⏱️ 1271.2s | 🟢 100% (1/1) / ⏱️ 7.9s / 💰 $0.07 |
| 73a_time_window_anomaly 🔗 | 🟢 100% (1/1) / ⏱️ 44.7s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 23.6s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 32.3s / 💰 $0.13 | 🟢 100% (1/1) / ⏱️ 12.4s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 30.5s / 💰 $0.14 | 🟢 100% (1/1) / ⏱️ 56.6s / 💰 $0.27 | 🟢 100% (1/1) / ⏱️ 18.1s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 46.8s / 💰 $0.32 | 🟢 100% (1/1) / ⏱️ 22.8s / 💰 $0.25 | 🟢 100% (1/1) / ⏱️ 26.5s / 💰 $0.25 | 🟢 100% (1/1) / ⏱️ 40.8s / 💰 $0.02 | ⏱️ 0% (0/1) / ⏱️ 1271.1s | 🟢 100% (1/1) / ⏱️ 44.6s / 💰 $0.14 |
| 73b_time_window_anomaly 🔗 | 🟢 100% (1/1) / ⏱️ 21.6s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 19.2s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 20.9s / 💰 $0.08 | 🔴 0% (0/1) / ⏱️ 4.7s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 19.8s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 47.1s / 💰 $0.22 | 🟢 100% (1/1) / ⏱️ 21.3s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 59.5s / 💰 $0.29 | 🟢 100% (1/1) / ⏱️ 19.2s / 💰 $0.24 | 🟢 100% (1/1) / ⏱️ 27.6s / 💰 $0.25 | 🔴 0% (0/1) / ⏱️ 45.7s / 💰 $0.02 | ⏱️ 0% (0/1) / ⏱️ 1271.1s | 🟢 100% (1/1) / ⏱️ 45.3s / 💰 $0.17 |
| 96_no_matching_skill 🔗 | 🟢 100% (1/1) / ⏱️ 43.7s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 47.0s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 98.0s / 💰 $0.34 | 🟢 100% (1/1) / ⏱️ 6.5s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 22.4s / 💰 $0.15 | 🟢 100% (1/1) / ⏱️ 67.6s / 💰 $0.45 | 🟢 100% (1/1) / ⏱️ 31.0s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 48.6s / 💰 $0.31 | 🟢 100% (1/1) / ⏱️ 59.4s / 💰 $0.52 | 🟢 100% (1/1) / ⏱️ 72.2s / 💰 $0.47 | 🔴 0% (0/1) / ⏱️ 20.0s / 💰 $0.02 | ⏱️ 0% (0/1) / ⏱️ 1271.3s | 🟢 100% (1/1) / ⏱️ 50.6s / 💰 $0.21 |
Results are automatically generated and updated weekly. View full traces and detailed analysis in Braintrust experiment: ci-benchmark-29177787728.