⚡ October 21, 2026¶
Generated: 2026-10-21 07:21 UTC
Total Duration: 1h 11m 22s
Iterations: 1
Judge (classifier) model: gpt-4.1
Fast Benchmark
Markers: regression or benchmark
Schedule: Weekly (Sunday 2 AM UTC)
Purpose: Quick regression tests to catch breaking changes
HolmesGPT is continuously evaluated against real-world Kubernetes and cloud troubleshooting scenarios.
If you find scenarios that HolmesGPT does not perform well on, please consider adding them as evals to the benchmark.
Model Accuracy Comparison¶
| Model | Pass | Fail | Skip/Error | Total | Success Rate |
|---|---|---|---|---|---|
| deepseek-v4-pro | 17 | 3 | 0 | 20 | 🟡 85% (17/20) |
| gemini-3.1-pro-preview | 16 | 4 | 0 | 20 | 🟡 80% (16/20) |
| glm-5.3 | 18 | 2 | 0 | 20 | 🟡 90% (18/20) |
| gpt-5.5 | 17 | 3 | 0 | 20 | 🟡 85% (17/20) |
| gpt-6.1-sol-high | 19 | 1 | 0 | 20 | 🟡 95% (19/20) |
| haiku-4.5 | 15 | 5 | 0 | 20 | 🟡 75% (15/20) |
| kimi-k3 | 18 | 2 | 0 | 20 | 🟡 90% (18/20) |
| opus-5.5-medium | 19 | 1 | 0 | 20 | 🟡 95% (19/20) |
| qwen-next-80B-instruct | 8 | 12 | 0 | 20 | 🟡 40% (8/20) |
| sonnet-5.5-medium | 18 | 2 | 0 | 20 | 🟡 90% (18/20) |
Model Cost Comparison¶
| Model | Tests | Avg Cost | Min Cost | Max Cost | Total Cost |
|---|---|---|---|---|---|
| deepseek-v4-pro | 20 | $0.10 | $0.00 | $0.92 | $2.08 |
| gemini-3.1-pro-preview | 18 | $0.09 | $0.03 | $0.38 | $1.59 |
| glm-5.3 | 20 | $0.02 | $0.00 | $0.16 | $0.43 |
| gpt-5.5 | 20 | $0.22 | $0.02 | $0.68 | $4.34 |
| gpt-6.1-sol-high | 20 | $0.06 | $0.00 | $0.14 | $1.12 |
| haiku-4.5 | 20 | $0.04 | $0.02 | $0.06 | $0.76 |
| kimi-k3 | 20 | $0.13 | $0.01 | $1.29 | $2.56 |
| opus-5.5-medium | 20 | $0.21 | $0.10 | $0.56 | $4.23 |
| qwen-next-80B-instruct | 18 | $0.02 | $0.00 | $0.13 | $0.44 |
| sonnet-5.5-medium | 20 | $0.08 | $0.04 | $0.11 | $1.50 |
Model Latency Comparison¶
| Model | Avg (s) | Min (s) | Max (s) | P50 (s) | P95 (s) |
|---|---|---|---|---|---|
| deepseek-v4-pro | 156.4 | 11.5 | 1271.2 | 56.0 | 1271.2 |
| gemini-3.1-pro-preview | 86.6 | 15.3 | 642.4 | 36.2 | 642.4 |
| glm-5.3 | 37.6 | 11.3 | 184.6 | 24.2 | 184.6 |
| gpt-5.5 | 48.1 | 12.2 | 162.2 | 44.9 | 162.2 |
| gpt-6.1-sol-high | 77.5 | 12.8 | 189.9 | 68.7 | 189.9 |
| haiku-4.5 | 27.0 | 13.5 | 48.2 | 26.6 | 48.2 |
| kimi-k3 | 67.3 | 13.6 | 488.2 | 39.9 | 488.2 |
| opus-5.5-medium | 40.6 | 14.4 | 221.5 | 27.4 | 221.5 |
| qwen-next-80B-instruct | 102.9 | 12.2 | 614.5 | 24.7 | 614.5 |
| sonnet-5.5-medium | 24.8 | 12.7 | 47.3 | 24.6 | 47.3 |
⚠️ Note: 2 test(s) excluded from latency calculations due to throttling/timeout errors (qwen-next-80B-instruct: 2)
Performance by Tag¶
Success rate by test category and model:
| Tag | deepseek-v4-pro | gemini-3.1-pro-preview | glm-5.3 | gpt-5.5 | gpt-6.1-sol-high | haiku-4.5 | kimi-k3 | opus-5.5-medium | qwen-next-80B-instruct | sonnet-5.5-medium | Warnings |
|---|---|---|---|---|---|---|---|---|---|---|---|
| advanced | 🟡 50% (½) | 🔴 0% (0/2) | 🟡 50% (½) | 🟡 50% (½) | 🟢 100% (2/2) | 🔴 0% (0/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🔴 0% (0/2) | 🟡 50% (½) | |
| benchmark | 🟡 67% (4/6) | 🟡 67% (4/6) | 🟡 83% (⅚) | 🟡 83% (⅚) | 🟢 100% (6/6) | 🟡 50% (3/6) | 🟢 100% (6/6) | 🟢 100% (6/6) | 🟡 17% (⅙) | 🟡 83% (⅚) | |
| context_window | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟡 50% (½) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🔴 0% (0/2) | 🟢 100% (2/2) | |
| counting | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟡 50% (½) | 🟢 100% (2/2) | |
| datetime | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟡 33% (⅓) | 🟢 100% (3/3) | |
| easy | 🟡 86% (6/7) | 🟡 86% (6/7) | 🟡 86% (6/7) | 🟡 71% (5/7) | 🟡 86% (6/7) | 🟡 86% (6/7) | 🟡 86% (6/7) | 🟡 86% (6/7) | 🟡 71% (5/7) | 🟡 86% (6/7) | |
| elasticsearch | 🟢 100% (3/3) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟡 67% (⅔) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🔴 0% (0/3) | 🟢 100% (3/3) | |
| grafana | 🔴 0% (0/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | |
| hard | 🔴 0% (0/2) | 🟡 50% (½) | 🟡 50% (½) | 🟡 50% (½) | 🟢 100% (2/2) | 🟡 50% (½) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟡 50% (½) | 🟡 50% (½) | |
| kubernetes | 🟡 78% (7/9) | 🟡 78% (7/9) | 🟡 89% (8/9) | 🟡 78% (7/9) | 🟡 89% (8/9) | 🟡 78% (7/9) | 🟡 89% (8/9) | 🟡 89% (8/9) | 🟡 44% (4/9) | 🟡 89% (8/9) | |
| logs | 🟡 67% (4/6) | 🟡 50% (3/6) | 🟡 67% (4/6) | 🟡 67% (4/6) | 🟡 83% (⅚) | 🟡 33% (2/6) | 🟡 83% (⅚) | 🟡 83% (⅚) | 🟡 17% (⅙) | 🟡 67% (4/6) | |
| loki | 🟡 50% (½) | 🔴 0% (0/2) | 🟡 50% (½) | 🟡 50% (½) | 🟡 50% (½) | 🔴 0% (0/2) | 🟡 50% (½) | 🟡 50% (½) | 🔴 0% (0/2) | 🟡 50% (½) | |
| medium | 🟢 100% (9/9) | 🟡 78% (7/9) | 🟢 100% (9/9) | 🟢 100% (9/9) | 🟢 100% (9/9) | 🟡 67% (6/9) | 🟡 89% (8/9) | 🟢 100% (9/9) | 🟡 22% (2/9) | 🟢 100% (9/9) | |
| metrics | 🔴 0% (0/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | |
| multi-cluster | 🟢 100% (3/3) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟡 67% (⅔) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🔴 0% (0/3) | 🟢 100% (3/3) | |
| one-test | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🟢 100% (1/1) | |
| port-forward | 🟡 33% (⅓) | 🟡 33% (⅓) | 🟡 67% (⅔) | 🟡 67% (⅔) | 🟡 67% (⅔) | 🟡 33% (⅓) | 🟡 67% (⅔) | 🟡 67% (⅔) | 🟡 33% (⅓) | 🟡 67% (⅔) | |
| question-answer | 🟡 50% (½) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟡 50% (½) | 🟢 100% (2/2) | |
| regression | 🟡 93% (13/14) | 🟡 86% (12/14) | 🟡 93% (13/14) | 🟡 86% (12/14) | 🟡 93% (13/14) | 🟡 86% (12/14) | 🟡 86% (12/14) | 🟡 93% (13/14) | 🟡 50% (7/14) | 🟡 93% (13/14) | |
| skills | 🟡 91% (10/11) | 🟡 91% (10/11) | 🟡 91% (10/11) | 🟡 82% (9/11) | 🟡 91% (10/11) | 🟡 91% (10/11) | 🟡 91% (10/11) | 🟡 91% (10/11) | 🟡 64% (7/11) | 🟡 91% (10/11) | |
| transparency | 🟢 100% (3/3) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟡 67% (⅔) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🔴 0% (0/3) | 🟢 100% (3/3) | |
| Overall | 🟡 85% (17/20) | 🟡 80% (16/20) | 🟡 90% (18/20) | 🟡 85% (17/20) | 🟡 95% (19/20) | 🟡 75% (15/20) | 🟡 90% (18/20) | 🟡 95% (19/20) | 🟡 40% (8/20) | 🟡 90% (18/20) |
Raw Results¶
Status of all evaluations across models. Color coding:
- 🟢 Passing 100% (stable)
- 🟡 Passing 1-99%
- 🔴 Passing 0% (failing)
- 🔧 Mock data failure (missing or invalid test data)
- ⚠️ Setup failure (environment/infrastructure issue)
- ⏱️ Timeout or rate limit error
- ⏭️ Test skipped (e.g., known issue or precondition not met)
Detailed Raw Results¶
| Eval ID | deepseek-v4-pro | gemini-3.1-pro-preview | glm-5.3 | gpt-5.5 | gpt-6.1-sol-high | haiku-4.5 | kimi-k3 | opus-5.5-medium | qwen-next-80B-instruct | sonnet-5.5-medium |
|---|---|---|---|---|---|---|---|---|---|---|
| 09_crashpod 🔗 | 🟢 100% (1/1) / ⏱️ 57.1s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 38.3s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 15.1s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 35.4s / 💰 $0.17 | 🟢 100% (1/1) / ⏱️ 40.5s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 26.5s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 39.9s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 24.6s / 💰 $0.15 | 🔴 0% (0/1) / ⏱️ 380.1s / 💰 $0.13 | 🟢 100% (1/1) / ⏱️ 20.4s / 💰 $0.07 |
| 100a_loki_historical_logs 🔗 | 🟢 100% (1/1) / ⏱️ 297.8s / 💰 $0.09 | 🔴 0% (0/1) / ⏱️ 44.7s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 24.2s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 75.1s / 💰 $0.38 | 🟢 100% (1/1) / ⏱️ 143.2s / 💰 $0.13 | 🔴 0% (0/1) / ⏱️ 29.1s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 93.7s / 💰 $0.17 | 🟢 100% (1/1) / ⏱️ 49.9s / 💰 $0.34 | 🔴 0% (0/1) / ⏱️ 20.1s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 28.4s / 💰 $0.09 |
| 101_loki_historical_logs_pod_deleted 🔗 | 🔴 0% (0/1) / ⏱️ 1271.2s / 💰 $0.92 | 🔴 0% (0/1) / ⏱️ 642.4s | 🔴 0% (0/1) / ⏱️ 184.6s / 💰 $0.16 | 🔴 0% (0/1) / ⏱️ 162.2s / 💰 $0.68 | 🔴 0% (0/1) / ⏱️ 137.1s / 💰 $0.09 | 🔴 0% (0/1) / ⏱️ 33.0s / 💰 $0.04 | 🔴 0% (0/1) / ⏱️ 488.2s / 💰 $1.29 | 🔴 0% (0/1) / ⏱️ 221.5s / 💰 $0.56 | 🔴 0% (0/1) / ⏱️ 21.4s / 💰 $0.01 | 🔴 0% (0/1) / ⏱️ 25.4s / 💰 $0.07 |
| 108_logs_nearby_lines 🔗 | 🔴 0% (0/1) / ⏱️ 148.9s / 💰 $0.11 | 🔴 0% (0/1) / ⏱️ 311.1s | 🔴 0% (0/1) / ⏱️ 22.1s / 💰 $0.01 | 🔴 0% (0/1) / ⏱️ 64.8s / 💰 $0.28 | 🟢 100% (1/1) / ⏱️ 113.7s / 💰 $0.08 | 🔴 0% (0/1) / ⏱️ 35.5s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 61.3s / 💰 $0.10 | 🟢 100% (1/1) / ⏱️ 38.0s / 💰 $0.24 | 🔴 0% (0/1) / ⏱️ 490.0s / 💰 $0.11 | 🔴 0% (0/1) / ⏱️ 26.1s / 💰 $0.08 |
| 112_find_pvcs_by_uuid 🔗 | 🟢 100% (1/1) / ⏱️ 27.3s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 33.6s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 17.9s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 29.1s / 💰 $0.09 | 🟢 100% (1/1) / ⏱️ 57.5s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 19.4s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 32.6s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 22.9s / 💰 $0.14 | 🟢 100% (1/1) / ⏱️ 119.1s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 19.0s / 💰 $0.06 |
| 12_job_crashing 🔗 | 🟢 100% (1/1) / ⏱️ 53.2s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 28.1s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 16.2s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 30.7s / 💰 $0.10 | 🟢 100% (1/1) / ⏱️ 40.4s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 24.4s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 25.9s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 20.2s / 💰 $0.13 | 🟢 100% (1/1) / ⏱️ 22.5s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 19.6s / 💰 $0.06 |
| 179_grafana_big_dashboard_query 🔗 | 🔴 0% (0/1) / ⏱️ 440.1s / 💰 $0.31 | 🟢 100% (1/1) / ⏱️ 33.3s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 14.5s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 27.4s / 💰 $0.12 | 🟢 100% (1/1) / ⏱️ 40.6s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 23.0s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 24.0s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 24.9s / 💰 $0.24 | 🟢 100% (1/1) / ⏱️ 15.8s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 21.5s / 💰 $0.11 |
| 227_count_configmaps_per_namespace[0] 🔗 | 🟢 100% (1/1) / ⏱️ 55.7s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 28.9s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 20.8s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 25.4s / 💰 $0.11 | 🟢 100% (1/1) / ⏱️ 26.3s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 18.0s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 25.4s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 24.5s / 💰 $0.14 | 🔴 0% (0/1) / ⏱️ 22.0s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 25.8s / 💰 $0.10 |
| 243_pod_names_contain_service 🔗 | 🟢 100% (1/1) / ⏱️ 44.1s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 33.0s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 46.6s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 44.9s / 💰 $0.16 | 🟢 100% (1/1) / ⏱️ 49.5s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 28.8s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 29.6s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 24.2s / 💰 $0.15 | 🟢 100% (1/1) / ⏱️ 31.9s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 22.7s / 💰 $0.06 |
| 24_misconfigured_pvc 🔗 | 🟢 100% (1/1) / ⏱️ 49.8s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 35.8s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 15.7s / 💰 $0.01 | 🔴 0% (0/1) / ⏱️ 58.9s / 💰 $0.24 | 🟢 100% (1/1) / ⏱️ 60.3s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 26.6s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 31.9s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 25.8s / 💰 $0.15 | 🟢 100% (1/1) / ⏱️ 24.4s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 20.6s / 💰 $0.07 |
| 254_elasticsearch_dr_test_log_check 🔗 | 🟢 100% (1/1) / ⏱️ 268.2s / 💰 $0.05 | 🔴 0% (0/1) / ⏱️ 53.3s / 💰 $0.09 | 🟢 100% (1/1) / ⏱️ 61.0s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 51.2s / 💰 $0.25 | 🟢 100% (1/1) / ⏱️ 83.3s / 💰 $0.04 | 🔴 0% (0/1) / ⏱️ 32.4s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 70.0s / 💰 $0.10 | 🟢 100% (1/1) / ⏱️ 42.0s / 💰 $0.17 | 🔴 0% (0/1) / ⏱️ 23.2s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 41.3s / 💰 $0.10 |
| 259_wrong_cluster_logs_confusion 🔗 | 🟢 100% (1/1) / ⏱️ 51.1s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 67.0s / 💰 $0.13 | 🟢 100% (1/1) / ⏱️ 30.9s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 49.0s / 💰 $0.25 | 🟢 100% (1/1) / ⏱️ 74.2s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 39.0s / 💰 $0.04 | 🔴 0% (0/1) / ⏱️ 98.6s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 39.5s / 💰 $0.18 | 🔴 0% (0/1) / ⏱️ 36.5s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 28.1s / 💰 $0.07 |
| 260_global_es_remote_cluster_logs 🔗 | 🟢 100% (1/1) / ⏱️ 57.2s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 69.3s / 💰 $0.14 | 🟢 100% (1/1) / ⏱️ 28.4s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 47.0s / 💰 $0.18 | 🟢 100% (1/1) / ⏱️ 105.1s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 48.2s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 59.3s / 💰 $0.09 | 🟢 100% (1/1) / ⏱️ 44.5s / 💰 $0.19 | 🔴 0% (0/1) / ⏱️ 40.7s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 47.3s / 💰 $0.10 |
| 283_todowrite_multistep_audit 🔗 | 🟢 100% (1/1) / ⏱️ 57.9s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 36.0s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 35.9s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 33.2s / 💰 $0.14 | 🟢 100% (1/1) / ⏱️ 68.7s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 29.5s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 28.9s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 25.6s / 💰 $0.15 | 🔴 0% (0/1) / ⏱️ 24.7s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 23.9s / 💰 $0.07 |
| 43_current_datetime_from_prompt 🔗 | 🟢 100% (1/1) / ⏱️ 12.5s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 15.3s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 11.3s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 12.2s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 12.8s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 13.5s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 16.7s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 14.4s / 💰 $0.10 | 🟢 100% (1/1) / ⏱️ 12.2s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 13.3s / 💰 $0.04 |
| 51_logs_summarize_errors 🔗 | 🟢 100% (1/1) / ⏱️ 28.3s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 30.3s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 41.2s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 23.3s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 47.7s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 22.3s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 37.8s / 💰 $0.09 | 🟢 100% (1/1) / ⏱️ 27.4s / 💰 $0.16 | 🟢 100% (1/1) / ⏱️ 28.0s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 19.2s / 💰 $0.07 |
| 61_exact_match_counting 🔗 | 🟢 100% (1/1) / ⏱️ 11.5s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 17.2s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 11.5s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 13.2s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 16.2s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 18.3s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 13.6s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 14.9s / 💰 $0.11 | 🟢 100% (1/1) / ⏱️ 19.2s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 12.7s / 💰 $0.05 |
| 73a_time_window_anomaly 🔗 | 🟢 100% (1/1) / ⏱️ 103.9s / 💰 $0.12 | 🟢 100% (1/1) / ⏱️ 37.5s / 💰 $0.09 | 🟢 100% (1/1) / ⏱️ 20.8s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 38.5s / 💰 $0.16 | 🟢 100% (1/1) / ⏱️ 118.7s / 💰 $0.10 | 🟢 100% (1/1) / ⏱️ 30.4s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 55.2s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 51.0s / 💰 $0.31 | 🔴 0% (0/1) / ⏱️ 24.7s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 24.6s / 💰 $0.08 |
| 73b_time_window_anomaly 🔗 | 🟢 100% (1/1) / ⏱️ 35.3s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 36.2s / 💰 $0.09 | 🟢 100% (1/1) / ⏱️ 107.8s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 62.7s / 💰 $0.29 | 🟢 100% (1/1) / ⏱️ 123.5s / 💰 $0.11 | 🔴 0% (0/1) / ⏱️ 24.1s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 52.5s / 💰 $0.12 | 🟢 100% (1/1) / ⏱️ 37.2s / 💰 $0.38 | 🔴 0% (0/1) / ⏱️ 614.5s | 🟢 100% (1/1) / ⏱️ 27.7s / 💰 $0.08 |
| 96_no_matching_skill 🔗 | 🟢 100% (1/1) / ⏱️ 56.0s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 141.3s / 💰 $0.38 | 🟢 100% (1/1) / ⏱️ 25.9s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 78.1s / 💰 $0.61 | 🟢 100% (1/1) / ⏱️ 189.9s / 💰 $0.14 | 🟢 100% (1/1) / ⏱️ 17.7s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 61.0s / 💰 $0.13 | 🟢 100% (1/1) / ⏱️ 39.2s / 💰 $0.22 | 🔴 0% (0/1) / ⏱️ 701.1s | 🟢 100% (1/1) / ⏱️ 28.1s / 💰 $0.08 |
Results are automatically generated and updated weekly. View full traces and detailed analysis in Braintrust experiment: ci-benchmark-39553316914.