⚡ October 10, 2026¶
Generated: 2026-10-10 21:37 UTC
Total Duration: 1h 47m 31s
Iterations: 3
Judge (classifier) model: gpt-4.1
Fast Benchmark
Markers: regression or benchmark
Schedule: Weekly (Sunday 2 AM UTC)
Purpose: Quick regression tests to catch breaking changes
HolmesGPT is continuously evaluated against real-world Kubernetes and cloud troubleshooting scenarios.
If you find scenarios that HolmesGPT does not perform well on, please consider adding them as evals to the benchmark.
Model Accuracy Comparison¶
| Model | Pass | Fail | Skip/Error | Total | Success Rate |
|---|---|---|---|---|---|
| gpt-5.6-sol | 46 | 14 | 0 | 60 | 🟡 77% (46/60) |
| gpt-6.1-sol-high | 54 | 6 | 0 | 60 | 🟡 90% (54/60) |
| gpt-6.1-sol-medium | 56 | 4 | 0 | 60 | 🟡 93% (56/60) |
| haiku-4.5 | 48 | 12 | 0 | 60 | 🟡 80% (48/60) |
| opus-5 | 56 | 4 | 0 | 60 | 🟡 93% (56/60) |
| opus-5.5-high | 56 | 4 | 0 | 60 | 🟡 93% (56/60) |
| opus-5.5-medium | 56 | 4 | 0 | 60 | 🟡 93% (56/60) |
| sonnet-5 | 53 | 7 | 0 | 60 | 🟡 88% (53/60) |
| sonnet-5.5-high | 53 | 7 | 0 | 60 | 🟡 88% (53/60) |
| sonnet-5.5-medium | 53 | 7 | 0 | 60 | 🟡 88% (53/60) |
Model Cost Comparison¶
| Model | Tests | Avg Cost | Min Cost | Max Cost | Total Cost |
|---|---|---|---|---|---|
| gpt-5.6-sol | 58 | $0.12 | $0.00 | $0.27 | $6.92 |
| gpt-6.1-sol-high | 60 | $0.05 | $0.00 | $0.13 | $3.13 |
| gpt-6.1-sol-medium | 60 | $0.04 | $0.00 | $0.09 | $2.22 |
| haiku-4.5 | 60 | $0.04 | $0.02 | $0.09 | $2.36 |
| opus-5 | 60 | $0.40 | $0.11 | $1.59 | $23.77 |
| opus-5.5-high | 60 | $0.22 | $0.09 | $0.77 | $13.34 |
| opus-5.5-medium | 60 | $0.19 | $0.00 | $0.42 | $11.31 |
| sonnet-5 | 60 | $0.11 | $0.04 | $0.31 | $6.49 |
| sonnet-5.5-high | 60 | $0.08 | $0.04 | $0.17 | $4.70 |
| sonnet-5.5-medium | 60 | $0.06 | $0.00 | $0.12 | $3.80 |
Model Latency Comparison¶
| Model | Avg (s) | Min (s) | Max (s) | P50 (s) | P95 (s) |
|---|---|---|---|---|---|
| gpt-5.6-sol | 50.7 | 10.2 | 322.2 | 36.4 | 170.7 |
| gpt-6.1-sol-high | 50.2 | 11.8 | 105.6 | 45.5 | 98.3 |
| gpt-6.1-sol-medium | 35.8 | 11.4 | 77.0 | 38.5 | 60.1 |
| haiku-4.5 | 26.2 | 10.8 | 57.4 | 23.7 | 51.5 |
| opus-5 | 71.4 | 10.2 | 359.7 | 46.6 | 306.0 |
| opus-5.5-high | 55.6 | 10.7 | 273.8 | 40.9 | 234.3 |
| opus-5.5-medium | 47.6 | 10.4 | 251.9 | 40.8 | 117.3 |
| sonnet-5 | 39.6 | 10.5 | 131.8 | 28.1 | 99.4 |
| sonnet-5.5-high | 22.7 | 9.8 | 46.0 | 21.2 | 37.0 |
| sonnet-5.5-medium | 18.6 | 9.8 | 30.0 | 17.9 | 28.0 |
Performance by Tag¶
Success rate by test category and model:
| Tag | gpt-5.6-sol | gpt-6.1-sol-high | gpt-6.1-sol-medium | haiku-4.5 | opus-5 | opus-5.5-high | opus-5.5-medium | sonnet-5 | sonnet-5.5-high | sonnet-5.5-medium | Warnings |
|---|---|---|---|---|---|---|---|---|---|---|---|
| advanced | 🟡 33% (2/6) | 🟡 67% (4/6) | 🟡 67% (4/6) | 🟡 33% (2/6) | 🟡 67% (4/6) | 🟡 67% (4/6) | 🟡 67% (4/6) | 🟡 17% (⅙) | 🟡 17% (⅙) | 🟡 17% (⅙) | |
| benchmark | 🟡 67% (12/18) | 🟡 83% (15/18) | 🟡 89% (16/18) | 🟡 61% (11/18) | 🟡 89% (16/18) | 🟡 89% (16/18) | 🟡 89% (16/18) | 🟡 72% (13/18) | 🟡 72% (13/18) | 🟡 72% (13/18) | |
| context_window | 🟡 67% (4/6) | 🟡 83% (⅚) | 🟢 100% (6/6) | 🟡 50% (3/6) | 🟢 100% (6/6) | 🟢 100% (6/6) | 🟢 100% (6/6) | 🟢 100% (6/6) | 🟢 100% (6/6) | 🟢 100% (6/6) | |
| counting | 🟢 100% (6/6) | 🟢 100% (6/6) | 🟢 100% (6/6) | 🟡 83% (⅚) | 🟢 100% (6/6) | 🟢 100% (6/6) | 🟢 100% (6/6) | 🟢 100% (6/6) | 🟢 100% (6/6) | 🟢 100% (6/6) | |
| datetime | 🟡 78% (7/9) | 🟡 89% (8/9) | 🟢 100% (9/9) | 🟡 67% (6/9) | 🟢 100% (9/9) | 🟢 100% (9/9) | 🟢 100% (9/9) | 🟢 100% (9/9) | 🟢 100% (9/9) | 🟢 100% (9/9) | |
| easy | 🟡 71% (15/21) | 🟡 86% (18/21) | 🟡 90% (19/21) | 🟡 81% (17/21) | 🟡 90% (19/21) | 🟡 90% (19/21) | 🟡 90% (19/21) | 🟡 90% (19/21) | 🟡 90% (19/21) | 🟡 90% (19/21) | |
| elasticsearch | 🟡 89% (8/9) | 🟢 100% (9/9) | 🟢 100% (9/9) | 🟡 89% (8/9) | 🟢 100% (9/9) | 🟢 100% (9/9) | 🟢 100% (9/9) | 🟢 100% (9/9) | 🟢 100% (9/9) | 🟢 100% (9/9) | |
| grafana | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | |
| hard | 🟡 67% (4/6) | 🟢 100% (6/6) | 🟢 100% (6/6) | 🟡 83% (⅚) | 🟢 100% (6/6) | 🟢 100% (6/6) | 🟢 100% (6/6) | 🟡 50% (3/6) | 🟡 50% (3/6) | 🟡 50% (3/6) | |
| kubernetes | 🟡 70% (19/27) | 🟡 81% (22/27) | 🟡 85% (23/27) | 🟡 81% (22/27) | 🟡 85% (23/27) | 🟡 85% (23/27) | 🟡 85% (23/27) | 🟡 85% (23/27) | 🟡 85% (23/27) | 🟡 85% (23/27) | |
| logs | 🟡 50% (9/18) | 🟡 72% (13/18) | 🟡 78% (14/18) | 🟡 44% (8/18) | 🟡 78% (14/18) | 🟡 78% (14/18) | 🟡 78% (14/18) | 🟡 61% (11/18) | 🟡 61% (11/18) | 🟡 61% (11/18) | |
| loki | 🟡 33% (2/6) | 🟡 33% (2/6) | 🟡 33% (2/6) | 🟡 17% (⅙) | 🟡 33% (2/6) | 🟡 33% (2/6) | 🟡 33% (2/6) | 🟡 33% (2/6) | 🟡 33% (2/6) | 🟡 33% (2/6) | |
| medium | 🟡 78% (21/27) | 🟡 89% (24/27) | 🟡 93% (25/27) | 🟡 74% (20/27) | 🟡 93% (25/27) | 🟡 93% (25/27) | 🟡 93% (25/27) | 🟡 93% (25/27) | 🟡 93% (25/27) | 🟡 93% (25/27) | |
| metrics | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | |
| multi-cluster | 🟡 89% (8/9) | 🟢 100% (9/9) | 🟢 100% (9/9) | 🟡 89% (8/9) | 🟢 100% (9/9) | 🟢 100% (9/9) | 🟢 100% (9/9) | 🟢 100% (9/9) | 🟢 100% (9/9) | 🟢 100% (9/9) | |
| one-test | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | |
| port-forward | 🟡 56% (5/9) | 🟡 56% (5/9) | 🟡 56% (5/9) | 🟡 44% (4/9) | 🟡 56% (5/9) | 🟡 56% (5/9) | 🟡 56% (5/9) | 🟡 56% (5/9) | 🟡 56% (5/9) | 🟡 56% (5/9) | |
| question-answer | 🟢 100% (6/6) | 🟢 100% (6/6) | 🟢 100% (6/6) | 🟢 100% (6/6) | 🟢 100% (6/6) | 🟢 100% (6/6) | 🟢 100% (6/6) | 🟢 100% (6/6) | 🟢 100% (6/6) | 🟢 100% (6/6) | |
| regression | 🟡 81% (34/42) | 🟡 93% (39/42) | 🟡 95% (40/42) | 🟡 88% (37/42) | 🟡 95% (40/42) | 🟡 95% (40/42) | 🟡 95% (40/42) | 🟡 95% (40/42) | 🟡 95% (40/42) | 🟡 95% (40/42) | |
| skills | 🟡 79% (26/33) | 🟡 91% (30/33) | 🟡 94% (31/33) | 🟡 88% (29/33) | 🟡 94% (31/33) | 🟡 94% (31/33) | 🟡 94% (31/33) | 🟡 94% (31/33) | 🟡 94% (31/33) | 🟡 94% (31/33) | |
| transparency | 🟡 89% (8/9) | 🟢 100% (9/9) | 🟢 100% (9/9) | 🟡 89% (8/9) | 🟢 100% (9/9) | 🟢 100% (9/9) | 🟢 100% (9/9) | 🟢 100% (9/9) | 🟢 100% (9/9) | 🟢 100% (9/9) | |
| Overall | 🟡 77% (46/60) | 🟡 90% (54/60) | 🟡 93% (56/60) | 🟡 80% (48/60) | 🟡 93% (56/60) | 🟡 93% (56/60) | 🟡 93% (56/60) | 🟡 88% (53/60) | 🟡 88% (53/60) | 🟡 88% (53/60) |
Raw Results¶
Status of all evaluations across models. Color coding:
- 🟢 Passing 100% (stable)
- 🟡 Passing 1-99%
- 🔴 Passing 0% (failing)
- 🔧 Mock data failure (missing or invalid test data)
- ⚠️ Setup failure (environment/infrastructure issue)
- ⏱️ Timeout or rate limit error
- ⏭️ Test skipped (e.g., known issue or precondition not met)
Detailed Raw Results¶
| Eval ID | gpt-5.6-sol | gpt-6.1-sol-high | gpt-6.1-sol-medium | haiku-4.5 | opus-5 | opus-5.5-high | opus-5.5-medium | sonnet-5 | sonnet-5.5-high | sonnet-5.5-medium |
|---|---|---|---|---|---|---|---|---|---|---|
| 09_crashpod 🔗 | 🟢 100% (3/3) / ⏱️ 33.8s / 💰 $0.11 | 🟢 100% (3/3) / ⏱️ 32.1s / 💰 $0.03 | 🟢 100% (3/3) / ⏱️ 24.7s / 💰 $0.02 | 🟢 100% (3/3) / ⏱️ 23.8s / 💰 $0.04 | 🟢 100% (3/3) / ⏱️ 41.4s / 💰 $0.27 | 🟢 100% (3/3) / ⏱️ 37.5s / 💰 $0.16 | 🟢 100% (3/3) / ⏱️ 25.2s / 💰 $0.14 | 🟢 100% (3/3) / ⏱️ 24.7s / 💰 $0.08 | 🟢 100% (3/3) / ⏱️ 19.4s / 💰 $0.06 | 🟢 100% (3/3) / ⏱️ 17.4s / 💰 $0.06 |
| 100a_loki_historical_logs 🔗 | 🟡 33% (⅓) / ⏱️ 77.1s / 💰 $0.25 | 🟡 33% (⅓) / ⏱️ 97.6s / 💰 $0.11 | 🟡 33% (⅓) / ⏱️ 67.1s / 💰 $0.07 | 🔴 0% (0/3) / ⏱️ 26.0s / 💰 $0.04 | 🟡 33% (⅓) / ⏱️ 268.3s / 💰 $1.01 | 🟡 33% (⅓) / ⏱️ 171.4s / 💰 $0.51 | 🟡 33% (⅓) / ⏱️ 90.8s / 💰 $0.36 | 🟡 33% (⅓) / ⏱️ 96.2s / 💰 $0.19 | 🟡 33% (⅓) / ⏱️ 35.4s / 💰 $0.13 | 🟡 33% (⅓) / ⏱️ 25.2s / 💰 $0.11 |
| 101_loki_historical_logs_pod_deleted 🔗 | 🟡 33% (⅓) / ⏱️ 67.0s / 💰 $0.20 | 🟡 33% (⅓) / ⏱️ 75.0s / 💰 $0.06 | 🟡 33% (⅓) / ⏱️ 41.3s / 💰 $0.02 | 🟡 33% (⅓) / ⏱️ 27.9s / 💰 $0.04 | 🟡 33% (⅓) / ⏱️ 207.5s / 💰 $0.93 | 🟡 33% (⅓) / ⏱️ 202.2s / 💰 $0.59 | 🟡 33% (⅓) / ⏱️ 180.7s / 💰 $0.34 | 🟡 33% (⅓) / ⏱️ 66.1s / 💰 $0.14 | 🟡 33% (⅓) / ⏱️ 26.5s / 💰 $0.08 | 🟡 33% (⅓) / ⏱️ 17.9s / 💰 $0.06 |
| 108_logs_nearby_lines 🔗 | 🟡 33% (⅓) / ⏱️ 55.0s / 💰 $0.22 | 🟢 100% (3/3) / ⏱️ 75.9s / 💰 $0.08 | 🟢 100% (3/3) / ⏱️ 49.0s / 💰 $0.06 | 🟡 67% (⅔) / ⏱️ 45.4s / 💰 $0.08 | 🟢 100% (3/3) / ⏱️ 66.4s / 💰 $0.42 | 🟢 100% (3/3) / ⏱️ 41.8s / 💰 $0.22 | 🟢 100% (3/3) / ⏱️ 45.8s / 💰 $0.21 | 🔴 0% (0/3) / ⏱️ 40.0s / 💰 $0.12 | 🔴 0% (0/3) / ⏱️ 27.4s / 💰 $0.09 | 🔴 0% (0/3) / ⏱️ 22.0s / 💰 $0.07 |
| 112_find_pvcs_by_uuid 🔗 | 🟢 100% (3/3) / ⏱️ 41.9s / 💰 $0.04 | 🟢 100% (3/3) / ⏱️ 30.9s / 💰 $0.02 | 🟢 100% (3/3) / ⏱️ 28.2s / 💰 $0.02 | 🟢 100% (3/3) / ⏱️ 17.7s / 💰 $0.03 | 🟢 100% (3/3) / ⏱️ 33.0s / 💰 $0.24 | 🟢 100% (3/3) / ⏱️ 46.7s / 💰 $0.18 | 🟢 100% (3/3) / ⏱️ 33.1s / 💰 $0.14 | 🟢 100% (3/3) / ⏱️ 18.6s / 💰 $0.06 | 🟢 100% (3/3) / ⏱️ 16.9s / 💰 $0.06 | 🟢 100% (3/3) / ⏱️ 16.1s / 💰 $0.06 |
| 12_job_crashing 🔗 | 🟢 100% (3/3) / ⏱️ 28.9s / 💰 $0.08 | 🟢 100% (3/3) / ⏱️ 30.2s / 💰 $0.03 | 🟢 100% (3/3) / ⏱️ 24.9s / 💰 $0.03 | 🟢 100% (3/3) / ⏱️ 23.1s / 💰 $0.04 | 🟢 100% (3/3) / ⏱️ 34.4s / 💰 $0.24 | 🟢 100% (3/3) / ⏱️ 36.2s / 💰 $0.14 | 🟢 100% (3/3) / ⏱️ 41.4s / 💰 $0.13 | 🟢 100% (3/3) / ⏱️ 25.3s / 💰 $0.08 | 🟢 100% (3/3) / ⏱️ 18.7s / 💰 $0.06 | 🟢 100% (3/3) / ⏱️ 14.7s / 💰 $0.05 |
| 179_grafana_big_dashboard_query 🔗 | 🟢 100% (3/3) / ⏱️ 20.7s / 💰 $0.07 | 🟢 100% (3/3) / ⏱️ 33.5s / 💰 $0.08 | 🟢 100% (3/3) / ⏱️ 26.4s / 💰 $0.08 | 🟢 100% (3/3) / ⏱️ 17.0s / 💰 $0.03 | 🟢 100% (3/3) / ⏱️ 41.1s / 💰 $0.39 | 🟢 100% (3/3) / ⏱️ 26.3s / 💰 $0.26 | 🟢 100% (3/3) / ⏱️ 24.1s / 💰 $0.26 | 🟢 100% (3/3) / ⏱️ 18.2s / 💰 $0.08 | 🟢 100% (3/3) / ⏱️ 18.1s / 💰 $0.10 | 🟢 100% (3/3) / ⏱️ 15.4s / 💰 $0.08 |
| 227_count_configmaps_per_namespace[0] 🔗 | 🟢 100% (3/3) / ⏱️ 21.1s / 💰 $0.04 | 🟢 100% (3/3) / ⏱️ 23.6s / 💰 $0.01 | 🟢 100% (3/3) / ⏱️ 20.3s / 💰 $0.01 | 🟢 100% (3/3) / ⏱️ 16.6s / 💰 $0.03 | 🟢 100% (3/3) / ⏱️ 29.3s / 💰 $0.25 | 🟢 100% (3/3) / ⏱️ 21.9s / 💰 $0.12 | 🟢 100% (3/3) / ⏱️ 21.0s / 💰 $0.12 | 🟢 100% (3/3) / ⏱️ 24.0s / 💰 $0.12 | 🟢 100% (3/3) / ⏱️ 19.1s / 💰 $0.07 | 🟢 100% (3/3) / ⏱️ 14.5s / 💰 $0.05 |
| 243_pod_names_contain_service 🔗 | 🟡 67% (⅔) / ⏱️ 31.0s / 💰 $0.08 | 🟢 100% (3/3) / ⏱️ 32.7s / 💰 $0.03 | 🟢 100% (3/3) / ⏱️ 24.3s / 💰 $0.02 | 🟢 100% (3/3) / ⏱️ 23.0s / 💰 $0.03 | 🟢 100% (3/3) / ⏱️ 38.8s / 💰 $0.23 | 🟢 100% (3/3) / ⏱️ 38.6s / 💰 $0.15 | 🟢 100% (3/3) / ⏱️ 37.0s / 💰 $0.14 | 🟢 100% (3/3) / ⏱️ 27.0s / 💰 $0.08 | 🟢 100% (3/3) / ⏱️ 18.9s / 💰 $0.07 | 🟢 100% (3/3) / ⏱️ 16.4s / 💰 $0.06 |
| 24_misconfigured_pvc 🔗 | 🔴 0% (0/3) / ⏱️ 128.9s / 💰 $0.06 | 🟡 67% (⅔) / ⏱️ 53.6s / 💰 $0.04 | 🟢 100% (3/3) / ⏱️ 33.3s / 💰 $0.03 | 🟢 100% (3/3) / ⏱️ 22.7s / 💰 $0.03 | 🟢 100% (3/3) / ⏱️ 43.7s / 💰 $0.27 | 🟢 100% (3/3) / ⏱️ 41.6s / 💰 $0.15 | 🟢 100% (3/3) / ⏱️ 40.2s / 💰 $0.14 | 🟢 100% (3/3) / ⏱️ 27.2s / 💰 $0.08 | 🟢 100% (3/3) / ⏱️ 19.8s / 💰 $0.07 | 🟢 100% (3/3) / ⏱️ 17.7s / 💰 $0.06 |
| 254_elasticsearch_dr_test_log_check 🔗 | 🟡 67% (⅔) / ⏱️ 133.6s / 💰 $0.08 | 🟢 100% (3/3) / ⏱️ 50.0s / 💰 $0.05 | 🟢 100% (3/3) / ⏱️ 41.2s / 💰 $0.04 | 🟡 67% (⅔) / ⏱️ 37.5s / 💰 $0.05 | 🟢 100% (3/3) / ⏱️ 85.7s / 💰 $0.38 | 🟢 100% (3/3) / ⏱️ 58.0s / 💰 $0.18 | 🟢 100% (3/3) / ⏱️ 75.5s / 💰 $0.19 | 🟢 100% (3/3) / ⏱️ 52.3s / 💰 $0.12 | 🟢 100% (3/3) / ⏱️ 29.2s / 💰 $0.08 | 🟢 100% (3/3) / ⏱️ 26.6s / 💰 $0.07 |
| 259_wrong_cluster_logs_confusion 🔗 | 🟢 100% (3/3) / ⏱️ 38.0s / 💰 $0.11 | 🟢 100% (3/3) / ⏱️ 45.2s / 💰 $0.04 | 🟢 100% (3/3) / ⏱️ 39.2s / 💰 $0.03 | 🟢 100% (3/3) / ⏱️ 47.9s / 💰 $0.06 | 🟢 100% (3/3) / ⏱️ 104.9s / 💰 $0.54 | 🟢 100% (3/3) / ⏱️ 58.7s / 💰 $0.21 | 🟢 100% (3/3) / ⏱️ 49.6s / 💰 $0.19 | 🟢 100% (3/3) / ⏱️ 70.0s / 💰 $0.19 | 🟢 100% (3/3) / ⏱️ 38.8s / 💰 $0.10 | 🟢 100% (3/3) / ⏱️ 25.5s / 💰 $0.07 |
| 260_global_es_remote_cluster_logs 🔗 | 🟢 100% (3/3) / ⏱️ 44.4s / 💰 $0.12 | 🟢 100% (3/3) / ⏱️ 56.3s / 💰 $0.03 | 🟢 100% (3/3) / ⏱️ 48.1s / 💰 $0.03 | 🟢 100% (3/3) / ⏱️ 44.1s / 💰 $0.06 | 🟢 100% (3/3) / ⏱️ 78.8s / 💰 $0.36 | 🟢 100% (3/3) / ⏱️ 66.1s / 💰 $0.18 | 🟢 100% (3/3) / ⏱️ 45.0s / 💰 $0.17 | 🟢 100% (3/3) / ⏱️ 54.8s / 💰 $0.12 | 🟢 100% (3/3) / ⏱️ 31.7s / 💰 $0.09 | 🟢 100% (3/3) / ⏱️ 26.2s / 💰 $0.07 |
| 283_todowrite_multistep_audit 🔗 | 🟢 100% (3/3) / ⏱️ 82.1s / 💰 $0.12 | 🟢 100% (3/3) / ⏱️ 54.6s / 💰 $0.06 | 🟢 100% (3/3) / ⏱️ 46.3s / 💰 $0.04 | 🟢 100% (3/3) / ⏱️ 24.5s / 💰 $0.04 | 🟢 100% (3/3) / ⏱️ 29.9s / 💰 $0.21 | 🟢 100% (3/3) / ⏱️ 29.2s / 💰 $0.14 | 🟢 100% (3/3) / ⏱️ 30.3s / 💰 $0.13 | 🟢 100% (3/3) / ⏱️ 24.6s / 💰 $0.08 | 🟢 100% (3/3) / ⏱️ 16.4s / 💰 $0.06 | 🟢 100% (3/3) / ⏱️ 14.1s / 💰 $0.05 |
| 43_current_datetime_from_prompt 🔗 | 🟢 100% (3/3) / ⏱️ 10.7s / 💰 $0.01 | 🟢 100% (3/3) / ⏱️ 12.0s / 💰 $0.00 | 🟢 100% (3/3) / ⏱️ 12.1s / 💰 $0.00 | 🟢 100% (3/3) / ⏱️ 11.1s / 💰 $0.02 | 🟢 100% (3/3) / ⏱️ 10.5s / 💰 $0.11 | 🟢 100% (3/3) / ⏱️ 13.3s / 💰 $0.09 | 🟢 100% (3/3) / ⏱️ 11.9s / 💰 $0.01 | 🟢 100% (3/3) / ⏱️ 10.7s / 💰 $0.04 | 🟢 100% (3/3) / ⏱️ 9.9s / 💰 $0.04 | 🟢 100% (3/3) / ⏱️ 10.1s / 💰 $0.00 |
| 51_logs_summarize_errors 🔗 | 🟡 67% (⅔) / ⏱️ 26.0s / 💰 $0.06 | 🟢 100% (3/3) / ⏱️ 31.4s / 💰 $0.02 | 🟢 100% (3/3) / ⏱️ 22.8s / 💰 $0.01 | 🟡 67% (⅔) / ⏱️ 21.3s / 💰 $0.03 | 🟢 100% (3/3) / ⏱️ 32.9s / 💰 $0.25 | 🟢 100% (3/3) / ⏱️ 34.1s / 💰 $0.16 | 🟢 100% (3/3) / ⏱️ 36.1s / 💰 $0.16 | 🟢 100% (3/3) / ⏱️ 23.4s / 💰 $0.07 | 🟢 100% (3/3) / ⏱️ 16.8s / 💰 $0.06 | 🟢 100% (3/3) / ⏱️ 15.6s / 💰 $0.06 |
| 61_exact_match_counting 🔗 | 🟢 100% (3/3) / ⏱️ 12.7s / 💰 $0.01 | 🟢 100% (3/3) / ⏱️ 16.0s / 💰 $0.00 | 🟢 100% (3/3) / ⏱️ 15.1s / 💰 $0.00 | 🟡 67% (⅔) / ⏱️ 13.8s / 💰 $0.02 | 🟢 100% (3/3) / ⏱️ 14.8s / 💰 $0.14 | 🟢 100% (3/3) / ⏱️ 15.3s / 💰 $0.11 | 🟢 100% (3/3) / ⏱️ 17.6s / 💰 $0.10 | 🟢 100% (3/3) / ⏱️ 13.2s / 💰 $0.05 | 🟢 100% (3/3) / ⏱️ 12.4s / 💰 $0.05 | 🟢 100% (3/3) / ⏱️ 11.8s / 💰 $0.05 |
| 73a_time_window_anomaly 🔗 | 🟡 67% (⅔) / ⏱️ 49.6s / 💰 $0.21 | 🟡 67% (⅔) / ⏱️ 82.6s / 💰 $0.11 | 🟢 100% (3/3) / ⏱️ 49.9s / 💰 $0.08 | 🟡 33% (⅓) / ⏱️ 23.7s / 💰 $0.04 | 🟢 100% (3/3) / ⏱️ 90.9s / 💰 $0.56 | 🟢 100% (3/3) / ⏱️ 63.7s / 💰 $0.30 | 🟢 100% (3/3) / ⏱️ 59.8s / 💰 $0.29 | 🟢 100% (3/3) / ⏱️ 39.2s / 💰 $0.12 | 🟢 100% (3/3) / ⏱️ 22.8s / 💰 $0.08 | 🟢 100% (3/3) / ⏱️ 19.4s / 💰 $0.07 |
| 73b_time_window_anomaly 🔗 | 🟡 67% (⅔) / ⏱️ 57.5s / 💰 $0.21 | 🟢 100% (3/3) / ⏱️ 78.7s / 💰 $0.11 | 🟢 100% (3/3) / ⏱️ 48.6s / 💰 $0.08 | 🟡 67% (⅔) / ⏱️ 24.3s / 💰 $0.04 | 🟢 100% (3/3) / ⏱️ 99.7s / 💰 $0.69 | 🟢 100% (3/3) / ⏱️ 52.4s / 💰 $0.31 | 🟢 100% (3/3) / ⏱️ 41.7s / 💰 $0.30 | 🟢 100% (3/3) / ⏱️ 60.7s / 💰 $0.14 | 🟢 100% (3/3) / ⏱️ 25.8s / 💰 $0.10 | 🟢 100% (3/3) / ⏱️ 20.5s / 💰 $0.07 |
| 96_no_matching_skill 🔗 | 🟢 100% (3/3) / ⏱️ 54.4s / 💰 $0.23 | 🟢 100% (3/3) / ⏱️ 92.0s / 💰 $0.12 | 🟢 100% (3/3) / ⏱️ 54.1s / 💰 $0.08 | 🟢 100% (3/3) / ⏱️ 33.5s / 💰 $0.06 | 🟢 100% (3/3) / ⏱️ 75.2s / 💰 $0.43 | 🟢 100% (3/3) / ⏱️ 56.4s / 💰 $0.27 | 🟢 100% (3/3) / ⏱️ 44.9s / 💰 $0.25 | 🟢 100% (3/3) / ⏱️ 76.3s / 💰 $0.20 | 🟢 100% (3/3) / ⏱️ 30.6s / 💰 $0.12 | 🟢 100% (3/3) / ⏱️ 24.8s / 💰 $0.08 |
Results are automatically generated and updated weekly. View full traces and detailed analysis in Braintrust experiment: ci-benchmark-38081085828.