⚡ November 04, 2026¶
Generated: 2026-11-04 15:04 UTC
Total Duration: 54m 49s
Iterations: 1
Judge (classifier) model: gpt-6.1-sol-medium
Fast Benchmark
Markers: regression or benchmark
Schedule: Weekly (Sunday 2 AM UTC)
Purpose: Quick regression tests to catch breaking changes
HolmesGPT is continuously evaluated against real-world Kubernetes and cloud troubleshooting scenarios.
If you find scenarios that HolmesGPT does not perform well on, please consider adding them as evals to the benchmark.
Model Accuracy Comparison¶
| Model | Pass | Fail | Skip/Error | Total | Success Rate |
|---|---|---|---|---|---|
| deepseek-v4-pro | 19 | 3 | 0 | 22 | 🟡 86% (19/22) |
| gemini-3.1-pro-preview | 17 | 5 | 0 | 22 | 🟡 77% (17/22) |
| glm-5.3 | 16 | 6 | 0 | 22 | 🟡 73% (16/22) |
| gpt-6.1-sol-high | 22 | 0 | 0 | 22 | 🟢 100% (22/22) |
| haiku-4.5 | 15 | 7 | 0 | 22 | 🟡 68% (15/22) |
| kimi-k3 | 22 | 0 | 0 | 22 | 🟢 100% (22/22) |
| opus-5.5-medium | 21 | 1 | 0 | 22 | 🟡 95% (21/22) |
| sonnet-5.5-medium | 19 | 3 | 0 | 22 | 🟡 86% (19/22) |
Skill-Suggestion Check Failures¶
These runs failed a deterministic skill-suggestion check, whatever the judge decided about the answer. They count as failures in the success rates above.
| Eval | Model | Check | Judge | Suggested skill(s) | |
|---|---|---|---|---|---|
| 🧩 | 273_skill_suggestion_prometheus | deepseek-v4-pro | suggested no skill, expected one | wrong | — |
| 🧩 | 12_job_crashing | glm-5.3 | suggested a skill, expected none | correct | Pod log fetching quirks in this environment |
| 🧩 | 227_count_configmaps_per_namespace[0] | glm-5.3 | suggested a skill, expected none | correct | Kubernetes query tool quirks in this environment |
| 🧩 | 51_logs_summarize_errors | glm-5.3 | suggested a skill, expected none | correct | Workload naming and log conventions in this cluster |
| 🧩 | 61_exact_match_counting | glm-5.3 | suggested a skill, expected none | correct | Kubernetes query tool null-selector pitfalls in this environment |
| 🧩 | 271_skill_suggestion_elasticsearch | haiku-4.5 | suggested no skill, expected one | wrong | — |
| 🧩 | 273_skill_suggestion_prometheus | haiku-4.5 | suggested no skill, expected one | wrong | — |
| 🧩 | 271_skill_suggestion_elasticsearch | sonnet-5.5-medium | suggested no skill, expected one | wrong | — |
| 🧩 | 273_skill_suggestion_prometheus | sonnet-5.5-medium | suggested no skill, expected one | wrong | — |
Model Cost Comparison¶
| Model | Tests | Avg Cost | Min Cost | Max Cost | Total Cost |
|---|---|---|---|---|---|
| deepseek-v4-pro | 22 | $0.04 | $0.00 | $0.14 | $0.93 |
| gemini-3.1-pro-preview | 22 | $0.10 | $0.03 | $0.24 | $2.09 |
| glm-5.3 | 22 | $0.09 | $0.00 | $0.33 | $1.92 |
| gpt-6.1-sol-high | 22 | $0.06 | $0.00 | $0.14 | $1.28 |
| haiku-4.5 | 22 | $0.04 | $0.02 | $0.11 | $0.94 |
| kimi-k3 | 22 | $0.12 | $0.04 | $0.31 | $2.54 |
| opus-5.5-medium | 22 | $0.20 | $0.10 | $0.45 | $4.36 |
| sonnet-5.5-medium | 22 | $0.08 | $0.04 | $0.24 | $1.71 |
Model Latency Comparison¶
| Model | Avg (s) | Min (s) | Max (s) | P50 (s) | P95 (s) |
|---|---|---|---|---|---|
| deepseek-v4-pro | 55.6 | 11.2 | 165.9 | 50.9 | 116.4 |
| gemini-3.1-pro-preview | 54.3 | 15.6 | 115.7 | 50.3 | 110.2 |
| glm-5.3 | 138.5 | 18.2 | 424.6 | 126.7 | 237.9 |
| gpt-6.1-sol-high | 57.4 | 14.9 | 123.7 | 61.1 | 108.0 |
| haiku-4.5 | 24.4 | 11.6 | 37.6 | 23.3 | 36.6 |
| kimi-k3 | 65.5 | 16.2 | 132.8 | 67.1 | 132.8 |
| opus-5.5-medium | 33.3 | 13.1 | 58.3 | 36.8 | 49.1 |
| sonnet-5.5-medium | 23.8 | 13.0 | 40.4 | 22.6 | 34.1 |
⚠️ Note: 5 test(s) excluded from latency calculations due to throttling/timeout/provider errors (kimi-k3: 5)
Performance by Tag¶
Success rate by test category and model:
| Tag | deepseek-v4-pro | gemini-3.1-pro-preview | glm-5.3 | gpt-6.1-sol-high | haiku-4.5 | kimi-k3 | opus-5.5-medium | sonnet-5.5-medium | Warnings |
|---|---|---|---|---|---|---|---|---|---|
| advanced | 🟡 33% (⅓) | 🟡 33% (⅓) | 🟡 33% (⅓) | 🟢 100% (3/3) | 🔴 0% (0/3) | 🟢 100% (3/3) | 🟡 67% (⅔) | 🟡 33% (⅓) | |
| benchmark | 🟡 62% (⅝) | 🟡 50% (4/8) | 🟡 75% (6/8) | 🟢 100% (8/8) | 🟡 50% (4/8) | 🟢 100% (8/8) | 🟡 88% (⅞) | 🟡 62% (⅝) | |
| context_window | 🟢 100% (2/2) | 🟡 50% (½) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | |
| counting | 🟢 100% (2/2) | 🟢 100% (2/2) | 🔴 0% (0/2) | 🟢 100% (2/2) | 🟡 50% (½) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | |
| datetime | 🟢 100% (3/3) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | |
| easy | 🟢 100% (7/7) | 🟢 100% (7/7) | 🟡 57% (4/7) | 🟢 100% (7/7) | 🟡 71% (5/7) | 🟢 100% (7/7) | 🟢 100% (7/7) | 🟢 100% (7/7) | |
| elasticsearch | 🟢 100% (4/4) | 🟡 75% (¾) | 🟢 100% (4/4) | 🟢 100% (4/4) | 🟡 50% (2/4) | 🟢 100% (4/4) | 🟢 100% (4/4) | 🟡 75% (¾) | |
| fast | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | |
| grafana | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | |
| hard | 🟡 50% (½) | 🟡 50% (½) | 🟡 50% (½) | 🟢 100% (2/2) | 🟡 50% (½) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟡 50% (½) | |
| kubernetes | 🟡 90% (9/10) | 🟡 90% (9/10) | 🟡 80% (8/10) | 🟢 100% (10/10) | 🟡 80% (8/10) | 🟢 100% (10/10) | 🟡 90% (9/10) | 🟡 90% (9/10) | |
| logs | 🟡 86% (6/7) | 🟡 71% (5/7) | 🟡 71% (5/7) | 🟢 100% (7/7) | 🟡 43% (3/7) | 🟢 100% (7/7) | 🟢 100% (7/7) | 🟡 71% (5/7) | |
| loki | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟡 50% (½) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | |
| medium | 🟡 82% (9/11) | 🟡 64% (7/11) | 🟡 91% (10/11) | 🟢 100% (11/11) | 🟡 64% (7/11) | 🟢 100% (11/11) | 🟡 91% (10/11) | 🟡 82% (9/11) | |
| metrics | 🟡 50% (½) | 🟡 50% (½) | 🟡 50% (½) | 🟢 100% (2/2) | 🟡 50% (½) | 🟢 100% (2/2) | 🟡 50% (½) | 🟡 50% (½) | |
| multi-cluster | 🟢 100% (3/3) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | |
| one-test | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | 🟢 100% (1/1) | |
| port-forward | 🟡 75% (¾) | 🟡 75% (¾) | 🟡 75% (¾) | 🟢 100% (4/4) | 🟡 50% (2/4) | 🟢 100% (4/4) | 🟡 75% (¾) | 🟡 75% (¾) | |
| prometheus | 🔴 0% (0/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🟢 100% (1/1) | 🔴 0% (0/1) | 🔴 0% (0/1) | |
| question-answer | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | 🟢 100% (2/2) | |
| regression | 🟢 100% (14/14) | 🟡 93% (13/14) | 🟡 71% (10/14) | 🟢 100% (14/14) | 🟡 79% (11/14) | 🟢 100% (14/14) | 🟢 100% (14/14) | 🟢 100% (14/14) | |
| skills | 🟡 85% (11/13) | 🟡 85% (11/13) | 🟡 62% (8/13) | 🟢 100% (13/13) | 🟡 69% (9/13) | 🟢 100% (13/13) | 🟡 92% (12/13) | 🟡 85% (11/13) | |
| transparency | 🟢 100% (3/3) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟡 67% (⅔) | 🟢 100% (3/3) | 🟢 100% (3/3) | 🟢 100% (3/3) | |
| Overall | 🟡 86% (19/22) | 🟡 77% (17/22) | 🟡 73% (16/22) | 🟢 100% (22/22) | 🟡 68% (15/22) | 🟢 100% (22/22) | 🟡 95% (21/22) | 🟡 86% (19/22) |
Raw Results¶
Status of all evaluations across models. Color coding:
- 🟢 Passing 100% (stable)
- 🟡 Passing 1-99%
- 🔴 Passing 0% (failing)
- 🔧 Mock data failure (missing or invalid test data)
- ⚠️ Setup or model-provider failure (environment/infrastructure issue; the model never answered, so the run is left out of success rates)
- ⏱️ Timeout or rate limit error (left out of success rates)
- ⏭️ Test skipped (e.g., known issue or precondition not met)
- 🧩 Failed the skill-suggestion check (counted as a failure; see Skill-Suggestion Check Failures for the skill and the judge's verdict)
Detailed Raw Results¶
| Eval ID | deepseek-v4-pro | gemini-3.1-pro-preview | glm-5.3 | gpt-6.1-sol-high | haiku-4.5 | kimi-k3 | opus-5.5-medium | sonnet-5.5-medium |
|---|---|---|---|---|---|---|---|---|
| 09_crashpod 🔗 | 🟢 100% (1/1) / ⏱️ 39.3s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 44.4s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 73.4s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 40.2s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 19.3s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 1529.6s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 27.2s / 💰 $0.17 | 🟢 100% (1/1) / ⏱️ 18.6s / 💰 $0.06 |
| 100a_loki_historical_logs 🔗 | 🟢 100% (1/1) / ⏱️ 95.3s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 44.0s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 226.4s / 💰 $0.17 | 🟢 100% (1/1) / ⏱️ 106.5s / 💰 $0.14 | 🔴 0% (0/1) / ⏱️ 27.5s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 132.8s / 💰 $0.26 | 🟢 100% (1/1) / ⏱️ 48.7s / 💰 $0.30 | 🟢 100% (1/1) / ⏱️ 28.8s / 💰 $0.09 |
| 101_loki_historical_logs_pod_deleted 🔗 | 🟢 100% (1/1) / ⏱️ 165.9s / 💰 $0.14 | 🟢 100% (1/1) / ⏱️ 110.2s / 💰 $0.24 | 🟢 100% (1/1) / ⏱️ 157.1s / 💰 $0.10 | 🟢 100% (1/1) / ⏱️ 61.1s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 22.3s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 65.0s / 💰 $0.13 | 🟢 100% (1/1) / ⏱️ 41.4s / 💰 $0.25 | 🟢 100% (1/1) / ⏱️ 26.8s / 💰 $0.06 |
| 108_logs_nearby_lines 🔗 | 🔴 0% (0/1) / ⏱️ 89.8s / 💰 $0.07 | 🔴 0% (0/1) / ⏱️ 59.5s / 💰 $0.12 | 🔴 0% (0/1) / ⏱️ 126.7s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 72.5s / 💰 $0.11 | 🔴 0% (0/1) / ⏱️ 35.8s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 83.8s / 💰 $0.17 | 🟢 100% (1/1) / ⏱️ 38.7s / 💰 $0.20 | 🔴 0% (0/1) / ⏱️ 28.1s / 💰 $0.09 |
| 112_find_pvcs_by_uuid 🔗 | 🟢 100% (1/1) / ⏱️ 21.5s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 39.6s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 124.7s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 36.4s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 15.7s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 41.7s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 21.3s / 💰 $0.14 | 🟢 100% (1/1) / ⏱️ 16.5s / 💰 $0.06 |
| 12_job_crashing 🔗 | 🟢 100% (1/1) / ⏱️ 40.1s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 28.4s / 💰 $0.05 | 🔴🧩 0% (0/1) / ⏱️ 74.9s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 29.5s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 23.3s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 171.8s / 💰 $0.10 | 🟢 100% (1/1) / ⏱️ 21.3s / 💰 $0.13 | 🟢 100% (1/1) / ⏱️ 18.7s / 💰 $0.06 |
| 179_grafana_big_dashboard_query 🔗 | 🟢 100% (1/1) / ⏱️ 28.8s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 33.1s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 424.6s / 💰 $0.33 | 🟢 100% (1/1) / ⏱️ 38.5s / 💰 $0.10 | 🟢 100% (1/1) / ⏱️ 19.2s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 29.6s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 21.3s / 💰 $0.26 | 🟢 100% (1/1) / ⏱️ 19.6s / 💰 $0.11 |
| 227_count_configmaps_per_namespace[0] 🔗 | 🟢 100% (1/1) / ⏱️ 23.5s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 36.9s / 💰 $0.05 | 🔴🧩 0% (0/1) / ⏱️ 63.7s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 21.7s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 18.8s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 23.5s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 29.5s / 💰 $0.17 | 🟢 100% (1/1) / ⏱️ 22.4s / 💰 $0.08 |
| 243_pod_names_contain_service 🔗 | 🟢 100% (1/1) / ⏱️ 39.8s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 50.3s / 💰 $0.10 | 🟢 100% (1/1) / ⏱️ 78.7s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 34.9s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 19.9s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 35.9s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 25.7s / 💰 $0.15 | 🟢 100% (1/1) / ⏱️ 22.3s / 💰 $0.06 |
| 24_misconfigured_pvc 🔗 | 🟢 100% (1/1) / ⏱️ 44.2s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 57.9s / 💰 $0.09 | 🟢 100% (1/1) / ⏱️ 69.2s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 41.3s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 21.5s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 45.4s / 💰 $0.09 | 🟢 100% (1/1) / ⏱️ 27.8s / 💰 $0.15 | 🟢 100% (1/1) / ⏱️ 22.7s / 💰 $0.06 |
| 254_elasticsearch_dr_test_log_check 🔗 | 🟢 100% (1/1) / ⏱️ 61.2s / 💰 $0.04 | 🔴 0% (0/1) / ⏱️ 86.2s / 💰 $0.11 | 🟢 100% (1/1) / ⏱️ 103.9s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 60.9s / 💰 $0.05 | 🔴 0% (0/1) / ⏱️ 28.7s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 75.9s / 💰 $0.13 | 🟢 100% (1/1) / ⏱️ 43.2s / 💰 $0.19 | 🟢 100% (1/1) / ⏱️ 34.1s / 💰 $0.08 |
| 259_wrong_cluster_logs_confusion 🔗 | 🟢 100% (1/1) / ⏱️ 56.5s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 73.0s / 💰 $0.13 | 🟢 100% (1/1) / ⏱️ 204.2s / 💰 $0.14 | 🟢 100% (1/1) / ⏱️ 63.1s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 31.2s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 180.1s / 💰 $0.10 | 🟢 100% (1/1) / ⏱️ 37.8s / 💰 $0.17 | 🟢 100% (1/1) / ⏱️ 29.1s / 💰 $0.08 |
| 260_global_es_remote_cluster_logs 🔗 | 🟢 100% (1/1) / ⏱️ 60.2s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 74.0s / 💰 $0.12 | 🟢 100% (1/1) / ⏱️ 107.3s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 67.1s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 35.7s / 💰 $0.05 | 🟢 100% (1/1) / ⏱️ 81.3s / 💰 $0.13 | 🟢 100% (1/1) / ⏱️ 36.8s / 💰 $0.16 | 🟢 100% (1/1) / ⏱️ 29.4s / 💰 $0.07 |
| 271_skill_suggestion_elasticsearch 🔗 | 🟢 100% (1/1) / ⏱️ 66.4s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 66.7s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 134.1s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 71.0s / 💰 $0.04 | 🔴🧩 0% (0/1) / ⏱️ 29.3s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 108.8s / 💰 $0.11 | 🟢 100% (1/1) / ⏱️ 58.3s / 💰 $0.14 | 🔴🧩 0% (0/1) / ⏱️ 20.7s / 💰 $0.05 |
| 273_skill_suggestion_prometheus 🔗 | 🔴🧩 0% (0/1) / ⏱️ 56.4s / 💰 $0.05 | 🔴 0% (0/1) / ⏱️ 44.9s / 💰 $0.10 | 🔴 0% (0/1) / ⏱️ 195.1s / 💰 $0.13 | 🟢 100% (1/1) / ⏱️ 108.0s / 💰 $0.08 | 🔴🧩 0% (0/1) / ⏱️ 25.5s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 94.6s / 💰 $0.10 | 🔴 0% (0/1) / ⏱️ 43.8s / 💰 $0.23 | 🔴🧩 0% (0/1) / ⏱️ 22.6s / 💰 $0.07 |
| 283_todowrite_multistep_audit 🔗 | 🟢 100% (1/1) / ⏱️ 28.1s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 27.7s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 110.2s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 62.1s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 21.6s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 152.2s / 💰 $0.07 | 🟢 100% (1/1) / ⏱️ 25.9s / 💰 $0.15 | 🟢 100% (1/1) / ⏱️ 19.1s / 💰 $0.06 |
| 43_current_datetime_from_prompt 🔗 | 🟢 100% (1/1) / ⏱️ 11.2s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 15.6s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 18.2s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 14.9s / 💰 $0.00 | 🟢 100% (1/1) / ⏱️ 11.6s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 16.2s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 13.1s / 💰 $0.10 | 🟢 100% (1/1) / ⏱️ 16.3s / 💰 $0.04 |
| 51_logs_summarize_errors 🔗 | 🟢 100% (1/1) / ⏱️ 23.5s / 💰 $0.01 | 🟢 100% (1/1) / ⏱️ 30.0s / 💰 $0.05 | 🔴🧩 0% (0/1) / ⏱️ 145.2s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 39.3s / 💰 $0.03 | 🔴 0% (0/1) / ⏱️ 15.2s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 151.2s / 💰 $0.06 | 🟢 100% (1/1) / ⏱️ 25.4s / 💰 $0.16 | 🟢 100% (1/1) / ⏱️ 19.2s / 💰 $0.07 |
| 61_exact_match_counting 🔗 | 🟢 100% (1/1) / ⏱️ 50.9s / 💰 $0.03 | 🟢 100% (1/1) / ⏱️ 18.9s / 💰 $0.03 | 🔴🧩 0% (0/1) / ⏱️ 86.5s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 15.5s / 💰 $0.00 | 🔴 0% (0/1) / ⏱️ 15.0s / 💰 $0.02 | 🟢 100% (1/1) / ⏱️ 17.9s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 17.1s / 💰 $0.11 | 🟢 100% (1/1) / ⏱️ 13.0s / 💰 $0.05 |
| 73a_time_window_anomaly 🔗 | 🟢 100% (1/1) / ⏱️ 47.2s / 💰 $0.05 | 🔴 0% (0/1) / ⏱️ 77.5s / 💰 $0.14 | 🟢 100% (1/1) / ⏱️ 128.7s / 💰 $0.09 | 🟢 100% (1/1) / ⏱️ 80.0s / 💰 $0.09 | 🟢 100% (1/1) / ⏱️ 25.3s / 💰 $0.04 | 🟢 100% (1/1) / ⏱️ 67.1s / 💰 $0.13 | 🟢 100% (1/1) / ⏱️ 37.1s / 💰 $0.31 | 🟢 100% (1/1) / ⏱️ 24.2s / 💰 $0.08 |
| 73b_time_window_anomaly 🔗 | 🟢 100% (1/1) / ⏱️ 116.4s / 💰 $0.14 | 🟢 100% (1/1) / ⏱️ 115.7s / 💰 $0.16 | 🟢 100% (1/1) / ⏱️ 156.7s / 💰 $0.10 | 🟢 100% (1/1) / ⏱️ 74.5s / 💰 $0.11 | 🟢 100% (1/1) / ⏱️ 36.6s / 💰 $0.11 | 🟢 100% (1/1) / ⏱️ 87.3s / 💰 $0.31 | 🟢 100% (1/1) / ⏱️ 49.1s / 💰 $0.45 | 🟢 100% (1/1) / ⏱️ 40.4s / 💰 $0.24 |
| 96_no_matching_skill 🔗 | 🔴 0% (0/1) / ⏱️ 57.7s / 💰 $0.05 | 🔴 0% (0/1) / ⏱️ 61.2s / 💰 $0.15 | 🟢 100% (1/1) / ⏱️ 237.9s / 💰 $0.18 | 🟢 100% (1/1) / ⏱️ 123.7s / 💰 $0.13 | 🟢 100% (1/1) / ⏱️ 37.6s / 💰 $0.08 | 🟢 100% (1/1) / ⏱️ 107.0s / 💰 $0.20 | 🟢 100% (1/1) / ⏱️ 41.7s / 💰 $0.27 | 🟢 100% (1/1) / ⏱️ 30.6s / 💰 $0.09 |
Results are automatically generated and updated weekly. View full traces and detailed analysis in Braintrust experiment: ci-benchmark-41844010045.