Skip to content

⚡ November 04, 2026

Generated: 2026-11-04 15:04 UTC
Total Duration: 54m 49s
Iterations: 1
Judge (classifier) model: gpt-6.1-sol-medium

Fast Benchmark

Markers: regression or benchmark
Schedule: Weekly (Sunday 2 AM UTC)
Purpose: Quick regression tests to catch breaking changes

HolmesGPT is continuously evaluated against real-world Kubernetes and cloud troubleshooting scenarios.

If you find scenarios that HolmesGPT does not perform well on, please consider adding them as evals to the benchmark.

Model Accuracy Comparison

Model Pass Fail Skip/Error Total Success Rate
deepseek-v4-pro 19 3 0 22 🟡 86% (19/22)
gemini-3.1-pro-preview 17 5 0 22 🟡 77% (17/22)
glm-5.3 16 6 0 22 🟡 73% (16/22)
gpt-6.1-sol-high 22 0 0 22 🟢 100% (22/22)
haiku-4.5 15 7 0 22 🟡 68% (15/22)
kimi-k3 22 0 0 22 🟢 100% (22/22)
opus-5.5-medium 21 1 0 22 🟡 95% (21/22)
sonnet-5.5-medium 19 3 0 22 🟡 86% (19/22)

Skill-Suggestion Check Failures

These runs failed a deterministic skill-suggestion check, whatever the judge decided about the answer. They count as failures in the success rates above.

Eval Model Check Judge Suggested skill(s)
🧩 273_skill_suggestion_prometheus deepseek-v4-pro suggested no skill, expected one wrong —
🧩 12_job_crashing glm-5.3 suggested a skill, expected none correct Pod log fetching quirks in this environment
🧩 227_count_configmaps_per_namespace[0] glm-5.3 suggested a skill, expected none correct Kubernetes query tool quirks in this environment
🧩 51_logs_summarize_errors glm-5.3 suggested a skill, expected none correct Workload naming and log conventions in this cluster
🧩 61_exact_match_counting glm-5.3 suggested a skill, expected none correct Kubernetes query tool null-selector pitfalls in this environment
🧩 271_skill_suggestion_elasticsearch haiku-4.5 suggested no skill, expected one wrong —
🧩 273_skill_suggestion_prometheus haiku-4.5 suggested no skill, expected one wrong —
🧩 271_skill_suggestion_elasticsearch sonnet-5.5-medium suggested no skill, expected one wrong —
🧩 273_skill_suggestion_prometheus sonnet-5.5-medium suggested no skill, expected one wrong —

Model Cost Comparison

Model Tests Avg Cost Min Cost Max Cost Total Cost
deepseek-v4-pro 22 $0.04 $0.00 $0.14 $0.93
gemini-3.1-pro-preview 22 $0.10 $0.03 $0.24 $2.09
glm-5.3 22 $0.09 $0.00 $0.33 $1.92
gpt-6.1-sol-high 22 $0.06 $0.00 $0.14 $1.28
haiku-4.5 22 $0.04 $0.02 $0.11 $0.94
kimi-k3 22 $0.12 $0.04 $0.31 $2.54
opus-5.5-medium 22 $0.20 $0.10 $0.45 $4.36
sonnet-5.5-medium 22 $0.08 $0.04 $0.24 $1.71

Model Latency Comparison

Model Avg (s) Min (s) Max (s) P50 (s) P95 (s)
deepseek-v4-pro 55.6 11.2 165.9 50.9 116.4
gemini-3.1-pro-preview 54.3 15.6 115.7 50.3 110.2
glm-5.3 138.5 18.2 424.6 126.7 237.9
gpt-6.1-sol-high 57.4 14.9 123.7 61.1 108.0
haiku-4.5 24.4 11.6 37.6 23.3 36.6
kimi-k3 65.5 16.2 132.8 67.1 132.8
opus-5.5-medium 33.3 13.1 58.3 36.8 49.1
sonnet-5.5-medium 23.8 13.0 40.4 22.6 34.1

⚠️ Note: 5 test(s) excluded from latency calculations due to throttling/timeout/provider errors (kimi-k3: 5)

Performance by Tag

Success rate by test category and model:

Tag deepseek-v4-pro gemini-3.1-pro-preview glm-5.3 gpt-6.1-sol-high haiku-4.5 kimi-k3 opus-5.5-medium sonnet-5.5-medium Warnings
advanced 🟡 33% (⅓) 🟡 33% (⅓) 🟡 33% (⅓) 🟢 100% (3/3) 🔴 0% (0/3) 🟢 100% (3/3) 🟡 67% (⅔) 🟡 33% (⅓)
benchmark 🟡 62% (⅝) 🟡 50% (4/8) 🟡 75% (6/8) 🟢 100% (8/8) 🟡 50% (4/8) 🟢 100% (8/8) 🟡 88% (⅞) 🟡 62% (⅝)
context_window 🟢 100% (2/2) 🟡 50% (½) 🟢 100% (2/2) 🟢 100% (2/2) 🟢 100% (2/2) 🟢 100% (2/2) 🟢 100% (2/2) 🟢 100% (2/2)
counting 🟢 100% (2/2) 🟢 100% (2/2) 🔴 0% (0/2) 🟢 100% (2/2) 🟡 50% (½) 🟢 100% (2/2) 🟢 100% (2/2) 🟢 100% (2/2)
datetime 🟢 100% (3/3) 🟡 67% (⅔) 🟢 100% (3/3) 🟢 100% (3/3) 🟢 100% (3/3) 🟢 100% (3/3) 🟢 100% (3/3) 🟢 100% (3/3)
easy 🟢 100% (7/7) 🟢 100% (7/7) 🟡 57% (4/7) 🟢 100% (7/7) 🟡 71% (5/7) 🟢 100% (7/7) 🟢 100% (7/7) 🟢 100% (7/7)
elasticsearch 🟢 100% (4/4) 🟡 75% (¾) 🟢 100% (4/4) 🟢 100% (4/4) 🟡 50% (2/4) 🟢 100% (4/4) 🟢 100% (4/4) 🟡 75% (¾)
fast 🟢 100% (1/1) 🟢 100% (1/1) 🟢 100% (1/1) 🟢 100% (1/1) 🔴 0% (0/1) 🟢 100% (1/1) 🟢 100% (1/1) 🔴 0% (0/1)
grafana 🟢 100% (1/1) 🟢 100% (1/1) 🟢 100% (1/1) 🟢 100% (1/1) 🟢 100% (1/1) 🟢 100% (1/1) 🟢 100% (1/1) 🟢 100% (1/1)
hard 🟡 50% (½) 🟡 50% (½) 🟡 50% (½) 🟢 100% (2/2) 🟡 50% (½) 🟢 100% (2/2) 🟢 100% (2/2) 🟡 50% (½)
kubernetes 🟡 90% (9/10) 🟡 90% (9/10) 🟡 80% (8/10) 🟢 100% (10/10) 🟡 80% (8/10) 🟢 100% (10/10) 🟡 90% (9/10) 🟡 90% (9/10)
logs 🟡 86% (6/7) 🟡 71% (5/7) 🟡 71% (5/7) 🟢 100% (7/7) 🟡 43% (3/7) 🟢 100% (7/7) 🟢 100% (7/7) 🟡 71% (5/7)
loki 🟢 100% (2/2) 🟢 100% (2/2) 🟢 100% (2/2) 🟢 100% (2/2) 🟡 50% (½) 🟢 100% (2/2) 🟢 100% (2/2) 🟢 100% (2/2)
medium 🟡 82% (9/11) 🟡 64% (7/11) 🟡 91% (10/11) 🟢 100% (11/11) 🟡 64% (7/11) 🟢 100% (11/11) 🟡 91% (10/11) 🟡 82% (9/11)
metrics 🟡 50% (½) 🟡 50% (½) 🟡 50% (½) 🟢 100% (2/2) 🟡 50% (½) 🟢 100% (2/2) 🟡 50% (½) 🟡 50% (½)
multi-cluster 🟢 100% (3/3) 🟡 67% (⅔) 🟢 100% (3/3) 🟢 100% (3/3) 🟡 67% (⅔) 🟢 100% (3/3) 🟢 100% (3/3) 🟢 100% (3/3)
one-test 🟢 100% (1/1) 🟢 100% (1/1) 🟢 100% (1/1) 🟢 100% (1/1) 🟢 100% (1/1) 🟢 100% (1/1) 🟢 100% (1/1) 🟢 100% (1/1)
port-forward 🟡 75% (¾) 🟡 75% (¾) 🟡 75% (¾) 🟢 100% (4/4) 🟡 50% (2/4) 🟢 100% (4/4) 🟡 75% (¾) 🟡 75% (¾)
prometheus 🔴 0% (0/1) 🔴 0% (0/1) 🔴 0% (0/1) 🟢 100% (1/1) 🔴 0% (0/1) 🟢 100% (1/1) 🔴 0% (0/1) 🔴 0% (0/1)
question-answer 🟢 100% (2/2) 🟢 100% (2/2) 🟢 100% (2/2) 🟢 100% (2/2) 🟢 100% (2/2) 🟢 100% (2/2) 🟢 100% (2/2) 🟢 100% (2/2)
regression 🟢 100% (14/14) 🟡 93% (13/14) 🟡 71% (10/14) 🟢 100% (14/14) 🟡 79% (11/14) 🟢 100% (14/14) 🟢 100% (14/14) 🟢 100% (14/14)
skills 🟡 85% (11/13) 🟡 85% (11/13) 🟡 62% (8/13) 🟢 100% (13/13) 🟡 69% (9/13) 🟢 100% (13/13) 🟡 92% (12/13) 🟡 85% (11/13)
transparency 🟢 100% (3/3) 🟡 67% (⅔) 🟢 100% (3/3) 🟢 100% (3/3) 🟡 67% (⅔) 🟢 100% (3/3) 🟢 100% (3/3) 🟢 100% (3/3)
Overall 🟡 86% (19/22) 🟡 77% (17/22) 🟡 73% (16/22) 🟢 100% (22/22) 🟡 68% (15/22) 🟢 100% (22/22) 🟡 95% (21/22) 🟡 86% (19/22)

Raw Results

Status of all evaluations across models. Color coding:

  • 🟢 Passing 100% (stable)
  • 🟡 Passing 1-99%
  • 🔴 Passing 0% (failing)
  • 🔧 Mock data failure (missing or invalid test data)
  • ⚠️ Setup or model-provider failure (environment/infrastructure issue; the model never answered, so the run is left out of success rates)
  • ⏱️ Timeout or rate limit error (left out of success rates)
  • ⏭️ Test skipped (e.g., known issue or precondition not met)
  • 🧩 Failed the skill-suggestion check (counted as a failure; see Skill-Suggestion Check Failures for the skill and the judge's verdict)
Eval ID deepseek-v4-pro gemini-3.1-pro-preview glm-5.3 gpt-6.1-sol-high haiku-4.5 kimi-k3 opus-5.5-medium sonnet-5.5-medium
09_crashpod 🔗 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢
100a_loki_historical_logs 🔗 🟢 🟢 🟢 🟢 🔴 🟢 🟢 🟢
101_loki_historical_logs_pod_deleted 🔗 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢
108_logs_nearby_lines 🔗 🔴 🔴 🔴 🟢 🔴 🟢 🟢 🔴
112_find_pvcs_by_uuid 🔗 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢
12_job_crashing 🔗 🟢 🟢 🔴🧩 🟢 🟢 🟢 🟢 🟢
179_grafana_big_dashboard_query 🔗 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢
227_count_configmaps_per_namespace[0] 🔗 🟢 🟢 🔴🧩 🟢 🟢 🟢 🟢 🟢
243_pod_names_contain_service 🔗 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢
24_misconfigured_pvc 🔗 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢
254_elasticsearch_dr_test_log_check 🔗 🟢 🔴 🟢 🟢 🔴 🟢 🟢 🟢
259_wrong_cluster_logs_confusion 🔗 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢
260_global_es_remote_cluster_logs 🔗 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢
271_skill_suggestion_elasticsearch 🔗 🟢 🟢 🟢 🟢 🔴🧩 🟢 🟢 🔴🧩
273_skill_suggestion_prometheus 🔗 🔴🧩 🔴 🔴 🟢 🔴🧩 🟢 🔴 🔴🧩
283_todowrite_multistep_audit 🔗 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢
43_current_datetime_from_prompt 🔗 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢
51_logs_summarize_errors 🔗 🟢 🟢 🔴🧩 🟢 🔴 🟢 🟢 🟢
61_exact_match_counting 🔗 🟢 🟢 🔴🧩 🟢 🔴 🟢 🟢 🟢
73a_time_window_anomaly 🔗 🟢 🔴 🟢 🟢 🟢 🟢 🟢 🟢
73b_time_window_anomaly 🔗 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢
96_no_matching_skill 🔗 🔴 🔴 🟢 🟢 🟢 🟢 🟢 🟢
SUMMARY 🟡 86% (19/22) 🟡 77% (17/22) 🟡 73% (16/22) 🟢 100% (22/22) 🟡 68% (15/22) 🟢 100% (22/22) 🟡 95% (21/22) 🟡 86% (19/22)

Detailed Raw Results

Eval ID deepseek-v4-pro gemini-3.1-pro-preview glm-5.3 gpt-6.1-sol-high haiku-4.5 kimi-k3 opus-5.5-medium sonnet-5.5-medium
09_crashpod 🔗 🟢 100% (1/1) / ⏱️ 39.3s / 💰 $0.02 🟢 100% (1/1) / ⏱️ 44.4s / 💰 $0.08 🟢 100% (1/1) / ⏱️ 73.4s / 💰 $0.04 🟢 100% (1/1) / ⏱️ 40.2s / 💰 $0.03 🟢 100% (1/1) / ⏱️ 19.3s / 💰 $0.03 🟢 100% (1/1) / ⏱️ 1529.6s / 💰 $0.08 🟢 100% (1/1) / ⏱️ 27.2s / 💰 $0.17 🟢 100% (1/1) / ⏱️ 18.6s / 💰 $0.06
100a_loki_historical_logs 🔗 🟢 100% (1/1) / ⏱️ 95.3s / 💰 $0.06 🟢 100% (1/1) / ⏱️ 44.0s / 💰 $0.07 🟢 100% (1/1) / ⏱️ 226.4s / 💰 $0.17 🟢 100% (1/1) / ⏱️ 106.5s / 💰 $0.14 🔴 0% (0/1) / ⏱️ 27.5s / 💰 $0.05 🟢 100% (1/1) / ⏱️ 132.8s / 💰 $0.26 🟢 100% (1/1) / ⏱️ 48.7s / 💰 $0.30 🟢 100% (1/1) / ⏱️ 28.8s / 💰 $0.09
101_loki_historical_logs_pod_deleted 🔗 🟢 100% (1/1) / ⏱️ 165.9s / 💰 $0.14 🟢 100% (1/1) / ⏱️ 110.2s / 💰 $0.24 🟢 100% (1/1) / ⏱️ 157.1s / 💰 $0.10 🟢 100% (1/1) / ⏱️ 61.1s / 💰 $0.04 🟢 100% (1/1) / ⏱️ 22.3s / 💰 $0.04 🟢 100% (1/1) / ⏱️ 65.0s / 💰 $0.13 🟢 100% (1/1) / ⏱️ 41.4s / 💰 $0.25 🟢 100% (1/1) / ⏱️ 26.8s / 💰 $0.06
108_logs_nearby_lines 🔗 🔴 0% (0/1) / ⏱️ 89.8s / 💰 $0.07 🔴 0% (0/1) / ⏱️ 59.5s / 💰 $0.12 🔴 0% (0/1) / ⏱️ 126.7s / 💰 $0.08 🟢 100% (1/1) / ⏱️ 72.5s / 💰 $0.11 🔴 0% (0/1) / ⏱️ 35.8s / 💰 $0.08 🟢 100% (1/1) / ⏱️ 83.8s / 💰 $0.17 🟢 100% (1/1) / ⏱️ 38.7s / 💰 $0.20 🔴 0% (0/1) / ⏱️ 28.1s / 💰 $0.09
112_find_pvcs_by_uuid 🔗 🟢 100% (1/1) / ⏱️ 21.5s / 💰 $0.01 🟢 100% (1/1) / ⏱️ 39.6s / 💰 $0.07 🟢 100% (1/1) / ⏱️ 124.7s / 💰 $0.06 🟢 100% (1/1) / ⏱️ 36.4s / 💰 $0.02 🟢 100% (1/1) / ⏱️ 15.7s / 💰 $0.03 🟢 100% (1/1) / ⏱️ 41.7s / 💰 $0.08 🟢 100% (1/1) / ⏱️ 21.3s / 💰 $0.14 🟢 100% (1/1) / ⏱️ 16.5s / 💰 $0.06
12_job_crashing 🔗 🟢 100% (1/1) / ⏱️ 40.1s / 💰 $0.02 🟢 100% (1/1) / ⏱️ 28.4s / 💰 $0.05 🔴🧩 0% (0/1) / ⏱️ 74.9s / 💰 $0.05 🟢 100% (1/1) / ⏱️ 29.5s / 💰 $0.04 🟢 100% (1/1) / ⏱️ 23.3s / 💰 $0.04 🟢 100% (1/1) / ⏱️ 171.8s / 💰 $0.10 🟢 100% (1/1) / ⏱️ 21.3s / 💰 $0.13 🟢 100% (1/1) / ⏱️ 18.7s / 💰 $0.06
179_grafana_big_dashboard_query 🔗 🟢 100% (1/1) / ⏱️ 28.8s / 💰 $0.05 🟢 100% (1/1) / ⏱️ 33.1s / 💰 $0.07 🟢 100% (1/1) / ⏱️ 424.6s / 💰 $0.33 🟢 100% (1/1) / ⏱️ 38.5s / 💰 $0.10 🟢 100% (1/1) / ⏱️ 19.2s / 💰 $0.04 🟢 100% (1/1) / ⏱️ 29.6s / 💰 $0.07 🟢 100% (1/1) / ⏱️ 21.3s / 💰 $0.26 🟢 100% (1/1) / ⏱️ 19.6s / 💰 $0.11
227_count_configmaps_per_namespace[0] 🔗 🟢 100% (1/1) / ⏱️ 23.5s / 💰 $0.02 🟢 100% (1/1) / ⏱️ 36.9s / 💰 $0.05 🔴🧩 0% (0/1) / ⏱️ 63.7s / 💰 $0.03 🟢 100% (1/1) / ⏱️ 21.7s / 💰 $0.01 🟢 100% (1/1) / ⏱️ 18.8s / 💰 $0.03 🟢 100% (1/1) / ⏱️ 23.5s / 💰 $0.06 🟢 100% (1/1) / ⏱️ 29.5s / 💰 $0.17 🟢 100% (1/1) / ⏱️ 22.4s / 💰 $0.08
243_pod_names_contain_service 🔗 🟢 100% (1/1) / ⏱️ 39.8s / 💰 $0.02 🟢 100% (1/1) / ⏱️ 50.3s / 💰 $0.10 🟢 100% (1/1) / ⏱️ 78.7s / 💰 $0.04 🟢 100% (1/1) / ⏱️ 34.9s / 💰 $0.03 🟢 100% (1/1) / ⏱️ 19.9s / 💰 $0.03 🟢 100% (1/1) / ⏱️ 35.9s / 💰 $0.07 🟢 100% (1/1) / ⏱️ 25.7s / 💰 $0.15 🟢 100% (1/1) / ⏱️ 22.3s / 💰 $0.06
24_misconfigured_pvc 🔗 🟢 100% (1/1) / ⏱️ 44.2s / 💰 $0.02 🟢 100% (1/1) / ⏱️ 57.9s / 💰 $0.09 🟢 100% (1/1) / ⏱️ 69.2s / 💰 $0.04 🟢 100% (1/1) / ⏱️ 41.3s / 💰 $0.03 🟢 100% (1/1) / ⏱️ 21.5s / 💰 $0.04 🟢 100% (1/1) / ⏱️ 45.4s / 💰 $0.09 🟢 100% (1/1) / ⏱️ 27.8s / 💰 $0.15 🟢 100% (1/1) / ⏱️ 22.7s / 💰 $0.06
254_elasticsearch_dr_test_log_check 🔗 🟢 100% (1/1) / ⏱️ 61.2s / 💰 $0.04 🔴 0% (0/1) / ⏱️ 86.2s / 💰 $0.11 🟢 100% (1/1) / ⏱️ 103.9s / 💰 $0.06 🟢 100% (1/1) / ⏱️ 60.9s / 💰 $0.05 🔴 0% (0/1) / ⏱️ 28.7s / 💰 $0.04 🟢 100% (1/1) / ⏱️ 75.9s / 💰 $0.13 🟢 100% (1/1) / ⏱️ 43.2s / 💰 $0.19 🟢 100% (1/1) / ⏱️ 34.1s / 💰 $0.08
259_wrong_cluster_logs_confusion 🔗 🟢 100% (1/1) / ⏱️ 56.5s / 💰 $0.03 🟢 100% (1/1) / ⏱️ 73.0s / 💰 $0.13 🟢 100% (1/1) / ⏱️ 204.2s / 💰 $0.14 🟢 100% (1/1) / ⏱️ 63.1s / 💰 $0.05 🟢 100% (1/1) / ⏱️ 31.2s / 💰 $0.04 🟢 100% (1/1) / ⏱️ 180.1s / 💰 $0.10 🟢 100% (1/1) / ⏱️ 37.8s / 💰 $0.17 🟢 100% (1/1) / ⏱️ 29.1s / 💰 $0.08
260_global_es_remote_cluster_logs 🔗 🟢 100% (1/1) / ⏱️ 60.2s / 💰 $0.05 🟢 100% (1/1) / ⏱️ 74.0s / 💰 $0.12 🟢 100% (1/1) / ⏱️ 107.3s / 💰 $0.07 🟢 100% (1/1) / ⏱️ 67.1s / 💰 $0.06 🟢 100% (1/1) / ⏱️ 35.7s / 💰 $0.05 🟢 100% (1/1) / ⏱️ 81.3s / 💰 $0.13 🟢 100% (1/1) / ⏱️ 36.8s / 💰 $0.16 🟢 100% (1/1) / ⏱️ 29.4s / 💰 $0.07
271_skill_suggestion_elasticsearch 🔗 🟢 100% (1/1) / ⏱️ 66.4s / 💰 $0.03 🟢 100% (1/1) / ⏱️ 66.7s / 💰 $0.07 🟢 100% (1/1) / ⏱️ 134.1s / 💰 $0.04 🟢 100% (1/1) / ⏱️ 71.0s / 💰 $0.04 🔴🧩 0% (0/1) / ⏱️ 29.3s / 💰 $0.04 🟢 100% (1/1) / ⏱️ 108.8s / 💰 $0.11 🟢 100% (1/1) / ⏱️ 58.3s / 💰 $0.14 🔴🧩 0% (0/1) / ⏱️ 20.7s / 💰 $0.05
273_skill_suggestion_prometheus 🔗 🔴🧩 0% (0/1) / ⏱️ 56.4s / 💰 $0.05 🔴 0% (0/1) / ⏱️ 44.9s / 💰 $0.10 🔴 0% (0/1) / ⏱️ 195.1s / 💰 $0.13 🟢 100% (1/1) / ⏱️ 108.0s / 💰 $0.08 🔴🧩 0% (0/1) / ⏱️ 25.5s / 💰 $0.04 🟢 100% (1/1) / ⏱️ 94.6s / 💰 $0.10 🔴 0% (0/1) / ⏱️ 43.8s / 💰 $0.23 🔴🧩 0% (0/1) / ⏱️ 22.6s / 💰 $0.07
283_todowrite_multistep_audit 🔗 🟢 100% (1/1) / ⏱️ 28.1s / 💰 $0.03 🟢 100% (1/1) / ⏱️ 27.7s / 💰 $0.06 🟢 100% (1/1) / ⏱️ 110.2s / 💰 $0.08 🟢 100% (1/1) / ⏱️ 62.1s / 💰 $0.07 🟢 100% (1/1) / ⏱️ 21.6s / 💰 $0.04 🟢 100% (1/1) / ⏱️ 152.2s / 💰 $0.07 🟢 100% (1/1) / ⏱️ 25.9s / 💰 $0.15 🟢 100% (1/1) / ⏱️ 19.1s / 💰 $0.06
43_current_datetime_from_prompt 🔗 🟢 100% (1/1) / ⏱️ 11.2s / 💰 $0.00 🟢 100% (1/1) / ⏱️ 15.6s / 💰 $0.03 🟢 100% (1/1) / ⏱️ 18.2s / 💰 $0.00 🟢 100% (1/1) / ⏱️ 14.9s / 💰 $0.00 🟢 100% (1/1) / ⏱️ 11.6s / 💰 $0.02 🟢 100% (1/1) / ⏱️ 16.2s / 💰 $0.04 🟢 100% (1/1) / ⏱️ 13.1s / 💰 $0.10 🟢 100% (1/1) / ⏱️ 16.3s / 💰 $0.04
51_logs_summarize_errors 🔗 🟢 100% (1/1) / ⏱️ 23.5s / 💰 $0.01 🟢 100% (1/1) / ⏱️ 30.0s / 💰 $0.05 🔴🧩 0% (0/1) / ⏱️ 145.2s / 💰 $0.06 🟢 100% (1/1) / ⏱️ 39.3s / 💰 $0.03 🔴 0% (0/1) / ⏱️ 15.2s / 💰 $0.02 🟢 100% (1/1) / ⏱️ 151.2s / 💰 $0.06 🟢 100% (1/1) / ⏱️ 25.4s / 💰 $0.16 🟢 100% (1/1) / ⏱️ 19.2s / 💰 $0.07
61_exact_match_counting 🔗 🟢 100% (1/1) / ⏱️ 50.9s / 💰 $0.03 🟢 100% (1/1) / ⏱️ 18.9s / 💰 $0.03 🔴🧩 0% (0/1) / ⏱️ 86.5s / 💰 $0.04 🟢 100% (1/1) / ⏱️ 15.5s / 💰 $0.00 🔴 0% (0/1) / ⏱️ 15.0s / 💰 $0.02 🟢 100% (1/1) / ⏱️ 17.9s / 💰 $0.04 🟢 100% (1/1) / ⏱️ 17.1s / 💰 $0.11 🟢 100% (1/1) / ⏱️ 13.0s / 💰 $0.05
73a_time_window_anomaly 🔗 🟢 100% (1/1) / ⏱️ 47.2s / 💰 $0.05 🔴 0% (0/1) / ⏱️ 77.5s / 💰 $0.14 🟢 100% (1/1) / ⏱️ 128.7s / 💰 $0.09 🟢 100% (1/1) / ⏱️ 80.0s / 💰 $0.09 🟢 100% (1/1) / ⏱️ 25.3s / 💰 $0.04 🟢 100% (1/1) / ⏱️ 67.1s / 💰 $0.13 🟢 100% (1/1) / ⏱️ 37.1s / 💰 $0.31 🟢 100% (1/1) / ⏱️ 24.2s / 💰 $0.08
73b_time_window_anomaly 🔗 🟢 100% (1/1) / ⏱️ 116.4s / 💰 $0.14 🟢 100% (1/1) / ⏱️ 115.7s / 💰 $0.16 🟢 100% (1/1) / ⏱️ 156.7s / 💰 $0.10 🟢 100% (1/1) / ⏱️ 74.5s / 💰 $0.11 🟢 100% (1/1) / ⏱️ 36.6s / 💰 $0.11 🟢 100% (1/1) / ⏱️ 87.3s / 💰 $0.31 🟢 100% (1/1) / ⏱️ 49.1s / 💰 $0.45 🟢 100% (1/1) / ⏱️ 40.4s / 💰 $0.24
96_no_matching_skill 🔗 🔴 0% (0/1) / ⏱️ 57.7s / 💰 $0.05 🔴 0% (0/1) / ⏱️ 61.2s / 💰 $0.15 🟢 100% (1/1) / ⏱️ 237.9s / 💰 $0.18 🟢 100% (1/1) / ⏱️ 123.7s / 💰 $0.13 🟢 100% (1/1) / ⏱️ 37.6s / 💰 $0.08 🟢 100% (1/1) / ⏱️ 107.0s / 💰 $0.20 🟢 100% (1/1) / ⏱️ 41.7s / 💰 $0.27 🟢 100% (1/1) / ⏱️ 30.6s / 💰 $0.09

Results are automatically generated and updated weekly. View full traces and detailed analysis in Braintrust experiment: ci-benchmark-41844010045.