Search results

Keeping track of SWTBench at scale

Status: Open.

#323 In OpenHands/benchmarks;

· simonrosenberg opened on Jan 15, 2026

Eval runtimes are force-stopped after ~20–24 minutes of runtime-api inactivity, causing mid-run 404s

Status: Open.

#320 In OpenHands/benchmarks;

· simonrosenberg opened on Jan 15, 2026

Restructure the reports and artifacts structure

Status: Open.

#319 In OpenHands/benchmarks;

· simonrosenberg opened on Jan 15, 2026

Prebake SWT-bench eval image to avoid runtime mamba hangs

Status: Open.

#317 In OpenHands/benchmarks;

· simonrosenberg opened on Jan 14, 2026

swtbench.run_infer getting stuck?

Status: Open.

#315 In OpenHands/benchmarks;

· enochii opened on Jan 14, 2026

SWTBench profiling

Status: Open.

#314 In OpenHands/benchmarks;

· simonrosenberg opened on Jan 14, 2026

Tool fall failures occurred during SWE-Bench testing based on DeepSeek-V3.2

Status: Open.

#312 In OpenHands/benchmarks;

· SefaZeng opened on Jan 14, 2026

Configure llm key for running benchmarks

openhands

Status: Open.

#301 In OpenHands/benchmarks;

· simonrosenberg opened on Jan 12, 2026

Make Datadog conversation event logging opt-in

Status: Open.

#300 In OpenHands/benchmarks;

· simonrosenberg opened on Jan 12, 2026

clarify retry semantics (max_attempts vs max_retries) in evaluation

Status: Open.

#295 In OpenHands/benchmarks;

· simonrosenberg opened on Jan 12, 2026

Investigate GAIA null entries in output.jsonl

Status: Open.

#291 In OpenHands/benchmarks;

· simonrosenberg opened on Jan 9, 2026

Apptainer for Swebench

Status: Open.

#283 In OpenHands/benchmarks;

· lilys012 opened on Jan 8, 2026