Hunter B
c8fcef7f1e
Runs PinchBench tasks directly through codewhale exec --auto instead of going through OpenClaw. Loads task markdown, creates workspace, runs the prompt, and grades using PinchBench's embedded automated checks. No external agent framework dependency — just codewhale + pyyaml.
Benchmark Scripts
Convenience runners for evaluating CodeWhale against external benchmarks.
Quick Start
# Set your API key
export DEEPSEEK_API_KEY="sk-..."
# SWE-bench (single instance)
./scripts/benchmarks/run-swebench.sh \
--instance-id django__django-12345 \
--issue-file ./issue.md
# Terminal-Bench (via Harbor)
./scripts/benchmarks/run-terminal-bench.sh \
--model deepseek/deepseek-chat
# PinchBench (auto-install + run)
./scripts/benchmarks/run-pinchbench.sh \
--install \
--model deepseek/deepseek-chat
Files
run-swebench.sh— SWE-bench batch driver and evaluatorrun-terminal-bench.sh— Terminal-Bench runner via Harborrun-pinchbench.sh— PinchBench runner with auto-installharbor/__init__.py— Harbor adapter for CodeWhale (Python)harbor/codewhale_agent.py— Adapter entry point
Documentation
See docs/BENCHMARKS.md for full setup instructions, reproducibility checklists, and references.