Hunter Bown
25ce4f5970
- SWE-bench: codewhale swebench run/export writes prediction JSONL from working-tree diff, with untracked-file inclusion via git add -N - CLI: --workspace / -C global flag forwards to TUI for file ops - CLI: codewhale exec --auto semantics clarified in help text - Markdown: table pipes inside inline code no longer create phantom columns (split_table_cells with backtick-awareness) - Receipt: floor_char_boundary prevents multibyte UTF-8 slice panic - Contributors: Ling (LING71671 #1839 #1911), Ben Younes (ousamabenyounes #1938), jeoor npm fix (#1860) credited across all 3 READMEs - ja-JP README: 19 contributors synced to parity with EN/zh-CN (80 each) - Docs: SWEBENCH.md, RECURSIVE_SELF_IMPROVEMENT.md, MODES.md exec clarification - Sub-agent footer: Alt+V hint now says 'details' not 'raw'
2.4 KiB
SWE-bench
CodeWhale's SWE-bench adapter writes the prediction file that the official
SWE-bench evaluation harness expects. It does not replace the harness; it
generates model_patch rows from a local task workspace.
One Instance
Start from a workspace checked out at the SWE-bench instance base commit, with the issue text saved locally:
codewhale swebench run \
--instance-id django__django-12345 \
--issue-file issue.md \
--predictions-path all_preds.jsonl
run invokes tool-backed non-interactive mode, equivalent to
codewhale exec --auto, with stream-json output by default. When the turn
finishes, CodeWhale exports git diff --binary --no-ext-diff as one JSONL
prediction row:
{"instance_id":"django__django-12345","model_name_or_path":"codewhale/deepseek-v4-pro","model_patch":"diff --git ..."}
If you already ran CodeWhale, or edited the workspace manually, export the current diff without another model turn:
codewhale swebench export \
--instance-id django__django-12345 \
--predictions-path all_preds.jsonl
Both commands update the row for the same instance_id instead of appending a
duplicate row. Untracked files are marked with git add -N before diff export
so newly-created files appear in the patch.
Evaluate
Install SWE-bench and Docker using the official SWE-bench setup instructions, then pass the prediction file to the official harness:
python -m swebench.harness.run_evaluation \
--dataset_name princeton-nlp/SWE-bench_Lite \
--predictions_path all_preds.jsonl \
--max_workers 1 \
--run_id codewhale-smoke
On Apple Silicon, the official SWE-bench docs recommend adding
--namespace '' so images build locally instead of pulling Linux images.
Batch Driver Shape
A simple batch runner should prepare each instance workspace, write the issue
body to issue.md, run codewhale swebench run, then call the harness once
on the accumulated all_preds.jsonl.
For reproducible runs, pin:
- CodeWhale version and commit:
codewhale --version - Model label:
--model-name-or-path codewhale/deepseek-v4-pro - Dataset and split used by the harness
- Docker platform and worker count
- The
all_preds.jsonlfile and CodeWhale stream logs
Official references:
- SWE-bench repository: https://github.com/SWE-bench/SWE-bench
- SWE-bench harness docs: https://www.swebench.com/SWE-bench/api/harness/