Comparing four models on my actual codebase
I have a real codebase I work on every week. It’s the homelab repo, about 4000 lines of Python plus some bash and a few Terraform files. I gave four models the same five tasks on it. The tasks are real things I needed done.
The models:
- Claude Sonnet 4.5
- GPT-5
- OpenAI Codex (CLI version)
- MiniMax M3
Same prompts. Same context (the whole repo, fed in via the API). Same evaluation criteria. I’m not a benchmark, I’m a guy with a codebase, and I graded the output on whether I’d ship it as-is or send it back for revision.
The five tasks
- Add a new CLI subcommand to the backup script that lists the last 7 days of backup runs
- Find why the morning agent stopped working on Tuesday (no hint, just “this stopped working”)
- Refactor the Proxmox VM provisioning script to use the official
proxmoxerlibrary instead of therequestscalls we have now - Add a test for the chunking function in the RAG
- Write a CLAUDE.md for the homelab repo
The results
| Task | Claude | GPT-5 | Codex | M3 |
|---|---|---|---|---|
| 1. New subcommand | Shipped | Shipped | Shipped | Shipped |
| 2. Bug hunt | Shipped (1st try) | Shipped (2nd try) | Almost (wrong file) | Shipped (2nd try) |
| 3. Refactor | Shipped | Shipped | Shipped (bigger diff than needed) | Shipped |
| 4. Test | Shipped | Shipped | Shipped | Shipped |
| 5. CLAUDE.md | Shipped (good) | Shipped (verbose) | Shipped (good) | Shipped (terse) |
The table is the short version. The interesting differences are in the diffs.
What I learned
All four shipped all five tasks. That was the surprise. I’d read enough discourse to think one of them would be clearly better. They weren’t. They each had a different failure mode and a different strength.
Claude is the most consistent. It shipped all five on the first try. The bug-hunt fix was the right one. The refactor was the smallest diff. The CLAUDE.md was the right length. Nothing stood out as better than the others, but nothing was worse either.
GPT-5 is verbose. The refactor was 2x the diff of Claude’s because GPT-5 added type hints to every function. Not wrong, just more than I asked for. The CLAUDE.md was 80 lines of stuff I didn’t need.
Codex is fast and minimal. The terminal version is great for the “make a small change” loop. The diff for the refactor was tight. The bug-hunt fix pointed at the wrong file because Codex didn’t read enough of the codebase to know where the actual error was.
M3 is the cheapest and ships terse code. The M3 output is consistently shorter than the others. The CLAUDE.md was 30 lines, which is what I would have written. The M3 was also the slowest of the four on the bug-hunt task, which surprised me.
What I’m doing with this
I’m not switching. The four are close enough that the difference is the workflow, not the model. I use M3 for the long-running, multi-turn work because the cost adds up and the terseness is a feature. I use Claude for bug-hunts because the consistency matters more than the speed. I use Codex for the quick single-file edits. I use GPT-5 when I want a different opinion on a hard refactor.
If you only pick one, pick the one that fits the loop you’re in. They’re all good enough for a 4000-line Python codebase.