Yaowei Zheng
d4faee3a1e
Branch-length batch covering tooling, the model layer, the Web App and the public surfaces. Highlights: - Changelog: a per-release `changelog/<version>/` tree, grouped by the surface each change touches, with a root CHANGELOG.md holding one line per release. - Dev startup: `scripts/dev-prebuild.mjs` serializes the skills+core prebuild behind a lock and keeps `pnpm install` current; `pnpm dev` runs server+web together. - AgentHub 0.3.3 -> 0.4.0: OmniMessage complete payloads carry one opaque `fidelity` object in place of item-level `signature`/`phase`, threaded verbatim through Trace, replay and resume; malformed classification adapted to the new error types. - Model layer: a model is always referenced by an explicit `(provider, model_id)` pair. The provider is never inferred, guessed or defaulted -- both the catalog inference and the unique-match config resolution are gone, and CLI, SDK, server routes and run_subagent all require the complete pair. Catalog gains the Qwen Token Plan, Qwen Pay-As-You-Go and Fireworks AI gateways, plus an expanded OpenRouter group. - Web App: catalog preset sync and per-group speed test on the Models page, positional slash commands, a markdown renderer, skill-library update reminders, and a vertically centred draft page whose upward menus size themselves to the room available. - Public surfaces: restructured READMEs, the penguin.ooo landing site and blog, refreshed benchmark results for both suites, and the demo videos playing on the landing page. Includes the fixes from a full review of the branch: 23 confirmed findings, among them a provider-inference bug that could send one vendor's API key to another vendor's endpoint, and an Escape handler that destroyed the composer's contents unrecoverably. Verified on the branch head: pnpm test (1127 passing, 7 packages), pnpm typecheck and pnpm format:check clean, Playwright e2e 14/14.
49 lines
6.9 KiB
XML
49 lines
6.9 KiB
XML
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 920 368" width="920" height="368" role="img" aria-label="Benchmark: PenguinHarness vs Claude Code vs OpenAI Codex on two suites — comparable accuracy at a small fraction of the cost">
|
||
<text x="148" y="36" font-size="12" font-weight="400" fill="#898781" text-anchor="start" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">Accuracy · suite total · higher is better</text>
|
||
<text x="596" y="36" font-size="12" font-weight="400" fill="#898781" text-anchor="start" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">Total cost (USD) · lower is better</text>
|
||
<text x="24" y="68" font-size="13" font-weight="600" fill="#1f2328" text-anchor="start" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">Data analysis — 15 tasks, single run</text>
|
||
<rect x="147" y="78" width="1" height="92" fill="#c3c2b7"/>
|
||
<text x="136" y="94.5" font-size="12" font-weight="600" fill="#1f2328" text-anchor="end" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">PenguinHarness</text>
|
||
<path d="M149 82 h172.0 a4 4 0 0 1 4 4 v8 a4 4 0 0 1 -4 4 h-172.0 z" fill="#2a78d6"/>
|
||
<text x="337" y="94.5" font-size="12" font-weight="600" fill="#1f2328" text-anchor="start" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">66.67%</text>
|
||
<text x="136" y="124.5" font-size="12" font-weight="400" fill="#52514e" text-anchor="end" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">Claude Code</text>
|
||
<path d="M149 112 h136.8 a4 4 0 0 1 4 4 v8 a4 4 0 0 1 -4 4 h-136.8 z" fill="#898781"/>
|
||
<text x="301.7841607919604" y="124.5" font-size="12" font-weight="400" fill="#1f2328" text-anchor="start" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">53.33%</text>
|
||
<text x="136" y="154.5" font-size="12" font-weight="400" fill="#52514e" text-anchor="end" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">OpenAI Codex</text>
|
||
<path d="M149 142 h136.8 a4 4 0 0 1 4 4 v8 a4 4 0 0 1 -4 4 h-136.8 z" fill="#898781"/>
|
||
<text x="301.7841607919604" y="154.5" font-size="12" font-weight="400" fill="#1f2328" text-anchor="start" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">53.33%</text>
|
||
<rect x="595" y="78" width="1" height="92" fill="#c3c2b7"/>
|
||
<text x="584" y="94.5" font-size="12" font-weight="600" fill="#1f2328" text-anchor="end" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">PenguinHarness</text>
|
||
<path d="M597 82 h10.0 a4 4 0 0 1 4 4 v8 a4 4 0 0 1 -4 4 h-10.0 z" fill="#2a78d6"/>
|
||
<text x="623" y="94.5" font-size="12" font-weight="600" fill="#1f2328" text-anchor="start" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">$0.55</text>
|
||
<text x="584" y="124.5" font-size="12" font-weight="400" fill="#52514e" text-anchor="end" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">Claude Code</text>
|
||
<path d="M597 112 h172.0 a4 4 0 0 1 4 4 v8 a4 4 0 0 1 -4 4 h-172.0 z" fill="#898781"/>
|
||
<text x="785" y="124.5" font-size="12" font-weight="400" fill="#1f2328" text-anchor="start" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">$38.48</text>
|
||
<text x="584" y="154.5" font-size="12" font-weight="400" fill="#52514e" text-anchor="end" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">OpenAI Codex</text>
|
||
<path d="M597 142 h84.8 a4 4 0 0 1 4 4 v8 a4 4 0 0 1 -4 4 h-84.8 z" fill="#898781"/>
|
||
<text x="697.7945961503353" y="154.5" font-size="12" font-weight="400" fill="#1f2328" text-anchor="start" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">$19.41</text>
|
||
<text x="24" y="210" font-size="13" font-weight="600" fill="#1f2328" text-anchor="start" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">Coding — 40 tasks × 2 runs</text>
|
||
<rect x="147" y="220" width="1" height="92" fill="#c3c2b7"/>
|
||
<text x="136" y="236.5" font-size="12" font-weight="600" fill="#1f2328" text-anchor="end" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">PenguinHarness</text>
|
||
<path d="M149 224 h141.4 a4 4 0 0 1 4 4 v8 a4 4 0 0 1 -4 4 h-141.4 z" fill="#2a78d6"/>
|
||
<text x="306.3913043478261" y="236.5" font-size="12" font-weight="600" fill="#1f2328" text-anchor="start" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">71.25%</text>
|
||
<text x="136" y="266.5" font-size="12" font-weight="400" fill="#52514e" text-anchor="end" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">Claude Code</text>
|
||
<path d="M149 254 h172.0 a4 4 0 0 1 4 4 v8 a4 4 0 0 1 -4 4 h-172.0 z" fill="#898781"/>
|
||
<text x="337" y="266.5" font-size="12" font-weight="400" fill="#1f2328" text-anchor="start" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">86.25%</text>
|
||
<text x="136" y="296.5" font-size="12" font-weight="400" fill="#52514e" text-anchor="end" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">OpenAI Codex</text>
|
||
<path d="M149 284 h141.4 a4 4 0 0 1 4 4 v8 a4 4 0 0 1 -4 4 h-141.4 z" fill="#898781"/>
|
||
<text x="306.3913043478261" y="296.5" font-size="12" font-weight="400" fill="#1f2328" text-anchor="start" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">71.25%</text>
|
||
<rect x="595" y="220" width="1" height="92" fill="#c3c2b7"/>
|
||
<text x="584" y="236.5" font-size="12" font-weight="600" fill="#1f2328" text-anchor="end" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">PenguinHarness</text>
|
||
<path d="M597 224 h10.0 a4 4 0 0 1 4 4 v8 a4 4 0 0 1 -4 4 h-10.0 z" fill="#2a78d6"/>
|
||
<text x="623" y="236.5" font-size="12" font-weight="600" fill="#1f2328" text-anchor="start" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">$3.81</text>
|
||
<text x="584" y="266.5" font-size="12" font-weight="400" fill="#52514e" text-anchor="end" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">Claude Code</text>
|
||
<path d="M597 254 h113.5 a4 4 0 0 1 4 4 v8 a4 4 0 0 1 -4 4 h-113.5 z" fill="#898781"/>
|
||
<text x="726.5332606324973" y="266.5" font-size="12" font-weight="400" fill="#1f2328" text-anchor="start" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">$146.97</text>
|
||
<text x="584" y="296.5" font-size="12" font-weight="400" fill="#52514e" text-anchor="end" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">OpenAI Codex</text>
|
||
<path d="M597 284 h172.0 a4 4 0 0 1 4 4 v8 a4 4 0 0 1 -4 4 h-172.0 z" fill="#898781"/>
|
||
<text x="785" y="296.5" font-size="12" font-weight="400" fill="#1f2328" text-anchor="start" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">$220.08</text>
|
||
<text x="24" y="338" font-size="12" font-weight="400" fill="#898781" text-anchor="start" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">Each harness runs the model it is normally paired with: PenguinHarness on DeepSeek V4 Pro, Claude Code on Claude Opus 4.8,</text>
|
||
<text x="24" y="356" font-size="12" font-weight="400" fill="#898781" text-anchor="start" font-family="system-ui, -apple-system, 'Segoe UI', sans-serif">OpenAI Codex on GPT-5.5. Accuracy, Tokens and cost are suite totals at official pricing.</text>
|
||
</svg>
|
