GPT-5.6、Grok 4.5、Claude 和 Muse Spark 构建了相同的 4 款应用
GPT-5.6, Grok 4.5, Claude, and Muse Spark build the same 4 apps

原始链接: https://www.tryai.dev/blog/gpt-5.6-build-off-12-models

此次评测对十二个 AI 模型进行了四项编程任务测试,每项任务均执行五次以排除变异性影响。参测阵容包括 GPT-5.6(Sol、Terra、Luna)、Claude(Opus 4.8、Fable 5)、Grok 4.5、Muse Spark 1.1 等前沿模型,以及若干开源权重模型。 **关键发现:** * **前沿模型优势:** 在构建 3D 光线追踪器或功能性魔方等复杂、创新性任务中,顶级模型明显优于开源权重模型。 * **任务简易度:** 对于“康威生命游戏”这类文档完善的任务,开源权重模型(Qwen 3.7 Plus、GLM-5.2)极具竞争力且性价比更高。 * **表现突出者:** GPT-5.6 Sol 在光线追踪器任务中表现出色;Claude Fable 5 在魔方开发和 SVG 设计方面名列前茅;Grok 4.5 被证明是高端模型的一种高能力且经济实惠的替代方案;Muse Spark 1.1 作为新晋选手展现了潜力。 * **结论:** 尽管旗舰模型在处理错综复杂的逻辑时仍是最佳选择,但它们并非万能。针对简单任务使用开源权重模型可大幅降低成本,而关键的创新性开发仍需依靠当前最先进(SOTA)模型的推理能力。 全部 240 次原始构建尝试均可供独立评估。

这篇 Hacker News 讨论帖探讨了 “tryai.dev” 项目,该项目要求包括 GPT-5.6、Grok 4.5、Claude 和 Muse 在内的多种 AI 模型构建相同的四个应用程序。 参与者们正在讨论这些对比测试的实用性和客观性。支持者认为,对模型进行横向对比能为性能、成本效益和设计感知力提供有价值的现实见解;例如,有人指出像 “Terra” 这样的模型在速度与成本之间提供了比其他模型更好的平衡。 然而,批评者质疑其方法论。一些人认为,构建计算器或基础游戏等重复性的“克隆”应用限制了对真正创新的评估,且忽略了市面上已存在数千个开源版本的事实。支持者则反驳称,由于大多数软件开发包含 80% 的样板代码,这些测试能有效衡量模型处理标准应用架构的能力。讨论还涉及了“基准测试”可能掩盖模型真实前沿性能的担忧,用户呼吁进行超越简单指标的更细致的评估。
相关文章

原文

Our last build-off hit the Hacker News front page, and the comments did not hold back. Fair enough, a lot of it was good feedback. So we took it, and with GPT-5.6 landing in three tiers (Sol, Terra, Luna) and Meta surprise-dropping a coding model (Muse Spark 1.1), we ran the whole thing again, bigger: twelve models, four apps, five attempts each.

What we changed based on your feedback:

  • You wanted open-weights models in the mix. So we added GLM-5.2, Qwen 3.7 Plus, DeepSeek V4 Pro, and Kimi K2.6 as comparison points, all served via Fireworks.
  • One attempt was weird, you said. Agreed. Every model now gets five attempts per task. Up top you get one sample run per model; each task table then says how many of the five we thought actually succeeded (and how we counted) and links the attempt we liked best; and every attempt is linked at the bottom so you can see how much these models swing run to run.
  • "This isn't objective." Correct, and we are not pretending it is. We are not handing down a scientific verdict. We generated a big pile of artifacts, we are publishing all of them, and you can form your own opinion. Everything below is just our observations from watching the results.

Want to skip straight to poking at the raw builds? Jump to every attempt and run them yourself.

The lineup, twelve strong: the new GPT-5.6 Sol, GPT-5.6 Terra, and GPT-5.6 Luna; Meta's Muse Spark 1.1; Grok 4.5, GPT-5.5, Claude Opus 4.8, and Claude Fable 5; plus the open-weights crew: Qwen 3.7 Plus, DeepSeek V4 Pro, Kimi K2.6, and GLM-5.2.

Here is each task: our pick of the five attempts playing up top, the cost and time for all five just below, and links to every raw attempt at the bottom of the post so you can judge for yourself.

Task 1: Doom-style raycaster maze

First-person raycaster you walk with WASD, shaded walls with depth, floor and ceiling, collision.

Grok 4.5Claude Opus 4.8GPT-5.5Qwen 3.7 Plus
Kimi K2.6Claude Fable 5DeepSeek V4 ProGLM-5.2
GPT-5.6 SolGPT-5.6 TerraGPT-5.6 LunaMuse Spark 1.1

How we counted "playable": the only question we cared about was whether you could actually walk through the labyrinth, move and turn. If yes, it counted.

ModelPlayableBest buildCost (5 runs)Avg timeOur take
Grok 4.55/5#40.27¢62sGreat for the price.
Claude Opus 4.84/5#20.54¢48sConsistent but dry.
GPT-5.54/5#11.44¢138sBest overall before GPT-5.6.
Qwen 3.7 Plus2/5#40.13¢43s
Kimi K2.62/5#21.37¢88s
Claude Fable 53/5#12.35¢107sGood results, but less consistent.
DeepSeek V4 Pro3/5#40.30¢318s
GLM-5.20/50.12¢133sRendered good detail, but the character could not move in any attempt.
GPT-5.6 Sol5/5#51.35¢120sBest results: consistent, better than GPT-5.5, and with more detail in the game overall.
GPT-5.6 Terra3/5#10.44¢39sGood detail, but less consistent, I couldn't always walk.
GPT-5.6 Luna5/5#50.15¢23sGreat results, though not as good as GPT-5.5 in my opinion.
Muse Spark 1.12/5#10.55¢169sSurprisingly awesome when it worked. Three of five were broken, but the working ones were on par with Fable and Sol, and better than the Grok and Opus ones.

Overall, Claude did worse than we expected here. GPT outperformed every other model, Grok was a genuinely usable alternative at its price point, and Muse Spark was a real surprise on the runs that actually worked.

Task 2: 3D Rubik's Cube (scramble + solve)

Build a colorful, 3D-looking Rubik's Cube with Scramble and Solve buttons that visibly animate the rotations.

Grok 4.5Claude Opus 4.8GPT-5.5Qwen 3.7 Plus
Kimi K2.6Claude Fable 5DeepSeek V4 ProGLM-5.2
GPT-5.6 SolGPT-5.6 TerraGPT-5.6 LunaMuse Spark 1.1

How we counted a "clean solve": we scrambled and solved the cube and only counted an attempt if both animations ran smooth, no glitches, no color changes.

ModelClean solveBest buildCost (5 runs)Avg timeOur take
Grok 4.53/5#40.65¢191sGood, simple results.
Claude Opus 4.80/5#10.56¢44sNo perfect example, everything had a small issue and the cube's colors would change, though some were close.
GPT-5.54/5#41.36¢136sGood results, with small nits like flickering colors or not-quite-smooth rotation animations.
Qwen 3.7 Plus1/5#50.07¢24sWhen it worked, it looked great!
Kimi K2.61/5#41.05¢59sNot great.
Claude Fable 55/5#22.03¢92sNo notes, really nice results.
DeepSeek V4 Pro1/5#30.35¢380s
GLM-5.20/50.08¢89sSurprisingly, no working results.
GPT-5.6 Sol4/5#51.06¢72sGreat results. One had weird animations and one rendered an all-black cube (interesting choice), but it did great on the ones that worked.
GPT-5.6 Terra4/5#40.42¢38sFine, with weird scramble animations, felt like a small step up from GPT-5.5.
GPT-5.6 Luna0/50.14¢24sUsually rendered correctly at first, but scrambling immediately broke it.
Muse Spark 1.12/5#10.54¢182sA step above the OSS models, but not something I'd use over Grok for the price.

Overall, we were surprised GPT underperformed here given its clear 3D lead in the raycaster. Claude did an amazing job again, though it was Fable carrying it with a clean five-for-five, while Opus, oddly, couldn't land a single flawless solve.

Play all five of any model live: Grok · Opus · Qwen (swap the number 1-5 for other attempts).

Task 3: Calculator

Digits, operators, clear, equals, correct operator precedence, real calculator look.

Grok 4.5Claude Opus 4.8GPT-5.5Qwen 3.7 Plus
Kimi K2.6Claude Fable 5DeepSeek V4 ProGLM-5.2
GPT-5.6 SolGPT-5.6 TerraGPT-5.6 LunaMuse Spark 1.1

How we counted "working": nothing exhaustive, just basic calculations like (((5 × 5) − 100) / 10) to see how each one handled order of operations and rendered the result.

ModelWorkingBest buildCost (5 runs)Avg timeOur take
Grok 4.55/5#50.37¢110sSimple, consistent results, no extra styling.
Claude Opus 4.85/5#10.46¢35s
GPT-5.54/5#10.91¢75sSometimes added extra buttons, and it tried to render in 3D, which often got cut off.
Qwen 3.7 Plus4/5#40.04¢12sWorked and pretty consistent, though one couldn't handle negative numbers.
Kimi K2.60/5#50.76¢50sWorked, but couldn't handle negative numbers.
Claude Fable 55/5#11.22¢48sMy personal favorite, purely on style.
DeepSeek V4 Pro3/5#30.28¢342sOne had out-of-order numbers, and one result didn't render.
GLM-5.22/5#10.07¢63sGreat results when it worked.
GPT-5.6 Sol5/5#10.84¢61sTries to go overboard on styling like GPT-5.5. That helped in the raycaster, but here it was annoying, some styling was off instead of clean and consistent.
GPT-5.6 Terra4/5#10.28¢20sGood results when it worked.
GPT-5.6 Luna5/5#10.11¢16sGood, consistent results, felt similar to Grok's.
Muse Spark 1.15/5#50.33¢94sPretty good overall with no glaring issues, on par with Grok 4.5. Occasionally out-of-order or oddly-placed buttons cost it some style points.

Overall, this was clearly Claude's best work: both Opus and Fable nailed all five, and Fable's was our favorite on style. GPT-5.6 Sol tries to go overboard with styles and render the calculator in 3D similar to Fable. However, it doesn't nail the styles, which leads to a worse overall experience. The simpler GPT models seemed to do a better job just because the experience worked out of the box. GLM-5.2, reasoning-off, also came back from nine-minute failures to a snappy, fraction-of-a-cent build.

Task 4: Conway's Game of Life

Grid canvas, Play/Pause/Step/Randomize/Clear, click to toggle cells, animated generations.

Grok 4.5Claude Opus 4.8GPT-5.5Qwen 3.7 Plus
Kimi K2.6Claude Fable 5DeepSeek V4 ProGLM-5.2
GPT-5.6 SolGPT-5.6 TerraGPT-5.6 LunaMuse Spark 1.1

We did not run a separate five-attempt scoring pass on Life, so here it is just cost and time plus the general impression below.

ModelCost (5 runs)Avg time
Grok 4.50.14¢38s
Claude Opus 4.80.48¢35s
GPT-5.51.06¢79s
Qwen 3.7 Plus0.04¢11s
Kimi K2.60.93¢53s
Claude Fable 51.27¢48s
DeepSeek V4 Pro0.25¢304s
GLM-5.20.10¢121s
GPT-5.6 Sol0.99¢62s
GPT-5.6 Terra0.36¢25s
GPT-5.6 Luna0.13¢18s
Muse Spark 1.10.32¢98s

Grok 4.5 did well here, but the bigger takeaway is that this task was simple enough for the OSS models to do extremely well on. There is probably enough open example code for Game of Life out there that they were able to do a much better job at a way lower cost. Qwen 3.7 Plus and GLM-5.2 are clearly the go-to for something like this, but I would not rely on them generally, the other tasks show they still struggle with genuinely novel or more complex work.

The receipts: raw speed and cost (short answers)

Separate question, separate table. This is our standard latency harness (three short prompts, five reps, 400-token cap), not the build tasks. tok/s is output tokens over wall-clock, uniform for all.

ModelMedian latencyThroughputCost / reply
GPT-5.6 Luna1.0s97 tok/s0.001¢
GPT-5.6 Terra1.5s62 tok/s0.001¢
GPT-5.6 Sol1.8s45 tok/s0.003¢
Qwen 3.7 Plus2.1s204 tok/s0.001¢
Claude Opus 4.82.5s44 tok/s0.004¢
GPT-5.53.0s45 tok/s0.003¢
Grok 4.53.0s112 tok/s0.003¢
Muse Spark 1.13.1s125 tok/s0.002¢
Kimi K2.64.5s83 tok/s0.01¢
Claude Fable 56.6s30 tok/s0.01¢
GLM-5.27.0s58 tok/s0.001¢
DeepSeek V4 Pro9.3s37 tok/s0.001¢

One honest caveat: several open-weights buffered their whole reply in a burst and hit the 400-token cap, so their tok/s is a ceiling, not a true decode rate. The clear read: the GPT-5.6 tiers are the snappiest models here on short prompts (Luna answers in about a second), Qwen is absurdly cheap and fast, and DeepSeek and GLM are the slowpokes, which matched the agony of generating their apps.

Bonus: draw a horse riding an astronaut (SVG, best of 5)

One-shot SVG, no libraries. We show each model's best-of-five (we prefer a strictly-valid SVG, then the most detailed).

Personally, Claude Fable does a great job with the SVG rendering. It's funny most of the time as well and came up with good quality results. The GPT-5.6 models were surprisingly lackluster here since none of them included clean renderings of the horse or the astronaut. Grok 4.5 did pretty well here as well.

Bonus 2: Elon and Bezos watch a Blue Origin landing (SVG, best of 5)

A harder scene by request: two recognizable tech-billionaire caricatures watching a Blue Origin booster land on a pad in the open ocean. This one leans on composition and likeness. Again, Claude Fable sweeps the field: it produces great detail with a clean render, a shiny spot on Bezos' forehead, and smoke around the landing pad. GPT again produces pretty cartoony results; if that's something you prefer then maybe you can try improving the one-shot results it produces, since there are always some minor errors with the generations. The OSS models, GLM-5.2 and Qwen 3.7, also did really well for this task.

TL;DR

  • The frontier still wins the hard tasks, and it is not particularly close on the complex ones. GPT-5.6 Sol and Claude Fable 5 were the standouts: Sol excelled at the raycaster, Fable at the Rubik's cube. Everything else did decently, but the best results are still at the frontier.
  • There is a clear gap between SOTA and open-weights on genuinely novel or complex work, and you can see it in the raycaster and cube. But on a simple, well-trodden task like Game of Life, the OSS models hold their own, there is enough example code out there that Qwen 3.7 and GLM-5.2 nail it at a fraction of the cost. Use them for that class of problem; just do not lean on them generally, because the other tasks show they still struggle with the hard stuff.
  • Grok 4.5 is genuinely "Opus level" on some tasks. If I cared about cost for my business, I would reach for it as a secondary execution model without hesitation. See Grok 4.5 vs Opus 4.8 and Grok 4.5 vs GPT-5.5.
  • Muse Spark 1.1 pleasantly surprised me. I was not really expecting much, and it felt a step below Grok 4.5 but generally better than the open-weights. It is a real debut, though not something I would reach for just yet.

The takeaway holds even after launch day: the newest, most expensive flagship is not the automatic winner. Want to run these prompts yourself? Every model here is on one TryAI account, pay-as-you-go. Browse the models and start your own build-off.

Play with every attempt

Every raw build we generated, twelve models times five attempts per task. Click any number to open that exact attempt and poke at it yourself; the swings between attempts are the point.

Raycaster maze

ModelAttempts
Grok 4.51 · 2 · 3 · 4 · 5
GPT-5.6 Sol1 · 2 · 3 · 4 · 5
GPT-5.6 Terra1 · 2 · 3 · 4 · 5
GPT-5.6 Luna1 · 2 · 3 · 4 · 5
Muse Spark 1.11 · 2 · 3 · 4 · 5
GPT-5.51 · 2 · 3 · 4 · 5
Claude Opus 4.81 · 2 · 3 · 4 · 5
Claude Fable 51 · 2 · 3 · 4 · 5
Qwen 3.7 Plus1 · 2 · 3 · 4 · 5
DeepSeek V4 Pro1 · 2 · 3 · 4 · 5
Kimi K2.61 · 2 · 3 · 4 · 5
GLM-5.21 · 2 · 3 · 4 · 5

Rubik's Cube

ModelAttempts
Grok 4.51 · 2 · 3 · 4 · 5
GPT-5.6 Sol1 · 2 · 3 · 4 · 5
GPT-5.6 Terra1 · 2 · 3 · 4 · 5
GPT-5.6 Luna1 · 2 · 3 · 4 · 5
Muse Spark 1.11 · 2 · 3 · 4 · 5
GPT-5.51 · 2 · 3 · 4 · 5
Claude Opus 4.81 · 2 · 3 · 4 · 5
Claude Fable 51 · 2 · 3 · 4 · 5
Qwen 3.7 Plus1 · 2 · 3 · 4 · 5
DeepSeek V4 Pro1 · 2 · 3 · 4 · 5
Kimi K2.61 · 2 · 3 · 4 · 5
GLM-5.21 · 2 · 3 · 4 · 5

Calculator

ModelAttempts
Grok 4.51 · 2 · 3 · 4 · 5
GPT-5.6 Sol1 · 2 · 3 · 4 · 5
GPT-5.6 Terra1 · 2 · 3 · 4 · 5
GPT-5.6 Luna1 · 2 · 3 · 4 · 5
Muse Spark 1.11 · 2 · 3 · 4 · 5
GPT-5.51 · 2 · 3 · 4 · 5
Claude Opus 4.81 · 2 · 3 · 4 · 5
Claude Fable 51 · 2 · 3 · 4 · 5
Qwen 3.7 Plus1 · 2 · 3 · 4 · 5
DeepSeek V4 Pro1 · 2 · 3 · 4 · 5
Kimi K2.61 · 2 · 3 · 4 · 5
GLM-5.21 · 2 · 3 · 4 · 5

Game of Life

ModelAttempts
Grok 4.51 · 2 · 3 · 4 · 5
GPT-5.6 Sol1 · 2 · 3 · 4 · 5
GPT-5.6 Terra1 · 2 · 3 · 4 · 5
GPT-5.6 Luna1 · 2 · 3 · 4 · 5
Muse Spark 1.11 · 2 · 3 · 4 · 5
GPT-5.51 · 2 · 3 · 4 · 5
Claude Opus 4.81 · 2 · 3 · 4 · 5
Claude Fable 51 · 2 · 3 · 4 · 5
Qwen 3.7 Plus1 · 2 · 3 · 4 · 5
DeepSeek V4 Pro1 · 2 · 3 · 4 · 5
Kimi K2.61 · 2 · 3 · 4 · 5
GLM-5.21 · 2 · 3 · 4 · 5

Try it yourself

Every model mentioned here is available on TryAI with one account, pay-as-you-go, no subscription.

Start free
联系我们 contact @ memedata.com