皇帝鹈鹕没有衣服

皇帝鹈鹕没有衣服
Why do LLM pelicans look like this?

原始链接: https://arcfu.com/post/the-emperor-pelican-has-no-clothes/

前沿AI模型在设计创造力方面仍不及人类设计师。一个新基准“Design Creativity Bench”用于评估AI在UI设计任务中的创造力。研究人员用80个开放式UI设计简报测试了16个前沿模型，每个简报都适配到两个产品领域，并附有人类设计师参考。它不采用单一质量指标，而是衡量三个维度：原创性（与其他模型输出的区分度）、创意范围（产品领域变化时的适应能力）和适当性（与简报的功能契合度）。结果显示存在显著创造力差距：最佳模型OpenAI GPT-6 Astra得分为65.5，而人类参考为92.3。完整排行榜、方法论和可复现评估框架正在发布，以使设计创造力成为前沿模型能力的一个可衡量维度。链接：排行榜、论文、HuggingFace数据集、GitHub。

该帖讨论了为什么各模型生成的“鹈鹕骑自行车”SVG图像看起来惊人的相似。评论者指出输出常共享相同的侧视构图、阳光海滩背景、围巾和运动线条。一些人认为这是模糊提示下的预期结果：LLM优化的是最可能、最通用的补全。另一些人将其视为更广泛的“多样性坍缩”或模式坍缩问题，RLHF/对齐加重了这一问题，后训练模型产生的输出比基础模型或人类更狭窄、更重复。几位用户表示，加入更多细节、风格或创意指令的提示可以提升多样性，而仅提高温度并不够。另一些人将LLM比作工具而非创意代理，认为人类必须提供品味和方向。讨论还涉及偏见（如从左到右的运动）、SVG/图标约定，以及“鹈鹕基准”等测试是否过拟合训练数据。总体而言，通用提示产生质心式结果，更大多样性需要明确提示或人类引导。

原文

Dipanjan Dey ✓

@Dipanjan_Dey

⚠️ Frontier AI models still fall significantly short of human designers on design creativity. ⚠️

Introducing Design Creativity Bench, a new benchmark evaluating AI models’ creativity on UI design tasks.

Despite rapid advances in their ability to generate functional websites and product UIs, leading AI models still struggle to make creative design decisions.

We evaluated 16 frontier models across 80 open-ended UI tasks, each adapted to two distinct product domains alongside human designer references.

Existing UI benchmarks often measure a single output quality metric. Instead, we measure three dimensions of design creativity:

– Originality: How distinct a model's designs are from other models' outputs for the same brief.
– Creative Range: How much a model adapts its design decisions when the product domain changes.
– Appropriateness: How well the generated UI satisfies the functional requirements of the brief.

The results reveal a substantial gap in overall creativity: The best model, @OpenAI GPT-6 Astra, scores 65.5 against a human reference of 92.3.

We're publishing the full leaderboard, methodology, and reproducible evaluation framework to make design creativity a measurable dimension of frontier model capability.

📊 Benchmark Leaderboard: https://kombai.com/benchmark/
📄 Paper: https://arxiv.org/abs/2610.11539
🤗 HuggingFace Datasets: https://huggingface.co/datasets/Kombai/design-creativity-bench
🔗 Github: https://github.com/kombai-io/design-creativity-bench/

皇帝鹈鹕没有衣服 Why do LLM pelicans look like this?

皇帝鹈鹕没有衣服
Why do LLM pelicans look like this?