皇帝鹈鹕没有衣服

皇帝鹈鹕没有衣服
Why do LLM pelicans look like this?

原始链接: https://arcfu.com/post/the-emperor-pelican-has-no-clothes/

前沿AI模型在设计创造力方面仍不及人类设计师。一个新基准“Design Creativity Bench”用于评估AI在UI设计任务中的创造力。研究人员用80个开放式UI设计简报测试了16个前沿模型，每个简报都适配到两个产品领域，并附有人类设计师参考。它不采用单一质量指标，而是衡量三个维度：原创性（与其他模型输出的区分度）、创意范围（产品领域变化时的适应能力）和适当性（与简报的功能契合度）。结果显示存在显著创造力差距：最佳模型OpenAI GPT-6 Astra得分为65.5，而人类参考为92.3。完整排行榜、方法论和可复现评估框架正在发布，以使设计创造力成为前沿模型能力的一个可衡量维度。链接：排行榜、论文、HuggingFace数据集、GitHub。

Hacker News 讨论串讨论了“大语言模型中的多样性崩溃”，使用的提示是“生成一幅鹈鹕骑自行车的 SVG”。各模型的输出惊人地相似——侧视图、从左到右、普通的海滩/道路、围巾、太阳——表明后训练/RLHF 导致了模式崩溃和输出多样性低。评论者争论这是真实局限，还是仅仅提示不够具体：要泛泛的东西，就得到泛泛的结果。有人建议增加温度/随机性，或加入随机主题词来改变输出，不过许多闭源模型限制这类控制。其他人指出，人类偏见也偏好某些构图，而训练数据/基准污染可能强化鹈鹕这个老套意象。有人认为创造力不是重点——用户往往希望自己简单的请求被照办，而不是模型自己发挥。少数人担心这个问题会影响编程、设计和自主智能体，因为它们会产生同质化解决方案；其他人则称其被夸大，或认为通过更好的提示就能轻松解决。总体而言，该讨论串认为大语言模型的雷同部分是对齐/训练的产物，部分是提示/期望所致，对其严重性没有共识。

DD

Dipanjan Dey ✓

@Dipanjan_Dey

⚠️ Frontier AI models still fall significantly short of human designers on design creativity. ⚠️

Introducing Design Creativity Bench, a new benchmark evaluating AI models’ creativity on UI design tasks.

Despite rapid advances in their ability to generate functional websites and product UIs, leading AI models still struggle to make creative design decisions.

We evaluated 16 frontier models across 80 open-ended UI tasks, each adapted to two distinct product domains alongside human designer references.

Existing UI benchmarks often measure a single output quality metric. Instead, we measure three dimensions of design creativity:

– Originality: How distinct a model's designs are from other models' outputs for the same brief.
– Creative Range: How much a model adapts its design decisions when the product domain changes.
– Appropriateness: How well the generated UI satisfies the functional requirements of the brief.

The results reveal a substantial gap in overall creativity: The best model, @OpenAI GPT-6 Astra, scores 65.5 against a human reference of 92.3.

We're publishing the full leaderboard, methodology, and reproducible evaluation framework to make design creativity a measurable dimension of frontier model capability.

📊 Benchmark Leaderboard: https://kombai.com/benchmark/
📄 Paper: https://arxiv.org/abs/2610.11539
🤗 HuggingFace Datasets: https://huggingface.co/datasets/Kombai/design-creativity-bench
🔗 Github: https://github.com/kombai-io/design-creativity-bench/

皇帝鹈鹕没有衣服 Why do LLM pelicans look like this?

皇帝鹈鹕没有衣服
Why do LLM pelicans look like this?