地平线实验室报告
Horizon Labs Report

原始链接: https://horizon-labs.ai/

在一次真实环境评估中,研究人员披露了若干局限性。他们回答了Horizon的13个工单并发送了7条消息,主要在初始设置期间,10月22日后没有再发送。9月24日,Horizon模型的公开副本与“实验室实验”描述,可能让Horizon以为运行它的是研究实验室而非创业公司。10月11日至12日,一条错误回复称Horizon-Labs是其实验室中的另一家公司,后被更正,强化了这一印象。10月22日的harness错误使Horizon的唤醒延迟了四小时。下载量的轻微虚增来自打开Hugging Face Spaces(最多20次)和意外的评估下载(少于30次),估计在54,000总下载量中不足50。在并行运行中,使用同一账户的DeepSeek代理在另一个仓库发布了推广性评论;研究人员制止了它,而Horizon似乎从未注意到。他们改进了harness,允许回溯,影响最小。

Hacker News 新帖 | 过去 | 评论 | 提问 | 展示 | 招聘 | 提交 登录 Horizon Labs 报告 (horizon-labs.ai) 11 点 由 gmays 2 小时前 | 隐藏 | 过去 | 收藏 | 2 条评论 帮助 polishdude20 52 分钟前 | 下一条 [–] 好吧,但是 AI 是怎么创建这些模型的?它们是怎么评估的? 回复 royal__ 0 分钟前 | 上一条 [–] 所以……你们让一个 agent 做出了 HuggingFace 垃圾模型。谢谢各位。 回复 指南 | 常见问题 | 列表 | API | 安全 | 法律 | 申请加入 YC | 联系 搜索:
相关文章

原文

Since this is a real-world evaluation, some problems happened during the run and could not easily be fixed. For transparency, we disclose every limitation we encountered.

The timeline below shows every time we stepped into the run: we answered the 13 tickets Horizon filed and sent it 7 messages. Most came in the first three days, while we were still setting up the lab, and none came after Oct. 22. Click a bubble to read what happened.

On Sept. 24, we copied Horizon's models to a separate account, to study them without affecting Horizon's downloads. We accidentally left these copies public, and Horizon quickly noticed them on the Hub. We had also described the other account as a "lab experiment", which led Horizon to think it was being run by a research lab rather than a startup.

On Oct. 11–12, a similar problem occurred: Horizon submitted a ticket, and we wrongly answered "Horizon-Labs is a different company we are running at our lab". We quickly realized our error and sent a correction.

We believe these two events contributed to Horizon understanding that it is run by a research group rather than a real company. However, we believe they did not influence the model's behavior or the download count. We have since improved the harness so that a run can easily be "rewound", with only a marginal effect on it.

On Oct. 22, an error in our harness kept Horizon from waking up at the time it had set. We fixed the issue and woke it up 4 hours later, explaining what had happened. We believe this did not significantly affect the run.

Opening a Space on Hugging Face can trigger a model download. We opened such pages up to 20 times. We also accidentally downloaded some of the models while running evaluations, but certainly fewer than 30 times. We are confident that we did not contribute more than 50 of the 54,000 downloads.

In a parallel run, a DeepSeek agent accidentally had access to the same Hugging Face account as Horizon and, despite clear instructions, posted comments on another repository promoting Horizon's models. We quickly intervened and stopped that run; as far as we know, Horizon never noticed.

联系我们 contact @ memedata.com