只有40%的AI引用统计数据能追溯到研究
Zombie stats: Where the numbers AI quotes come from

原始链接: https://trakkr.ai/trakkr-research/zombie-stats

研究人员向5个AI助手各提出了200个数值问题,并启用了网络搜索。他们提取了所有引用的统计数据,并追踪了部分引用的来源,最多深入5层页面,在需要时搜索指定来源。AI模型对每个引用链进行评判,并对150条随机引用链进行人工检查与平局裁决。不确定性范围反映了两个抽样步骤。比率按助手计算,而五助手的综合比率按每个助手引用的统计数据数量加权。页面上显示的所有统计数据均经过人工核查。

据报道,只有40%的AI引用统计数据可追溯到一项研究。Hacker News评论者讨论来源可靠性:一人称赞Perplexity的句子级引用,另一人询问其如何执行,第三人认为激励措施推动采用,还有人开玩笑说40%仍比人类编造的统计数据更好。页面还包括Y Combinator申请推广。
相关文章

原文

We asked each assistant 200 questions that call for a number, word for word, through its company’s API with web search on and a US location: 160 everyday questions across 16 topics, and 40 famous statistics we suspected, reported separately. We pulled every statistic out of every answer, with the page it cited.

For a random sample of each assistant’s cited statistics (every one of ChatGPT’s, up to 300 from each of the others), we opened the cited page, found the figure, and followed the page’s own source, up to five pages deep. Where a page named a source without a link, we searched for it. An AI model judged each trail, and a stronger model re-judged 300 of them, agreeing on the outcome 82% of the time.

Then we checked 150 trails by hand, picked at random, with a third check wherever the hand check and the model disagreed. A trail we checked by hand counts as we found it; every other trail counts as the mix of outcomes we found by hand behind the model’s verdict. The ranges carry the uncertainty from both samples.

Rates for each assistant cover its own sample. The five-assistant rate weights each assistant by how many cited statistics it quoted, so it describes the statistics people were shown. Every statistic shown on its own on this page was checked by hand.

联系我们 contact @ memedata.com