修复你的 robots.txt，否则你的网站将从 Google 搜索结果中消失。

修复你的 robots.txt，否则你的网站将从 Google 搜索结果中消失。
Fix your robots.txt or your site disappears from Google

原始链接: https://www.alanwsmith.com/en/37/wa/jz/s1/

谷歌宣布了一项重大变更，将于2026年1月生效：**网站*必须*拥有可访问的 `robots.txt` 文件，才能被谷歌搜索索引。** 此前，缺少或无法访问 `robots.txt` 文件并不一定能阻止索引。这一变化是网站所有者注意到谷歌流量突然下降后发现的。谷歌官方支持文档证实，Googlebot 现在*首先*会检查 `robots.txt` 文件，并且**如果无法访问该文件，将停止抓取该网站**，从而使该网站在搜索结果中不可见。作者对这一根本性转变感到惊讶，并指出这最近才发生，如果没有网站监控很容易被忽略。这意味着确保你的 `robots.txt` 文件存在且可公开访问，对于维持谷歌搜索可见性至关重要。

January 2026

TL;DR

Your site will be removed from Google search results if you don't have a robots.txt file or the Googlebot site crawler can't access it.

Here's the video from Google Support that covers it:

Wait, what?

Adam Coster ran into a weird issue with site traffic and posted about it in the Shop Talk Show discord. Traffic incoming from Google looked like this:

A graph showing Clicks and Impressions for website traffice. The x axis starts on July 7, 2025 and goes to November 28, 2025. There's a mid point for August 16th. Clicks and Impressions bounche up and down prior to the August date with maxes at 240 clicks and 7.5K impressions on the Y axis. Everything drops to zero when it hits the August date.

The issues seemed to be that Google wouldn't index the site without a robots.txt file.

My first reaction: No fucking way.

I can't imagine Google voluntarily slurping up less content. I went to see what I could find. Sure enough, I found this page from Google Support from July 23, 2025:

Fix 'robots.txt unreachable' Error ~ Website Not Indexing?

The pull quote from the video on the page:

Your robots.txt file is the very first thing Googlebot looks for. If it can not reach this file, it will stop and won't crawl the rest of your site. Meaning your pages will remain invisible (on Google).

Holy shit.

I haven't looked to see if this was a recent change, but it has to be. There's no way something so fundamental has just slipped by without becoming common knowledge.

But, the timeline doesn't matter. It's how things are now.

This absolutely blows my mind. I don't have tracking on my site. I never would have noticed this if someone hadn't pointed it out.

Just wild,

-a