总是 DNS 出问题,那我们为什么还要自己运行?
It's always DNS, so why are we running our own?

原始链接: https://s2.dev/blog/dns-server

他们用自定义权威 DNS 服务器替代了 Route53,以支持无限数量的 basin,其中每个 basin 都成为一个子域名 CNAME。Route53 约每秒 5 次 UPSERT 的限制在滥用后造成积压,其他托管提供商也有类似限制。新的 Go 服务器使用 miekg/dns,并实时查询 Cloud Spanner 以获取一致的记录,支持 A、AAAA、NS、CNAME 和 SOA。ACME DNS-01 挑战仍委托给 Route53。为缓解 DNS 攻击,尤其是 NXDOMAIN,他们使用 Cloudflare DNS Firewall,并将源 IP 加入允许列表。服务器运行在两个区域 GKE 集群上,位于 TCP/UDP 负载均衡器之后,SOA/NS 返回 Cloudflare IP。系统现在可处理数千个即时可用的 basin,并且一直稳定。

Hacker News 链接到了一篇 s2.dev 文章,misker1 在其中分享了运行自定义 DNS 服务器的经验。该帖子有 9 个积分和 4 条评论。评论者询问使用 PowerDNS 与 CloudSQL、图表规模,并指出 Kubernetes 已包含 DNS。一位评论者讨论了 DNS 通常的最终一致性与文章中需要强一致性的张力,因为新 basins 必须立即解析;他们引用 RFC-1034 优先保证访问而非即时更新,推测 S2 客户端查询特定 DNS 服务器,并要求更清楚地解释 S2 和客户端访问。总体而言,讨论围绕自托管 DNS 的权衡与一致性要求展开。
相关文章

原文

This will be a rollercoaster story on how we ended up replacing Route53 and running our own authoritative DNS server.

At some point, we decided we should be able to provide unlimited basins to our clients. Basins are a core namespace construct on S2 and act as top-level containers for streams (already unlimited in number). We found that customers were hacking around our not-so-arbitrary limit of 100 basins per account. That’s not a cool experience.

For the purposes of this story, what is important is that each basin created by a customer becomes its own subdomain: my-basin.b.s2.dev (not different from the way buckets work on AWS S3). The DNS entry associated with each basin is a CNAME, which routes clients to the correct cell in our architecture. To support unlimited basins, we'd therefore need to support unlimited DNS records.

These records would need strong consistency too. When a basin is created, its address should be instantly resolvable, otherwise clients would be waiting around for a cached NXDOMAIN to expire.

My pulse went through the roof at the thought of running our own DNS server, yikes!

My research phase started with hope. Hope that our existing DNS setup with AWS Route53 would work with these new requirements. I really really did not want us to build and run our own DNS server.

Route53 is an awesome system and can, it appears, support an unlimited number of records. But, there's a fatal limitation: their API rate limits at a max of about 5 UPSERTs per second (AWS throttling docs have recently changed, the limits may be higher now).

This was something we discovered the hard way, actually. I did read the docs but the API limits didn’t raise any concerns. It became obvious when a nefarious actor rapidly created 10s of thousands of basins, saturating our entire quota of Route53 records. This generated a backlog of work that required a few hours to process, during which our normal DNS updates were stuck, with processes trying to delete records while others tried to add new records. It did make for a great post-incident review though. This was when we started enforcing the 100 basins per account limit.

We looked at the option of batching, but that would break our goal of making entries instantly available. And even with batching, the setup did not feel “unlimited”; it would still be possible for legitimate use of the system to hit the rate limit.

Other managed DNS providers we looked at (GCP Cloud DNS, Cloudflare, Azure DNS) have similar limitations: if you update/insert/upsert records using the provider API, there’s a heavy limiting component.

No way around it, a custom authoritative DNS server had to be built (double yikes!)

Our new authoritative DNS server

The technical requirements were pretty straightforward: given any DNS request that comes in, we need to query an up-to-date zone map, which should be able to hold unlimited records, and send a corresponding response back to the client.

To avoid syncing to a separate system and having yet another moving piece, the DNS records would come directly from our existing basin database. A live SELECT would consistently “resolve” the required mapping in the DNS request. In other words, we wanted DNS to simply become a new caller of our existing, strongly consistent account database.

Our account database lives in Google’s Cloud Spanner where performance was expected to fare well (given enough money that is).

Next came the core functionality.

We went back and forth between building on a library or writing a CoreDNS module. A library ended up feeling like the most flexible path, which would allow us to build something very specific to our use case. We are a Rust shop, but we decided to go with Go for this project, mostly because the Spanner tooling (and the GCP ecosystem) is comparatively more mature. We quickly settled on the miekg/dns library due to its excellent doc quality and the many RFCs under its belt.

We needed to be able to support authoritative DNS queries for several types of records: A, AAAA, NS, CNAME, SOA. Since we were using a library we had to build the responses ourselves. Our correctness testing focused on these, and consisted of unit tests, RFC compliance testing using Ferret (which we discovered in this great paper: SCALE: Automatically Finding RFC Compliance Bugs in DNS Nameservers) and a post-deploy smoke test with live data.

S2 uses wildcard certificates for the basin zone and they need ACME DNS-01 challenges to work. We decided not to serve the DNS challenge from our server and offloaded it to a separate Route53 zone using CNAME delegation, to avoid tweaking our cert renewal systems as well as serving dynamic TXT records.

We then went through a few planning sessions with a popular AI system. The design was simple enough, things made a lot of sense (even for me), except that the recommendations constantly included various extra features: coalescing, negative caching, throttling, rate limiting, DNS cookies, etc. Expected, for a system that would sit live on the open scary internet; however, it was a lot of suggested protection.

The more we pulled on this thread ... the scarier it got. There are many types of attacks that can happen at the DNS level, scary things I knew of and things I never heard of.

NXDOMAIN attack is the one that made us the most afraid. Every random non-existent DNS query would be a fresh database lookup. We tried building in-app defenses for NXDOMAIN however, it is, in my opinion, the worst DDoS attack and nothing can really stop it. Not cool.

Long story short, the possible attacks on DNS infrastructure scared me more than I would like to admit. We went looking for a DNS firewall solution. We were sure there must be something like a reverse proxy setup for DNS serving. Either this is a niche setup or it’s a very dangerous world, but for our requirements of unlimited records and DNS reverse-proxy firewall type, we found only two companies: Cloudflare and Akamai. Both giants in their own rights, both with similar offerings (it’s the kind that has no public pricing).

A few POCs later, we ended up picking Cloudflares DNS Firewall solution. Our DNS server origins were configured to only allow traffic from Cloudflare’s known IPs, a pretty standard reverse-CF setup (Cloudflare in front and nobody else gets to the origin, otherwise what’s the point). With this missing piece, we were now at our final design and ready to roll the new system into production.

As an authoritative DNS server, we have to reply to SOA and NS queries, and they point to nameserver IPs. We took extra care to ensure the IPs returned are always the Cloudflare IPs and not the load balancer ones, which included a ns-leak-test that ensures the origin IPs don't leak as part of some regression change.

The same authoritative server with protection

For the final architecture and HA (yes, failures will always happen), we run the DNS server on two separate GKE clusters (in separate regions, choosing the closest possible to Spanner), each with multiple replicas behind a TCP and UDP load balancer.

Observability and training everyone on the new system were another key component of the rollout. While I don’t think anyone gives Route53 a second thought and I don’t think anyone monitors it separately (managed services are someone else’s pager after all), we had to create alerts and runbooks with late-night processes to recover from failures.

There is one funny part to this setup: explaining the full delegation chain. The main domain is managed by Cloudflare in a regular zone. Then it delegates the basin subdomain to Cloudflare, but not a zone, to a service (DNS Firewall). That service then connects to GKE. And there’s another zone in Route53 for the certs (good luck understanding that, Codex).

This isn’t even its final form, the internal TODO includes tuning DNS caching and how to properly optimize between clients/DNS Firewall/DNS app (no amount of AI can solve that), future DNSSEC extensions and many more. It is an unknown system when it comes to failures we don’t see (or the way I like to think about this, how do you know things fail when they look fine).

Served queries

For now, things have been stable. Looking at stats for the past few weeks, live traffic looks good, the commit history for the past few weeks is mostly CVE fixes, dependency bumps and cost savings. Our customers now create thousands of basins, and they are instantly available for streaming.

I think we’re now in a place where we can blame DNS in one of our post-incident reviews.

联系我们 contact @ memedata.com