支撑我们过去十年所有游戏运行的后端系统。
The backend that ran every game we made, ten years and counting

原始链接: https://www.m2h.nl/writing/the-toolkit-behind-ten-years-of-games/

在过去十年中,一位开发者独自创建并维护了“游戏开发工具包”(GDT)。这是一个定制的后端系统,为包括《凡尔登战役》(Verdun)、《伊松佐河》(Isonzo)和《撞车嘉年华3》(Crash Drive 3)在内的七款游戏提供了支持,覆盖了11个平台。 该系统最初使用PHP在Google App Engine上构建,近期迁移至Cloud Run上的C#环境。GDT作为统一中心,负责处理分析、身份验证、远程配置、崩溃报告以及跨平台好友系统。通过优先考虑简洁性并拒绝繁重的框架,该系统保持了极高的效率,每月成本仅在10至50欧元之间。 该后端的成功源于“以存储桶代替事件”的理念,确保无论玩家数量如何,数据库成本都能保持在较低水平。它具备一个强大的Unity平台层,只需一次性处理各类主机的特殊需求,即可实现无缝的多平台发布。最令人印象深刻的是,该工具历经团队扩张和公司变迁,依然是多家工作室的核心基础设施。 通过规避复杂的“教科书式”云架构,创建者打造了一个稳定且易于维护的系统,实现了功能的快速部署与便捷调试。十年后的今天,GDT依然是“简单、专用代码优于现成复杂方案”这一理念的最佳证明。

这篇 Hacker News 讨论聚焦于 MikeHer 的一篇博文,文中详细介绍了支撑其游戏十余年的后端架构。 讨论帖的观点主要分为两派:一派批评文章的写作风格,另一派则赞赏其技术理念。几位用户对文中“Claude 味”浓厚的 AI 生成语气表示厌倦,认为作者本应对内容进行更彻底的润色。 然而,支持者认为文章所传达的核心信息依然很有价值。讨论强调了文章的一个关键要点:利用简单且经过验证的技术来实现快速部署的重要性。虽然一些用户将其与“快速行动,打破常规”(move fast and break things)的口号相提并论并表示怀疑,但另一些人指出,对于小型独立工作室而言,这种方法论是一种战略优势。通过将复杂性降至最低,一名开发者可以在一天内发布功能——这种敏捷性往往是那些规模更大、行动迟缓的游戏工作室所不具备的。归根结底,参与者认为尽管文章由 AI 辅助撰写,但其中关于维持可持续且简单的开发工作流的核心建议仍值得一读。
相关文章

原文

Every game I worked on in the last ten years talks to the same backend. Marooners, Verdun, Tannenberg, Isonzo, Crash Drive 2 and Crash Drive 3, and since this year a seventh game that another studio built on it. Steam, Epic, the Windows Store, PlayStation 4 and 5, Xbox One and Series, Switch, iOS, Android, WebGL. All of it reports to one Google App Engine project in PHP, backed by Firestore. I called it the GDT, short for Game Development Toolkit. The name sounds generic, but so is the tool: backend, admin, a small player portal for account linking and closed tests, and the Unity package, and it took on a new job with every game. The PHP version ran ten years without an outage a player noticed, and has now been replaced by a C# version. This piece covers the PHP years; the screenshots are from the C# version, and a section near the end covers what is new.

Our hosting bill for all of those games together was between 10 and 50 euros a month, depending on how many people were playing, most of them in the free mobile games. If nobody played, it would cost nothing. For the premium games the cost per paying player was negligible. I built it alone, and there has been almost nothing to maintain: Google kept the old PHP runtimes alive, the few moves to a newer one broke nothing, and it just runs. Keeping it simple, PHP included, has paid off tremendously.

It is also the part of our work nobody sees. Players see the game, reviewers see the game, and the thing that made every launch and every port possible sits behind an admin login. I have wanted to show it around for years. So here it is.

Games on it7, six of ours and one by another studio
Platforms11
In use2016 to now
BackendPHP on Google App Engine, Firestore, about 20,000 lines; C# on Cloud Run since September 2026
Monthly cost10 to 50 euros, all games together
Built and maintained byone developer

The Crash Drive 3 dashboard: new players per day, the last 300 players by platform and build, and active players per hour

The Crash Drive 3 dashboard, four years after launch. Of the last three hundred players to sign in, four in ten were on Android, a quarter on iPhone, a quarter on Switch. That mix is why we ship everywhere: the people who paid on Steam or Switch always have someone to drive into.

Where it started

Before this I wrote small PHP scripts for every game, lots of them, whenever a game needed something online: live news in the main menu, a counter, a version check, a little event. They were ugly and I kept writing them again from scratch.

What changed that was the Marooners console release. We were taking a party game made by six friends to PlayStation 4, Xbox One and PC, and I was the porting team, with Matt on testing and UI. Once the game left our hands I had no idea what happened inside it. On PC you get a Steam forum post when something breaks. On console you get silence, and later a certification failure.

The first attempt was Piwik with a Redis backend. I knew the database would be the bottleneck and Redis is very fast, so I had a freelancer set it up, because I had no idea how Redis worked. I never got it running properly. Power tends to bring complexity you did not ask for, and every time something went wrong I was reading someone else’s architecture instead of shipping. That is the one thing I cannot stand in a tool. I want to build a feature in the morning and have it live in the afternoon, and anything that gets in the way of that has to go. So I threw it out and built my own on App Engine and Firestore, in PHP, with no schema and no framework. I applied for WBSO, the Dutch R&D tax credit, for that rebuild. The money was modest, but writing it down as a serious project made me treat it as one.

A lot of it was built over a Christmas holiday. It was already working in Marooners, so dropping it into Verdun after the break took days rather than weeks, and someone on the WW1 team asked whether I had worked straight through the holidays.

From there it grew one game at a time. The WW1 games were made by WW1 Game Series, the company Jos, Matt and I founded, with a team that grew to twenty-five over the years. Crash Drive 2 was me again, with Matt on testing and UI. Crash Drive 3 was four of us. Every release added a feature or a platform, and every feature was built so that the older games kept working. The API root is still called api_v1; a new game would start on api_v2 the day backwards compatibility became impossible. That day never came.

What the backend does

People hear “analytics backend” and picture a dashboard with a line going up. It does a lot more than that, for teams of anywhere between two and twenty-five people:

  • 🐛 Crash and error reporting. Exceptions, crash dumps, bug reports with screenshots and log files, in the dashboard for that game.
  • 🔐 Cloud authentication. Steam tickets, Epic tokens, Windows Store, PSN. One endpoint, one user record per platform. Bans work per game or across all our games at once. A banned Steam player also gets a game ban on their Steam profile, and helping Steam out that way is satisfying.
  • 🎛️ Remote config. Typed settings per game that the client caches for a day. Toggle analytics, set minimum versions, schedule a cash boost weekend, change a balance number without a patch.

Remote config for Crash Drive 3: ad timing, the alpha switch, and a cash boost event with its dates

  • 📈 Online stats. Global counters that only go up, so a tampered client cannot rewrite history. Crash Drive 3 players have driven 149 million kilometers, drifted 35 million of them, and pushed 38 million barrels off cliffs.
  • 🧾 DLC and ownership checks for Steam and Epic, so the game does not have to trust the client.
  • 👥 Friends, invites and sessions across platforms, with Photon webhooks underneath.
  • 🗳️ Polls, vouchers, key sets per platform for press and testers, and a link tracker with clicks and, because the game reports back, actual conversions. That is how we learned that a link in Discord converts around ten times better than the same link on Twitter.
  • 🏗️ Builds. Jenkins is wired into the dashboard. One button starts a build, the result lands on the Steam latest branch by itself, another button makes it public. The build log keeps the timing of every step, which is how we found what made builds slow and kept an eye on it afterwards.
  • ⏱️ Benchmarks. The game runs a scene or a scenario, posts the numbers, and they land in the dashboard comparable across builds.
  • 💬 Discord. New reviews, curator mentions, sales milestones, crash spikes, cheaters, new builds, commits. Every Steam review lands in a channel with the reviewer’s playtime and whether Steam counts it toward the score. A player with 210 hours in Crash Drive 2 wrote “i truly miss playing this game!” and the whole team saw it within the hour. A negative review gets the same treatment, so we can respond almost immediately.

Steam reviews as they arrive in Discord, one message per review with playtime and whether it counts toward the score

  • 🧪 Closed testing. For the WW1 games and again for Crash Drive 3, we invited players who owned an earlier game of ours to test the next one. They sign in on the player portal, the backend checks on Steam that they own it and have played it enough, sends them to HelloSign for the NDA, confirms it came back signed, links their Discord and gives them the tester role, and hands out a Steam key. Hundreds of testers without a single spreadsheet.

The commit history page: SVN commits with a changelog per Steam branch

The same commits as they arrive in Discord through the post-commit hook, with the changed paths

  • 📜 A commit history page, first SVN and later Git, fed by a post-commit hook, plus Favro, Trello and UserEcho. The backend already knew everything, so it was cheaper to put those there than anywhere else.

All of that is roughly 20,000 lines of PHP, and this is how it fits together:

Architecture: Unity client, one App Engine project routed by URL, Firestore and Cloud Storage, platform APIs and Discord

The decisions that mattered

Alert on the rate, not the count

The most useful rule in the whole system: the crash alert does not look at the number of exceptions. It fires when exceptions per active user go over a threshold, currently 0.05, or when errors per session go over 1.5. On launch day that means thousands of players before anything triggers. Two years later, with a few hundred people online, the same rule still catches a broken patch within the hour. I have not had to retune it in years.

Two panels: absolute exceptions per day versus exceptions per active user over a two-year lifecycle

What this bought us, over and over, is speed on mysterious bugs. A crash that only happens on one platform, on one build, for a handful of players, shows up in the logs page with a screenshot and a log file within the hour. We fix it, press build, the build lands on the Steam latest branch, we press once more to make it public, and the alert goes quiet.

The build page: Steam branches with build IDs and depot sizes, and the build PC log with timings per step

Sales milestones that scale with the game

Under 100 owners the backend pings Discord every 10 sales. Under 1,000, every 100. Under 100,000, every 5,000. After that, every 100,000. On launch day the whole team watches the first hundred sales come in one at a time, and a year later nobody is being spammed. The messages come from a bot we call the Milestone penguin, and the penguin still visits: Crash Drive 2 passed 2.95 million unique players in March 2026, and that only counts players since the backend arrived. Before then it had done over 100 million plays on the web and over ten million Android downloads, when all I had was a store dashboard and a guess. I did not expect a small feature like this to do so much for morale.

Two visits from the Milestone penguin, the second one for Crash Drive 2 at 2.95 million unique users

Step chart of owners between Discord pings as a function of owners so far

Buckets, not events

The whole design bends around the database. It was called Datastore when I started and Firestore later, and either way it is not MySQL: no joins, limited queries, and a price per read and per write. Storing every event as its own row is too heavy that way. A game with thousands of players would write millions of rows a day, and reading them back for one chart would cost more than the hosting. So analytics are batched on the client and aggregated on the server into hourly and daily buckets, one row per hour per thing you count, with a running average. A dashboard for a game with thousands of daily players is a few hundred Firestore reads, cached hourly.

I did miss a real query language now and then. But if you plan the buckets ahead of time you rarely need it, and the trade is what keeps App Engine on one small instance and the bill tied to the player count.

The cracked build that poisoned the logs

At some point a pirated Crash Drive 3 build started submitting exceptions by the thousands. Not an attack, just a broken crack crashing on machines around the world. The fix was one config line: ignore any PC build below version 5000. Online play was never in reach for those builds anyway; cloud auth had already refused them. I had not planned for piracy as a data quality problem, but with remote config it took five minutes. The same version and build checks turned out to be a cheap way to spot players running modified executables, trainers and cheat builds, and to ban them.

Three crons for every job

Every scheduled job is registered three times, at minute 00, 01 and 02, and each run takes the games whose index matches its minute modulo three. One cron that walks every game in a row would run into timeouts and a load spike each hour. Three small ones finish comfortably and keep the instance count flat. It looks like a copy-paste mistake in the config file, and it is my kind of solution: one modulo, and no queue or scheduler to keep alive.

Hourly cron at :00, :01 and :02 with games spread across the three runs

Five attempts to repair broken JSON

Clients send garbage. Control characters from stack traces, invalid UTF-8 from device names, backslashes from Windows file paths, parental filters that rewrite the request body, and countries where the connection gets intercepted or blocked outright, Iran being the most memorable one. The analytics endpoint tries five increasingly aggressive repairs before giving up, and each successful repair posts a note to my debug channel saying which one it was. So I can debug production without ever touching a player’s machine.

Six parse attempts in a cascade, each feeding into the same success path

Cheaters, banned on context

The cloud auth call also sends along a bit of game data: playtime, currency, unlocked items, purchases. The server compares those against what is physically possible. Under an hour of playtime and every car unlocked, or ten million in-game cash without a single purchase, is an instant ban. Because the rules live in the backend, we can tighten them the same afternoon a new cheat shows up.

A ban can be for one game or for all of them, and a blocked device stays blocked across every game we make. The global ban list has over 130,000 entries by now, nearly 10,000 of them from Crash Drive 3 alone. Reacting fast has mattered more against cheaters than any detection scheme I could have built. The server assumes every client can lie. It checks everything itself, and the counters only go up.

The Unity platform layer: eleven modules, eleven platforms

The backend is why I started. The platform layer is why the whole thing paid off.

Every platform SDK wants the same things from you in a different shape: sign in a user, unlock an achievement, write a save, show presence, join a friend’s session. The Unity side of the toolkit defines one abstract module for each of those, and one implementation per platform.

Matrix of eleven modules against eleven platforms

From the game’s side it looks like this. These four lines are taken from Crash Drive 3 as it ships:

// Unlock an achievement, on whatever platform this build runs on
AchievementsModule.SP.SubmitAchievementStatus(SystemModule.SP.PrimaryPlayer, "FirstStunt", 100, null);

// Count something for the community totals
M2HGDT.AddQueuedOnlineStat("pushedCacti", 1);

// Read a value we can change without a patch
int secondsBetweenAds = OnlineSettings.GetInt("MinTimeBetweenAds", 240);

// Tell the platform what the player is doing right now
PresenceModule.SP.SetPresence(SystemModule.SP.PrimaryPlayer, PresenceDataCD3.GetData(scene, eventType));

The first line becomes a Steam, Epic, Xbox, Game Center or Play Games achievement or a PlayStation trophy, depending on the build, queued and retried and never resubmitted with lower progress. The second is batched with everything else the client reports and lands as one increase-only counter, which is where the 38 million barrels come from. The third is a typed lookup in a settings file the client refreshed that day, with a default for a client that has never been online. The fourth updates rich presence on Steam, Epic, Xbox and PlayStation, and does nothing where there is none. The game code is the same on all of them.

The game implements one abstract class that answers the questions the toolkit cannot: are we in the main menu, show this popup, is the player loading a level, join this session. With that, the toolkit handles console engagement screens, controller-to-user mapping, suspend and resume, cross-play invites and PSN activities on its own.

This did not make consoles easy. Certification is still certification. What it did was make sure each platform only had to be figured out once. I did the Marooners ports to PlayStation 4, Xbox One and PC alone. For the first Switch port and the first Xbox Series port we brought in an outside company, once each, mostly to save time. Every port after that, on every game, we did ourselves. When Crash Drive 3 shipped on twelve platforms on the same day, most of the platform work had already been paid for by an earlier game.

Why it stayed simple

PHP has no build step. Firestore has no schema. App Engine adds an instance when it gets busy and removes it when it does not. One entry point routes every request by URL, and a per-game config overrides the defaults. That is the entire architecture.

I have watched studios our size adopt textbook cloud architectures with queues, containers and a service per concern, and then spend their time keeping that alive instead of shipping. A Dutch Unity developer once told me about the multiplayer backend he had built with advice from one of the big cloud vendors. It was designed to scale to anything. It also cost at least a thousand euros a month at zero players, twenty times what ours costs with thousands of them online.

Every abstraction you add is something you have to understand at three in the morning when a launch goes wrong, and something you pay for every month whether or not anyone is playing. I wanted to be able to read the whole system in an afternoon, and I still can. Adding a feature was a new file and a new else if. Adding a game was a config file. Adding a platform was a folder of modules. Nothing ever needed a migration.

The doubt, and how it aged

There were moments I wondered whether building this myself was a mistake. Unity announced remote settings. Other companies launched hosted analytics and config products with nicer dashboards than mine had at the time. Switching would have been easy, and I remember thinking I might just be stubborn.

Then those products got retired, or repriced, or rebuilt in a way that would have meant rewriting our integration. My backend never changed underneath us, because nobody but me could change it. For a small studio that ships a game every year or two and supports each one for years after, that stability is worth more than any feature a hosted product could have given us.

What I would do differently, and did

Less than you might think, and while drafting this piece I did most of it: the lessons went straight into the C# version over the past weeks, so the toolkit got its biggest upgrade in years at the same time as its write-up.

  • C# instead of PHP, on Cloud Run instead of App Engine. Done: every game we have runs on it now. One language on both ends of the wire is the real gain. The request code sits in a project the Unity client shares, so a wrong field name is a compile error instead of an empty value in production. PHP was the right call for those years; the switch is about one language, not about PHP falling short.
  • A cleanup of the rougher features. Polls, vouchers and the link tracker were built for one game’s need and never generalized. They got it.
  • Less Photon in the core. The new Unity package has no Photon reference; a game that wants ping statistics in the dashboard hands it a small adapter.

What I kept: the buckets, the schemaless store, the redundant crons, the alert per active user, and a bill that follows the players. Those are the reasons it lasted.

2026: the C# version

The rewrite took the last two months. It is one C# server on Cloud Run against the same Firestore data, and Cloud Run scales to zero like App Engine, so the bill behaves the same. The URLs and the JSON stayed what they were, because a Switch build from 2021 is not likely to get an update and must not notice the difference. A comparison tool checked every read-only answer against the PHP, and a replay tool played a game’s live traffic against the test game before the switch. New games get a signed API next to it and a Unity package that shares its request code with the server, so a wrong field name is a compile error.

The package reads the way the old one did: GDTOnlineSettings.GetInt("MinTimeBetweenAds", 240), GDTFeatureFlags.IsOn("newGarage"), GDTAnalytics.TrackEvent("Gameplay", "MatchEnd", map, seconds). Consoles plug their storage and browser in through one small interface, so the package never references a console SDK and can live in a public repository.

The admin got the redesign it was time for, and the features I had wanted for years went in with it: settings with a history and values per platform, build range and language; feature flags rolled out to a percentage of players; live events on a calendar; bug reports grouped by exception, with the build that fixes them and automatic regression marks; one profile per player across games, with cloud saves to put back and an export and delete for privacy requests; leaderboards with per-platform groups; purchases checked with Apple and Google; an analytics wizard that writes the Unity call for the question you want answered; and heatmaps, which hosted products offered once and dropped. Positions are summed per cell before sending and per day on the server, so the cost stays flat.

Analytics: saved views, a picker instead of guessing, and a line per map

The wizard on the Analytics page: pick a goal, fill in the names your game uses, copy the call

A heatmap of the test game: a track, and a hairpin where the crashes pile up

The same admin in dark mode

These screenshots show the test game, with made-up numbers.

Ten years on

I am proud of this thing. It has proven itself on every launch we did, and it is still in daily use by people other than me: BlackMill Games, who make the WW1 games now, still run Verdun, Tannenberg and Isonzo on it, and when they built their new game this year they kept it rather than replace it. A backend that survives a change of owner is doing something right.

Every game we have runs on the C# version now, and the next one will from its first day.

One more thing, since every small Unity studio I talk to solves the same platform problem again from scratch: the C# version can run in another studio’s Google Cloud project, on their own bill, with their players’ data in their own account. If that sounds useful, you know where to find me.

联系我们 contact @ memedata.com