One of the most important engineering decisions we made this year was sending the following emoji in our company Slack channel.
If you’d told me that 2 years ago, I probably would’ve quit on the spot; assuming, naturally, that the shitcoiners trying to infiltrate our Discord community had done so successfully and I was an NFT shill.
But this is not that. The green toad is called Bufo, and the yellow rope-looking thing is an andon cord. Stay with me.
What’s an andon cord?
It’s an actual, physical rope, originally used in Japan’s manufacturing industry, most notably Toyota factories, and popularized in Western culture through Mike Rother’s Toyota Kata. In the event of a defect, bottleneck, or other known issue in a factory, an employee would pull the andon cord and all factory production would stop.
There’s no point in continuing production if a bottleneck exists in the assembly line, because the productivity losses caused by the bottleneck (backpressure on upstream components, underutilization of downstream components) would exceed those caused by fully stopping production to fix the issue.
Productivity gains aside, pulling the andon cord was a celebrated cultural norm. The worker who pulled it was thanked and praised, creating a culture of continuous improvement and discouraging the normalization of production issues.
What does this have to do with software?
In Hatchet’s early days, we spoke often of the kind of culture we’d like to build. One of our founding team members mentioned the andon cord as a cultural norm they’d appreciated in a previous job. It immediately resonated with us, despite the reflex opposing anything you might read at Wharton.
Putting my best managerialist hat on: every software startup is a factory. The raw materials are computers, human brains, and your other tooling. The output is software delivered to end users.
Our assembly line involves a lot of steps. Writing code is a small part of the process. Gathering user feedback, getting feedback internally, reviewing PRs, staging deployments, load tests, and production rollouts comprise the bulk of the work.
Anything that impacts our ability to ship and deploy a stable, reliable platform impacts our bottom line. We decided that slowdowns or bottlenecks in our factory would be the overriding priority.
And thus, :bufo-pulls-the-andon-cord: was born.
When everything’s highest priority, nothing is
Note the emphasis on overriding priority.
We’re an ambitious team, so we tend to bite off a lot of work at once. When a customer asks for something, and we think it’s a great feature request, we usually make it a priority. When we notice a small spike in latency on some critical-path service, we also make it a priority. When we postmortem an outage, it’s the priority. And so on, until the priority tickets pile up.
In a world of hundreds of high-priority tickets, it becomes impossible to see through the noise. You’re constantly shipping, fighting fires, and feeling like you’re falling behind.
The andon cord makes the ultimate priority very clear: there is nothing more important than our ability to ship reliable software. A clear and resolvable problem in our software factory is more important than anything else (yes, even new features and launches). It’s worth stopping all work to resolve it.
Our andon cord-worthy problems
Ok, enough managerial fluff. Let’s get into some concrete examples.
For some background on our infrastructure setup, Hatchet runs on over 25 shards in 3 regions. Our busiest shards are running tens of thousands of transactions per second. This volume naturally makes certain things difficult. We also have a product with very strong reliability and correctness guarantees, which means there are a lot more steps in the process for getting a new feature shipped than you might see in, say, OpenClaw.
Here they are, in chronological order:
- We didn’t have a sensible rollout strategy, leading to releases piling up in our backlog, and multiple big risky changes being shipped simultaneously. Before continuing deployments, we rewrote our entire deployment process. Work paused for 1 week.
- We didn’t have good enough observability, so customers alerted us about unexpected latency before we were aware of it. We started very closely tracking latency metrics and instrumented all critical-path endpoints. Work paused for 2 weeks.
- We had a ton of observability data, but we were over-alerting and over-paging. We stopped all work to move towards zero error logs in production. Work paused for 3 days.
- Our staging environment wasn’t catching enough issues during the load testing process. We rolled out a new environment called
staging-chonkyand rewrote our load testing harness. Work paused for 1 week.
These days, we still have andon cord moments, but they’re resolved very quickly, and usually by one or two team members, instead of the entire team. We’ve ramped up our shipping velocity significantly in the past few months. This past week, we shipped 3 new core engine features, and a ton of smaller improvements and fixes.
It’s slightly counterintuitive: you would think that throwing a signal interrupt every few weeks would lead to more noise and overhead. And in the short term, it’s painful. But the core idea is to reduce the amount of unplanned work you’re doing each day.
(At this point, if you think this is sounding more and more like The Phoenix Project, you’re correct. It’s quickly become required reading at our little shop.)
The burden of unplanned work
At startups, there’s a continuous pressure to ship all of the time. Dropping everything to invest in our tooling and process is rare among startups that I talk to; most teams tend to normalize and adopt the mentality that “things are on fire because we’re a startup.”
The issue is that there is nothing more burdensome than unplanned work, and it’s usually invisible to teams when it becomes normalized. This unplanned work regularly produces heroic, monumental efforts—like pulling an all-nighter to resolve some critical incident—which just leads to more pressure to meet tight deadlines, which leads to more unplanned work, and so on.
I used to think this was just part of startup culture. But culture is just your habits and tendencies, and those aren’t inevitable. It just might require some discipline to shift it.
PostScript: Bufo has PMF
An aside to the aside: Bufo is a genus of true toads in the amphibian family Bufonidae.
This post is not about Bufo, but at this point you might be wondering who he is.
We provide a lot of customer support through shared Slack channels. These are Slack channels which are connected from one workspace to another. This also means that we get to see which emojis other companies are using.
Earlier this year, we started seeing this little toad emoji reacting to various messages in our Slack channel. I only know of 2 named toads: Toad from Frog & Toad, and Pepe.
Pepe was born out of 4chan, the birthplace of incel culture, and in my mind has a fairly negative connotation. Thankfully, a quick google search revealed that this is a different green toad called Bufo.
Within a week of seeing Bufo, he appeared in our internal emoji set, much to the chagrin of our overly curmudgeonly head of growth, who attempted her own andon cord moment of sorts:
It didn’t work. And within a few months, Bufo had successfully infiltrated at least five other shared Slack channels.
I’ve never seen clearer signs of product-market fit.