Measuring engineers by their AI exhaust (token burn and lines of code) is a bad idea. Code generated and tokens spent are not the same thing as impact. A staff engineer can spend most of their time on non-coding activities (polishing a vision document, reviewing complex pull requests, influencing other teams, mentoring others) and still have more impact than an engineer who authored twice as many PRs.

In High Output Management, Andy Grove described this as leverage. The output of a leader extends beyond the work they produce directly into what they enable across the organization. Grove calls training one of the highest-leverage activities a leader can perform.

The more senior an engineer is, they are evaluated more heavily based on peer feedback and the impact of the work they led or contributed to, and less so based on the number of impactful PRs. Historically, this created an inverse relationship between seniority and direct technical output. As engineers became more senior, they were expected to find leverage other than coding.

But AI has changed our practice, and I think this is no longer the case.

Nobody knows the right way to code with agents

For a long time, the practice of software engineering was stable enough that a senior engineer could move away from primarily coding without completely losing touch with how software was made. Sure, languages, infra, and frameworks changed every now and then, but the basic principles remained true. The primary interface between humans and software was an IDE and a terminal. Engineers designed systems, wrote code, reviewed pull requests, tested, shipped, and built CI/CD.

Yet now, we have entered a period of actively re-litigating the interface between human intent and software. There is no settled way to build with coding agents, and technical leaders need firsthand experience with the new means of production.

Here’s a non-exhaustive list of open questions I am personally wrestling with:

On code review:

• Should engineers read all of the code?
• Should we mostly review the tests and resulting behavior?
• Is it okay to rely on AI to explain the code changes?
• Is one AI code review sufficient, or do we need multiple gates?

On context management:

• What goes into an AGENTS.md?
• What’s an effective context window?
• Should agents be allowed to invoke skills on their own?

On codebases:

• Should an LLM maintain a wiki of the codebase?
• Should Markdown plans be stored in the repo?
• Does codebase curation and organization still matter?

On autonomy:

• Should we work with agents as copilots, or should they work unsupervised?
• Should we adopt spec-driven development?
• Should agents have their own identity or act on behalf of the user?
• Should agents focus solely on coding or be connected to external services like GitHub and Jira?

Meanwhile, new tools pop up every day. Here are some tool categories (again, non-exhaustive) that I’ve been experimenting with:

UIs:

• Terminals
• Terminal multiplexers
• Agent GUIs
• Diff viewers
• Artifact viewers

Agent delegation:

• Harnesses
• Plugins
• Skill packs
• Orchestrators
• Task managers

Security:

• Sandboxes
• Pre-tool-use hooks
• Credential vaults
• Credential proxies

I don’t think we are close to agreeing on a standard way to build production software. These questions cannot be answered by reasoning about engineering principles. They are empirical questions.

You have to run agents against real codebases and find ways to push them into failing. You have to stand up your preferred stack, build your own custom tooling, and be ready to throw it all away and start again three months later. Otherwise, you cannot develop an intelligent position. Secondhand reports, demos, or articles will not suffice.

Seniority should produce more exhaust

An engineer uses coding agents to complete a task. A technical leader now has a broader mandate: figuring out effective ways to leverage coding agents collectively.

Where do agents work well? Where do they fail? What context do they need? What should humans review? Which controls should be deterministic? What should be standardized across the team, and what should be left to individual engineers?

To influence these decisions, you need a lot of firsthand experience. Staff and principal engineers should be running experiments, trying new tools, and pushing models the hardest against real problems. That should inevitably leave a sizable trace of AI exhaust: burned tokens, lines of code, failed prototypes, and abandoned branches.

Here are some conclusions I’ve reached through experimentation:

• On context windows: Even the most capable models, such as Fable and Sol, still make a ton of mistakes and ignore instructions. This is especially apparent at longer context windows. Thus, I have concluded that agents have an effective context window. I am still not sure what that is, and I suspect it is not a hard number and instead depends a lot on the task. Nonetheless, I am now capping my context windows at 50%, with a warning triggered after I pass 20%.

• On autonomy: Models are extremely agentic these days. They will file a PR, push to main, and build features you didn’t ask for. This is especially true when agent instructions are vague. I discovered this when I recently deleted my whole stack to start from zero. I cleared all my skills and my AGENTS.md, and I stopped using my spec-driven development (SDD) framework. I am now rebuilding my harness with instruction guardrails and deterministic hooks and workflows. Well-defined tasks upfront minimize surprises and keep agents more predictable.

• On code review: There’s still a lot of value in humans reviewing code. Agents will often fill in the blank when the spec or instructions do not specify a decision, which often results in over-engineered solutions, tests you didn’t ask for, or just outright terrible code. They also often will not tell you about it unless you ask. You might never ask because you forgot to put it in the spec in the first place. If something is critical, scanning the code before shipping a PR is absolutely still necessary. There’s a ton of innovation happening in human-agent understanding, and I’m watching this space closely.

• On maintaining codebases: Agents use grep to build context for a codebase. If your codebase is not greppable, agents will struggle to load all the necessary context for the task. A codebase is greppable when an agent that knows the domain words can find a concept and its wiring by searching without having to read whole files. I often have agents audit my codebase against principles like “names mean what they say,” “side effects have obvious owners,” and “directories have one state responsibility.” I don’t store Markdown files (like spec.md) in my repo because they end up being discovered with grep and then load unwanted context.

• On AI reviews: Agents will introduce bugs in their first pass at implementing a spec. Having an independent QA agent gate implementation work is effective at surfacing bugs. A QA agent from a different model family than the implementer’s model tends to produce better results. But QA agents tend to nitpick, so a human gate on what gets fixed is still needed because otherwise agents will be overly critical and introduce “defense in depth” that you don’t need.

Note that I gathered a lot of these conclusions by doing my own pet projects. In my professional work, I keep agents on a tight leash because much of what I own is critical.

The AI exhaust is not the contribution

High token consumption in itself proves nothing and should never be a performance target. The point I’m making is that serious experimentation produces visible exhaust. If you are testing frontier models, building harnesses, comparing workflows, studying agent failures, and pushing agents against difficult problems, you will consume tokens and generate code. High AI exhaust does not prove that someone is a technical leader, but low AI exhaust from those setting technical direction should raise questions.

In the past, a senior engineer could rely on delegation as a way to build influence in an organization because the underlying practice was well understood. In today’s world, the practice itself is the thing being designed. Thus, delegating experimentation means delegating the source of your own judgment. A new mistake would be assuming that seniority means you no longer need as much firsthand contact with how software is made. Perhaps our practice will stabilize, and that might be true again.