In the previous article we took apart the reflect package and found that its magic is mostly the compiler leaving very good notes — type descriptors frozen into read-only data at build time, and a package that knows how to walk them. The whole article was about reading metadata that was already sitting in memory before main even started.

Today we shift the perspective. Profiling is the runtime catching your program in motion — sampling what it’s doing and where it’s spending its time, then accumulating that into something you can open with go tool pprof. The heart of every sample is a call stack, produced with the same unwinder we saw in the Stacktraces article . So profiling is really live moment-catching sitting on top of build-time stack-reading.

But Go doesn’t have just one type of profile — it provides five: CPU, heap, block, mutex, and goroutine. At first they look like five unrelated subsystems, but they all share the same skeleton, and once you see it, the whole thing collapses into one idea repeated five ways. (A sixth, a goroutine-leak profile, is on its way in Go 1.27 — Alex Rios has a great series of posts digging into it — but we’ll stick to the five that ship today.) Let’s start there.

The shape they all share

The deepest similarity between all five profiles is the thing you actually walk away with: the file. No matter which profile you collected, what lands on disk is the same format — and it’s surprisingly simple: the pprof profile, a gzip-compressed protocol buffer . It’s not even Go-specific; it’s the same format Google’s C++ profiler (gperftools) emits, and one go tool pprof shares with the wider pprof ecosystem. So before we look at how each profile is collected, let’s look at what they’re all collected into.

At the top level it’s one Profile message, and the part that matters is just a handful of repeated fields:

message Profile {
  repeated ValueType sample_type = 1;  // what each number in a sample means
  repeated Sample    sample      = 2;  // the actual data
  repeated Mapping   mapping     = 3;  // the loaded binary / shared libraries
  repeated Location  location    = 4;  // a PC, resolved to a place in the code
  repeated Function  function    = 5;  // name, file, start line
  repeated string    string_table = 6; // every string, deduplicated
}

The clever part is how little is stored inline. A Sample is almost nothing — a list of values and a list of location IDs, leaf first:

message Sample {
  repeated uint64 location_id = 1;  // the call stack, as references
  repeated int64  value       = 2;  // e.g. [sample count, cpu nanoseconds]
  repeated Label  label       = 3;  // extra key/value tags attached to the sample
}

Each of those pieces lives in its own table inside the Profile, and they all reference each other by id. Put together, the whole thing looks like this (a simplified visual of it):

What the diagram makes visible is the indirection. A Sample never holds a function name or even a stack frame — it holds a list of location_ids, one per frame in the stack trace, leaf first. Follow one of those into the location table and you reach a Location, which points (via a Line) at a Function and also records which Mapping it came from — the loaded binary or shared library that address lives in. The Function finally holds the human-readable bits — name, file, start line — except those aren’t strings either; they’re integer indices into the one shared string_table at the bottom. So it’s a chain of lookups: sample → location → function → string. And nothing is duplicated: the same Location, Function, and string are referenced by every sample that touches that spot in your code, so a stack that shows up in ten thousand samples stores its function names just once.

This is also where the abstract “value” of each profile gets its meaning: the sample_type declares what the numbers in each sample actually measure. That’s the one part of the shape that genuinely differs between profiles — same container, different labels on the columns — so we’ll fill it in as we get to each profile type in the rest of the article.

So much for what a profile is. Now let’s see how the runtime actually fills one in.

How the data gets there

Whichever profile you’re collecting, one thing is constant: somewhere along the way the runtime captures a call stack with the unwinder, and at the very end it all comes out as the pprof file we just saw. How a stack gets from the one to the other, though, is not the same across the five — and that difference is the real story.

The five sort into three collection models:

• CPU records asynchronously. A signal interrupts a running thread, the handler captures the stack and logs it into a ring buffer, and a separate background goroutine drains that buffer as it fills. A genuine streaming pipeline.
• Heap, block, and mutex record in place. When the event fires, the stack is hashed into a long-lived table of per-stack records and the matching record’s counters are bumped right there. Nothing streams and nothing drains in the background — the data just sits and accumulates, and the profile is assembled on demand, the moment you ask for it.
• Goroutine doesn’t record during execution at all. There’s no trigger and no buffer. When you ask for the profile, the runtime walks every live goroutine’s stack right then, in one snapshot.

So the thing that really changes from profile to profile isn’t a single knob — it’s the whole collection model. Let’s walk the five, grouped by the model they use.

CPU profiling: the signal-driven one

This is the most distinctive of the five because its trigger comes from outside the program entirely: the operating system interrupts a running thread with a signal, roughly 100 times a second, and asks “what were you doing right now?”

When you call pprof.StartCPUProfile, the runtime sets up the timers it needs to interrupt the running threads — so that time spent burning CPU gets sampled — and starts a background goroutine to collect the results (src/runtime/pprof/pprof.go:888). Hold on to that background goroutine — we’ll see in a moment why it matters.

Arming the timers is the easy part — the interesting bit is what happens when one of them fires.

Catching a tick

On every timer tick, the program gets interrupted: the thread is yanked out of whatever it was doing, mid-instruction, to run the handler, and then it’s expected to pick up exactly where it left off. That puts the handler in a very awkward spot — it can’t allocate memory, and it can’t take an ordinary lock (src/runtime/proc.go:5748).

The handler then captures the call stack with the unwinder, fills in the sample’s values and your labels (the ones you can attach with pprof.Do), and stores it. But remember, it can’t allocate or lock — so it needs somewhere to put the sample that demands neither. That’s exactly what it has: a structure designed for this constraint, a lock-free, preallocated ring buffer (src/runtime/profbuf.go:91), which can be appended to without allocating and without taking a lock. Being a ring buffer, though, it’s finite: if it ever fills up faster than it’s emptied, there’s no room for the next sample — so rather than make the interrupted thread wait for space, the buffer simply drops samples and counts how many it lost (losing a sample is fine; freezing a thread is not). Which sounds like a problem — nobody wants to drop samples on the floor. And that’s exactly where the background goroutine we started earlier earns its keep.

Its job is to drain the ring buffer as fast as it can — not to write anything out yet, but precisely so the ring rarely fills up and samples aren’t lost. Each batch it pulls out is folded into an in-memory map keyed by call stack (the profMap), where identical stacks are deduplicated and consolidated on the spot, bumping the count on the matching entry. The output file isn’t touched during the session at all. Only when you call StopCPUProfile, once the last samples have drained, does the builder make a single pass over that map and convert it into the pprof format — emitting every sample, location, function, and string in one go — and close the file you open with go tool pprof.

Let’s see the whole thing visually:

As we can see here, roughly 100 times a second the interrupted threads drop call stacks into the ring buffer, the background collector drains those and aggregates them into the profMap, and eventually — when you stop profiling — it’s all transformed and written out to the pprof file.

We’ve followed a single stack all the way from interrupt to file. But a stack on its own isn’t a profile — what makes it one is the numbers that ride alongside it.

The values each sample carries

The whole point of a CPU profile is to know how much CPU each stack uses — and that’s where the value[] slot on each sample comes in. A CPU sample carries two values: a plain count of how many times the timer caught this exact stack, and an estimate of CPU time in nanoseconds. The neat part is that the runtime never actually measures time — it only counts ticks. The nanosecond figure is just the count multiplied by the sampling period, and since the timer fires at 100 Hz, each tick stands for about 10 ms. So a stack caught 50 times is credited with roughly 500 ms, and “this function used 3.2 seconds” is really ~320 ticks times 10 ms each (src/runtime/pprof/proto.go:348).

CPU profiling’s trigger came from outside the program. The next profile’s comes from deep inside it.

Heap profiling: the allocator-sampled one

Heap profiling shares its bones with CPU — the unwinder captures a stack, and the result is the same pprof file — but it belongs to the in-place model, not the streaming one. A trigger fires, the stack is folded into a long-lived table right there, and the profile is assembled only when you ask for it. So we’ll focus on what makes it different, and three things do: where the samples accumulate, what triggers one, and what data each one collects. Let’s take them in that order, starting with the storage (which the block and mutex profiles share too).

A table of per-stack buckets

CPU profiling needed that lock-free ring buffer because it records from inside a signal handler, where it can neither allocate nor take a lock. Heap profiling records from inside the allocator instead — still a delicate place, but not a signal handler — so it can take a lock. And being able to lock is what lets it do something the CPU handler couldn’t: aggregate on the spot. So instead of a streaming buffer it uses a hash table of records keyed by the call stack, called buckets (src/runtime/mprof.go:75). Identical stacks collapse into the same record: a million allocations at the same line in your code don’t create a million entries, they all find the one bucket for that stack and bump its counters.

Let’s see how this changes the lifecycle of collection:

As we can see in the image, allocations are aggregated directly into the bucket table as they happen — no draining goroutine and no Start/Stop window, the bucket table just lives in the runtime and is always accumulating. Then, whenever the profile is required, it’s exported in a single pass to the pprof file.

That’s where the samples accumulate. Now, what decides when to take one?

The trigger: the allocator, sampled

The trigger is the allocator itself: every allocation goes through runtime code, so there’s nothing to interrupt — the runtime just notices and records the sample right there (profilealloc, src/runtime/malloc.go:2238).

Recording every allocation would be far too expensive, so it only samples some of them — on average one for every MemProfileRate bytes allocated, 512 KiB by default. It doesn’t sample on a fixed 512 KiB boundary, though, because a regular allocation pattern could fall in step with it and keep sampling the exact same spot. Instead the gap between samples is randomized around that average, so over time every allocation site has a fair chance of being caught. (MemProfileRate = 1 samples everything; 0 turns it off.)

We know when a heap sample gets taken. Here’s what it records when it does.

The data: allocations and what’s still live

Where a CPU sample carried a tick count, a heap sample carries four numbers, in two pairs: how many objects were allocated at this stack and how many bytes that came to (alloc_objects / alloc_space), and how many of those are still live — not yet freed — and the bytes they hold (inuse_objects / inuse_space). The bucket keeps a running tally of allocations and frees, and “in use” is just allocations minus frees.

That last part hides a subtlety. Allocations are counted the instant they happen, but frees aren’t — an object is only counted as freed once the garbage collector sweeps it, much later. If you just counted both as they arrived, any snapshot would be unbalanced: it would show lots of allocations but very few of their frees, simply because those frees hadn’t been counted yet. So the runtime deliberately delays counting: it arranges for an allocation and its eventual free to fall in the same accounting cycle, and only publishes a cycle once it’s fully settled — the “don’t announce the result until all the votes are counted” trick.

There’s one more adjustment: scaling. Since the profiler records only about one allocation per 512 KiB, each sample it keeps stands in for the many it didn’t. So before showing you the numbers, it scales each one back up to estimate the real total. The catch is that bigger objects are more likely to be caught (they take up more of that 512 KiB budget), so the runtime scales small, easy-to-miss allocations up a lot and large, almost-always-caught ones barely at all (src/runtime/pprof/protomem.go:16).

Heap profiling watched memory. The next two profiles watch something else entirely — time lost to waiting.

Block profiling: the waiting one

Like heap profiling, block profiling reuses machinery we’ve already seen — including the same stack-keyed bucket buffer — so again we’ll focus on what’s different. The trigger is contention: it fires when a goroutine finishes waiting on something — a channel, a select, a sync.Cond, a semaphore — and the runtime records how long it was blocked, keyed by the stack where it happened (blockevent, src/runtime/mprof.go:506). In short, it answers “where do my goroutines spend time waiting?”, catching the goroutine that did the waiting.

Each sample carries two values: a count of blocking events (contentions) and the total time waited, in nanoseconds (delay). The interesting part is how it decides which events to keep. Block profiling samples by duration: long waits are always recorded, and the shorter a wait is, the more likely it is to be skipped. So the long blocks worth chasing are never missed, while the swarm of tiny waits gets thinned out.

This selective keeping also means that we need to scale the numbers back up. If a short event was kept with a 1-in-20 chance, then for every one we see, roughly 19 like it were thrown away — so the runtime counts the survivor as 20 events, and inflates its delay to match. Doing that for every kept sample brings the totals back to an estimate of what you’d have measured if nothing had been dropped.

Block profiling tells you a goroutine waited, but not what it was stuck behind. That’s the gap the next profile fills.

Mutex profiling: the lock-contention one

Mutex profiling is block profiling’s twin — same bucket buffer, same two values (contentions and delay). What differs is which contention it watches. Where block profiling catches the goroutine that waited, mutex profiling catches the lock that made it wait: when a goroutine releases a lock that others were waiting on, the runtime records how long they waited, keyed by the stack (mutexevent, src/runtime/mprof.go:838). So it answers “where is my lock contention?”

Its sampling is simpler than block’s, though. Instead of weighting by duration, mutex profiling keeps a flat fraction — on average one event in every N, chosen at random regardless of how long the wait was. The scaling back up then mirrors that exactly: since each kept event stands in for N events, the runtime multiplies both its count and its delay by N, so 1-in-100 sampling reports totals about 100× the raw captured figures — again, an estimate of the true contention.

All four profiles so far do their recording while your program runs, each set off by an event — a timer signal, an allocation, a goroutine finishing its wait. The fifth records nothing until you ask.

Goroutine profiling: the on-demand snapshot

It breaks the mold completely: there’s no background trigger and no sampling. A goroutine profile is a snapshot — a census of every live goroutine and its current stack, taken the instant you ask for it.

The hard part is getting that snapshot without freezing the program. The obvious approach — stop the world, walk every goroutine, start it again — works, but with hundreds of thousands of goroutines that pause is brutal. So Go captures almost all the stacks while the program keeps running. During a brief stop-the-world it marks every goroutine “not yet recorded,” then lets the program resume while a collector walks the goroutines one by one. The clever bit: if the scheduler wants to run a goroutine the collector hasn’t reached yet, that goroutine records its own stack first. So the two cooperate instead of fighting — whoever reaches a goroutine first records it, nobody records it twice, and the one giant pause becomes two tiny ones with the work done concurrently in between (src/runtime/mprof.go:1356).

The value is the simplest of all five: each sample carries a single count (goroutine) of how many goroutines are sitting at that exact stack right now. Since it’s a one-shot census there’s nothing to sample or scale — identical stacks are just tallied together, so 10,000 goroutines blocked on the same channel collapse into one entry with a count of 10,000.

That’s all five. Step back, and the shared skeleton is hard to miss.

Summary

So that’s the five. They all end up as the same pprof file, and every sample is just a call stack — what really changes is how the stack gets recorded:

• CPU streams: a timer interrupts your threads, and from inside the signal handler (no allocating, no locking) each stack goes into a ring buffer that a background goroutine keeps draining.
• Heap aggregates in place: the allocator folds each sampled stack straight into a bucket table that lives in the runtime, and the file is written only when you ask.
• Block and mutex ride on that same bucket table, but fire on waiting instead of allocating — one catches the goroutine that waited, the other the lock that made it wait.
• Goroutine doesn’t record anything until you ask; then it just snapshots every live goroutine’s stack at once.

And the lesson underneath it all: profiling looks effortless because the runtime does the awkward work for you — capturing data in places where it can’t even allocate or lock, and lining up events that arrive at completely different moments into one consistent picture.

And with that, I’m going to close this series on the Go runtime — at least for now. We’ve gone from the scheduler and the memory allocator all the way down to how the runtime watches itself with profiling, and that feels like a good place to pause. Thanks for reading along; if there’s a corner of the runtime you’d still like to see taken apart, let me know — there’s always more down there to explore.