You can think of speculative execution in terms of hiking. At some point you arrive at a branch and have to predict where to go. If the prediction was correct, you saved some time and could enjoy the sun and a refreshing drink at a mountain hut. However, if you speculate in the wrong direction, you have to turn back. The trail looks untouched, but your footsteps remain in the mud. 

Speculative execution in CPUs works similarly. The branch prediction performs an educated guess about a branch’s outcome ahead of time and the CPU speculatively executes it. If the prediction was correct, speculative execution saved some time. However, if the prediction is incorrect, the CPU has to discard the results, roll back and execute the other branch. Because these speculatively executed instructions only exist temporarily in the CPU pipeline and are never permanently retired or committed, the literature refers to them as transient instructions and generalizes the concept as transient execution.

However, due to the transient execution, there are still some traces left in the microarchitectural state for instance in CPU caches. Thus, an attacker can use Spectre to transiently access memory out of bounds, encode a single bit of information into the cache state and exploit the latency of reaccessing data to infer whether the bit was set or not. 

To mitigate against in-process Spectre attacks, Cloudflare Workers freezes local timers, disallows multithreading and shared memory and actively detects, periodically shuffles memory and isolates malicious-looking scripts into separate processes.

Attack primitives

The Cloudflare Workers platform deliberately restricts timers. During CPU-only execution, time is effectively frozen. Date.now() and performance.now() do not provide a continuously advancing high-resolution clock. There is no shared memory and no multithreading, so the classic counter-thread timer via a SharedArrayBuffer is not available. 

To successfully mount an attack, several challenges have to be solved. First, Workers runtime is limited and co-location between an attacker and victim has to be guaranteed. Second, a reliable, ideally co-located, remote timer has to be discovered, which allows stable timing measurements.
Third, the attack runs under production conditions, meaning it requires additional stability measures such as a reliable Spectre gadget enabling transient 64-bit out-of-bounds accesses, robust signal amplification to deal with systems and networking noise, and a primitive to reliably evict data out of the cache. 

Spectre gadget

return probeArray[
          obj instanceof ObjP
            ? PROBEARRAY_OFFSET + ((obj.ptr[0] >> bit) & 1) * 0x800
            : 0x400
];

Speculative type confusion Spectre gadget

With the right Spectre gadget (snippet above), an attacker can transiently access out-of-bounds memory and encode a single bit into the cache (probeArray). The attacker then measures the memory access latency to confirm whether data has been cached or not. A faster access means the line was cached and the bit was 1. Conversely, a slower access means it was uncached and the bit was 0. In our attack, we use two different Spectre gadget types. The first one leaks compressed heap pointers, e.g., the isolate’s heap base address (root), and the other one leverages a speculative type confusion to leak from an arbitrary, attacker-crafted userspace 64-bit pointer. At the time of performing the research, the V8 Sandbox was not yet implemented at Cloudflare Workers. Under pointer compression, most objects use 32-bit compressed pointers. TypedArray was one of the few exceptions that still stored a raw 64-bit pointer to its backing store, which is exactly what our gadget abuses.

The branch obj instanceof ObjP performs a type check, i.e., a branch. To mistrain the branch prediction, we call the gadget many times on real ObjP instances, then call it on a different object with an attacker-controlled memory layout ObjI. The CPU speculates on the taken branches and follows obj.ptr[0], even though the object has a different type. To leak a single bit, we mask out one bit and use it to select one of two probeArray lines. Whether that line is cached encodes the bit. 

Exploiting the heap leakage gadget, we map neighboring objects and locate an attacker-controlled array. Our second gadget confuses two large objects that span several cache lines, so the type field lands on a different cache line than the field we read. Evicting the type field opens the speculation window while the target field stays cached, and the transient read follows an attacker-controlled 64-bit value. That turns the leak into an arbitrary-address read. A more thorough description of this technique can be found in the paper.

Local demo of leaking an arbitrary 64-bit address.

Signal amplification

A cache hit and a cache miss differ by a few nanoseconds. Moreover, a remote timer is noisy at the scale of a few microseconds up to a few milliseconds. Therefore, some form of signal amplification is required to differentiate a cache hit from a miss. Stephen Röttger and Artur Janc discovered a way to amplify a single memory access, by exploiting the tree-based pseudo least recently used (PLRU) cache-replacement policy in L1 caches. Tree-based PLRU organizes each cache set as a binary tree whose nodes point to the side used least recently, so the CPU evicts by following those pointers. With the right access pattern, an attacker can keep a target line cached indefinitely by touching its tree neighbor whenever the pointers turn toward the target. Quite elegant, right? Leveraging that behavior, the timing of a single cache event can be arbitrarily amplified such that it leads to a lot of L1 hits (faster) compared to lots of L1 misses in the opposite case.

The figure below illustrates whether a memory address X is cached or not. If it’s not cached, the access pattern leads to a lot of cache hits. If it is present, it occupies one node in the tree, and subsequently four cache lines try to fit into three nodes, which results in a lot of L1 misses.

Remote timer

As long as the signal can be amplified, a noisy remote timer is sufficient to differentiate an encoded bit. For instance, a WebSocket connection to an external server serving high-resolution timestamps is enough. The timer could be hosted at Cloudflare or at a co-located data center to the target data center running the Worker. The Worker asks the remote timer to mark a timestamp for a certain event and compute the delta for another request once the event has stopped. 

In the paper, we evaluated several different timer setups and were able to reliably achieve sub-ms resolutions on the Median with only a handful of samples even over larger topological distances. The figure below shows an amplified cache event using the tree-based PLRU amplification.

Repeatable measurements  

A single measurement is not enough to differentiate timing-encoded data reliably. Production machines are noisy, thus an attacker has to repeat each measurement at least a few times and use some statistical discriminator. Repeating a measurement in our case means resetting the cache state. Two things have to be uncached before each round. The value the speculative branch depends on has to be evicted, so branch resolution stalls long enough to open a speculation window. The probe line that encodes the leaked bit has to be evicted, so the next transient access can re-cache it.

Since there is no direct instruction available in JavaScript, the classic way to do this is to build an eviction set. An eviction set is a group of addresses that map to the same cache set as the target. Accessing them in the right pattern pushes the target out of the cache. In their attack, Stephen Röttger and Artur Janc used an eviction list to reliably evict at least into the L2 cache. This works, but it is expensive. Constructing a precise eviction set requires many timed measurements, and our timer is a noisy remote timer. The previous remote attack against Workers sidestepped the search by traversing an array larger than the L1 and L2 caches on every round. That is an option, but even slower.

Dougall Johnson described a more elegant way in his really cool blog post on portable JavaScript Spectre exploitation. The idea follows directly from the pigeonhole principle. If you allocate far more data than the cache can hold, a randomly chosen cache line is almost certainly not cached. For a 256 KB L2 cache, allocating 64 MB leaves at most a 1/256 chance that a random cache line is still in L2. So instead of evicting a specific line, you never evict at all. You pick a fresh random location that is already evicted with overwhelming probability. The cool side effect of looping frequently over that array of objects is that this will lead to an auto-eviction effect. 

To leverage this in JavaScript, we allocate a large pool of attacker and victim object pairs that exceeds the last-level cache. Each measurement round selects a fresh random pair. The object's map pointer, the hidden-class descriptor that the speculative type check reads, is therefore almost certainly already evicted.

Co-locating the attacker and victim isolate

For the attack to work, both the attacker and victim isolate must be scheduled in the same process on the same edge server. One might intuitively think this would be difficult, considering Cloudflare operates tens of thousands of edge servers, but this is in fact quite trivial on Cloudflare Workers. Because Cloudflare Workers are designed to execute on any Cloudflare edge server, invoking the victim script from the attacker script with a fetch(“https://victim.example”) will in most cases cause the scheduler to spin up an instance of the victim worker in the exact same process. The victim isolate can be kept alive by repeatedly making subrequests to it at a certain interval.

What is more, because the attack stability is highly dependent on the CPU load of the edge server running the worker script, this allows an attacker to strategically run the attack in an off-peak colo (e.g. in an Australian colo during European business hours) where the traffic levels are comparatively low.

Defeating isolate resource limits

The Cloudflare Workers runtime enforces a set of limits on all isolates to protect the platform and prevent abuse. For the purposes of conducting this attack, the relevant limits were 30 seconds of CPU time and 1,000 subrequests per invocation. These limits have since been increased, but the following principles are still relevant.

For a regular Worker, each HTTP request, a fetch event, is a new invocation that resets these limits. The catch is landing sequential requests on the same edge server. Load balancing and shifting network conditions make that unreliable. Durable Objects solve it for us.

Durable Objects are built for real-time coordination between clients, so the runtime treats every incoming WebSocket message as an invocation that resets the CPU time and request limits. The attacker opens a persistent WebSocket to a Durable Object worker and sends regular keep-alive messages. This keeps a single isolate alive and gives us a persistent, bi-directional channel to run the attack over.

One quirk cost us some time. An isolate is single-threaded, so incoming WebSocket messages are only processed when the script hands control back to the event loop. During synchronous code the runtime never sees the keep-alive, so it never resets the CPU time. If the thread stays blocked for more than 30 seconds, the runtime kills the isolate. This puts an upper bound on how much we can amplify in a single synchronous burst. Yielding regularly between bursts lets us keep an isolate alive from five to more than 20 hours.

Putting everything together

The previous attack relied mostly on repetition to amplify a single cache access, and therefore, was slowly leaking 120 bit/h. We combined tree-based PLRU amplification with measurement loops. Each iteration re-creates the cache state and thereby adds more timing difference. If an interrupt destroys the cache state in one iteration, it doesn’t matter, since later iterations cancel it out. This made the signal strong enough to classify bits with a remote WebSocket timer. The overall idea is now to combine.

for (let s = 0; s < SAMPLE_NUM; s++) {
  timer.mark("mark S" + s);
  for (let r = 0; r < OUTER_REP_NUM; r++) {
    setup();                   // branch mistraining and cache control
    leak(secretBit);           // transient access
    PLRU(cacheSet, INNER_REP); // amplify
  }
  timer.mark("mark E" + s);
}

delta = fetchFromServer(SAMPLE_NUM);
return median(delta);

We demonstrated the full end-to-end attack in the Cloudflare Workers production environment, against Workers we controlled. We first leaked memory from the attacker Worker. From there, we leaked data from a co-located victim Worker where we had intentionally placed a secret.

First, we established co-location between an attacker Worker, a victim Worker we owned, and a remote timer. Durable Objects gave us a long-lived execution context. WebSocket messages gave us a repeatable timing source. The /cdn-cgi/trace endpoint helped us confirm machine placement by looking at the fl value.

Second, we added a calibration step to probe the timer with speculatively reachable values. This step matters because production machines are noisy. Per-invocation calibration lets us classify bits from the relative difference between the zero and one distribution. This last test should lead to two clearly separable distributions.