As the eleventh hour dawns all the pieces start to fall into place. I lived my life knowing this would happen, yet when it is I may be just as unprepared as anyone else. As any self driving car maker knows, predicting doesn’t mean you can act.
comma almost has an exaflop. Just one little exaflop. We dream bigger.
A gigawatt of power, a million GPUs, 1000 exaflops, a zettaflop. 1e21 FLOPS. 1e27 training runs are now. 100 lifetimes in 1e6 seconds – 2 weeks on my zettaflop machine.
But they were experiencing what no human had ever known before, a sensory bandwidth thousands of times normal. For seconds that seemed without end, their minds were filled with a jumble verging on pain, data that was not information and information that was not knowledge. To hear ten million simultaneous phone conversations, to see the continent’s entire video output, should have been a white noise. Instead it was a tidal wave of detail rammed through the tiny aperture of their minds.
– Vernor Vinge - True Names
I want to feel it. I want to command that kind of power. The same way I command my little teraflop laptop. The same way I talk to petaflop claude. Get it all to think for me.
Just an exaflop would feel amazing. 1000 Claudes. And I already have one of these, I just don’t have the software finished yet to command it all together. We’re working on it.
But a zettaflop. One million Claudes. To be able to search every book in history, solve math problems, write novels, read every comment, watch every reel, iterate over and over on a piece of code until it’s perfect – spend a human year in 10 minutes. 50,000 people working for you, all aligned with you, all answering as one.
The biggest bottleneck is power. The dtype stuff probably stops at FP4, currently B200s are getting 10 TFLOPS/W. That’s still 100 MW, that’s going to be hard to get my hands on. I think it will get 10x better, so I need 10 MW.
Solar yields 394 MWh/acre-year, or 45 kW per acre. 250 acres of solar panels all feeding my computer, a hill to pump water up for energy storage. $100 for 100W of real output.
100,000 chips with 10 PFLOPS each, get that down to $100 per chip.
- $10M for the machine.
- $10M for the solar panels.
- $10M for the land and construction.
I’ll own this before I die.