Tinybox- 离线AI设备 120B参数

Tinybox- 离线AI设备 120B参数
Tinybox – A powerful computer for deep learning

## tinygrad & tinybox 摘要 tinygrad 是一个快速发展的、简单但强大的神经网络框架，专注于速度和效率。它将复杂的网络分解为三种核心操作类型——逐元素运算、规约和移动——并通过自定义内核编译、延迟张量求值和简化的后端实现性能。目前处于 alpha 阶段，tinygrad 旨在超越 PyTorch 在标准基准测试中的速度。 tinygrad 背后的团队 tiny corp 正在获得资金并招聘软件工程师（对 tinygrad 的贡献备受重视）。他们还为贡献提供赏金。除了框架之外，tiny corp 还销售“tinybox”电脑——专为深度学习设计的高性能机器。目前有三种配置（red v2 售价 12,000 美元，green v2 售价 65,000 美元，以及 2027 年上市的 exa-scale 模型，售价约 1000 万美元），tinybox 拥有令人印象深刻的规格，包括高达 25,920 GB 的 GPU 内存和 1244 TB/s 的 GPU 内存带宽。它们的基准测试结果优于价格更高的系统，并且在付款后会快速发货。 tinygrad 已经在 openpilot 等项目中得到应用，tiny corp 的最终目标是通过让所有人都能访问 petaflop 级别的计算，来实现 AI 的民主化。

原文

tinygrad: A simple and powerful neural network framework

tinygrad

We write and maintain tinygrad, the fastest growing neural network framework

It's extremely simple, and breaks down the most complex networks into 3 OpTypes

ElementwiseOps are UnaryOps, BinaryOps, and TernaryOps.
They operate on 1-3 tensors and run elementwise.
example: SQRT, LOG2, ADD, MUL, WHERE, etc...

ReduceOps operate on one tensor and return a smaller tensor.
example: SUM, MAX

MovementOps are virtual ops that operate on one tensor and move the data around
Copy-free with ShapeTracker.
example: RESHAPE, PERMUTE, EXPAND, etc...

But how...where are your CONVs and MATMULs? Read the code to solve this mystery.

Work at tiny corp

We are now funded and hiring full time software engineers. Very talented interns okay.

See our bounty page to judge if you might be a good fit. Bounties pay you while judging that fit.

We are also hiring for operations and hardware, but if you haven't contributed to tinygrad your application won't be considered.

tinybox (now shipping)

We sell a computer called the tinybox. It comes in red, green, and soon, exa.

	red v2	green v2 blackwell	exabox
FP16 (FP32 acc) FLOPS	778 TFLOPS	3086 TFLOPS	~1 EXAFLOP
GPU Model	4x 9070XT	4x RTX PRO 6000 Blackwell	720x RDNA5 AT0 XL
GPU RAM	64 GB	384 GB	25,920 GB
GPU RAM bandwidth	2560 GB/s	7168 GB/s	1244 TB/s
GPU link bandwidth	full fabric PCIe 4.0 x16	full fabric PCIe 5.0 x16	full fabric 400 GbE
CPU	32 core AMD EPYC	32 core AMD GENOA	120x 32 core AMD GENOA
System RAM	128 GB	192 GB	23,040 GB
System RAM bandwidth	204.8 GB/s	460.8 GB/s	55.2 TB/s
Disk size	2 TB fast NVMe	4 TB raid + 1 TB boot	480 TB raid
Disk read bandwidth	7.3 GB/s	59.3 GB/s	7.1 TB/s
Networking	2x 1GbE + OCP3.0	2x 10GbE + OCP3.0 PCIe5	3.2 TB/s scale out
Noise	< 50 dB, 31 low speed fans		65 db @ 10 meters
Power Supply	one 1600W, 100V~240V	2x 1600W, 100V~240V	600 kW, 200V~240V
BMC	AST2500	AST2600	custom
Operating System	Ubuntu 24.04
Dimensions	12U, 16.25" deep, 60-90 lbs		20x8x8.5 ft, 20,000 lbs
Rack?	Freestanding or rack mount		concrete slab
Driver Quality	Good	Great	functions as single GPU
SHIPPING	IN STOCK - $12,000	IN STOCK - $65,000	coming 2027 - ~$10M

for updates on products and inventory, sign up for the mailing list

FAQ

What is a tinybox?: It is a very powerful computer for deep learning, and likely the best performance/$. It was benchmarked in MLPerf Training 4.0 vs computers that cost 10x as much. And of course, anything that can train can do inference.
How do I get a tinybox?: Place an order through the links above. The factory is up and running, and it will ship within one week of us receiving the payment. Currently offering pickup in San Diego + shipping worldwide.
Where can I learn more about the tinybox?: We have a lot of content on our Twitter, we also have a tinybox docs page and a #tinybox discord channel.
Can I customize my tinybox?: In order to keep prices low and quality high, we don't offer any customization to the box or ordering process. Of course, after you buy the tinybox, it's yours and you are welcome to do whatever you want with it!
Can you fill out this supplier onboarding form?: In order to keep prices low and quality high, we don't offer any customization to the box or ordering process. If you aren't capable of ordering through the website, I'm sorry but we won't be able to help.
Can I pay with something besides wire transfer?: In order to keep prices low and quality high, we don't offer any customization to the box or ordering process. Wire transfer is the only accepted form of payment.
Can I get a W-9 for this purchase?: Yes, you can download that here.
Is tinygrad used anywhere?: tinygrad is used in openpilot to run the driving model on the Snapdragon 845 GPU. It replaces SNPE, is faster, supports loading onnx files, supports training, and allows for attention (SNPE only allows fixed weights).
Is tinygrad inference only?: No! It supports full forward and backward passes with autodiff. This is implemented at a level of abstraction higher than the accelerator specific code, so a tinygrad port gets you this for free.
How can I use tinygrad for my next ML project?: Follow the installation instructions on the tinygrad repo. It has a similar API to PyTorch, yet simpler and more refined. Less stable though while tinygrad is in alpha, so be warned, though it's been fairly stable for a while.
When will tinygrad leave alpha?: When we can reproduce a common set of papers on 1 NVIDIA GPU 2x faster than PyTorch. We also want the speed to be good on the M1. ETA, Q2 next year.
How is tinygrad faster than PyTorch?: For most use cases it isn't yet, but it will be. It has three advantages:
It compiles a custom kernel for every operation, allowing extreme shape specialization.

All tensors are lazy, so it can aggressively fuse operations.

The backend is 10x+ simpler, meaning optimizing one kernel makes everything fast.
Where is tinygrad development happening?: On GitHub and on Discord
How can the tiny corp work for me?: Email me, [email protected]. We are looking for contracts and sponsorships to improve various aspects of tinygrad.
How can I work for the tiny corp?: See hiring above. Contributions to tinygrad on GitHub always welcome, and a good way to get hired.
Can I invest in the tiny corp?: Invest with your PRs.
What's the goal of the tiny corp?: To accelerate. We will commoditize the petaflop and enable AI for everyone.