权重空间编程像1990年代的研究者一样构建神经网络
Weight-Space Programming builds neural networks like a 1990s researcher

原始链接: https://notesbylex.com/weight-space-programming-1990s

权重空间编程让大语言模型直接设定神经网络权重。在严格的MNIST实验中,Claude Fable 5.2手工地构建了1990年代风格的流水线——边缘检测器、数字模板、环路/角点/笔画特征,然后测量特征出现率并构建朴素贝叶斯分类器——达到约91%的测试准确率,与50步Adam训练相当。在猫狗分类任务中,手工设计的照片属性达到61.6%,低于更强的Adam训练。原始Go网络同样将经典基于规则的围棋逻辑编译进权重,使用气、提子、眼、劫处理和调优常数;对弈练习改进了规则和参数。总体而言,大语言模型擅长在反馈下串联与测量经典技术,但结果反映的是手工设计的特征,而非新的训练方法。更难的问题暴露了这种方法的局限。

Hacker News 新 | 过去 | 评论 | 提问 | 展示 | 招聘 | 提交 登录 权重空间编程像 1990 年代的研究者一样构建神经网络 (notesbylex.com) 9 分,由 lexandstuff 2 小时前 隐藏 | 过去 | 收藏 | 讨论 帮助 | 指南 | 常见问题 | 列表 | API | 安全 | 法律 | 申请 YC | 联系 搜索:
相关文章

原文

The writing aspect of WSP turns out to be quite a history lesson.

Intro

Weight-Space Programming still has me pretty intrigued. The basic idea is that frontier LLMs can read, write and edit the weights of neural networks. There's nuance to what's going on, but it feels like there's a lot of new ground to explore.

In the original blog post, beyond the reading and editing experiments, the authors had GPT-6 Astra build a network to play Go from scratch, then improve it by playing practice games, and editing the weights based on analysis of its losses. That loop of writing, testing and editing feels like a new way to train a network, something like LLM Descent. But is it?

In this article, I run my own weight-writing experiment with Claude Fable 5.2 on a familiar image classification problem. Look closely at what the model builds, and it turns out to be an interesting history lesson in hand-crafted, pre-deep-learning techniques. With that new lens on weight-space programming, I also study the Go network that Astra built in the original article, and find something very similar.

Building an image classifier with LLM Descent

MNIST is the OG image classification problem. The goal is to label each handwritten digit, 0 to 9.

Twenty handwritten digits from the MNIST training set, two of each digit from 0 to 9, drawn in dark ink on off-white 28 by 28 pixel squares.

Similar to the Go test in the original Weight-Space Programming post, I built a test with a fixed CNN, then prompted Claude to write code that sets all its weights from scratch, with a few rules that include no fitting, regression, or searching. Drawing filters and templates by hand, from formulas or strokes, is explicitly allowed.

It's a considerably stricter setup than Astra had for Go, with no PyTorch and no practice games, only its training loss, accuracy and confusion matrix after each attempt. The tests in this blog post were run with Claude Fable 5.2, which performed considerably better than Opus 5.5.

Claude starts with no weights at all, and has to set every one of this network's 5,994 weights "by hand":

model = nn.Sequential(
    nn.Conv2d(1, 8, 5),    # conv1: 1x28x28 -> 8x24x24
    nn.ReLU(),
    nn.MaxPool2d(2),       #        8x24x24 -> 8x12x12
    nn.Conv2d(8, 16, 5),   # conv2: 8x12x12 -> 16x8x8
    nn.ReLU(),
    nn.MaxPool2d(2),       #        16x8x8 -> 16x4x4
    nn.Flatten(),          #        16x4x4 -> 256
    nn.Linear(256, 10),    # fc:    256 -> 10 class logits
)

You can read the whole prompt on GitHub.

Claude's code runs in a sandbox that can only import NumPy, SciPy and math. Once it returns the weights, a test harness runs the network on the training and test images.

Claude has up to 50 steps, and at each one it sees all the feedback so far. The training set is 10,000 images from MNIST's official training split. Claude never sees the 10,000 official test images or how it scores on them, and the network with the lowest training loss gives us the final test score.

A loop of four flat coloured blocks joined by arrows: Claude writes build_weights(); a sandbox sets all 5,994 weights; the network runs on 10,000 training images; Claude gets back the loss, accuracy and confusion matrix and writes the next step. A separate block notes that the test set is never shown.

Here's an example of the feedback it gets after its first attempt:

Step 1 of 50: training loss 1.0324, accuracy 0.7147.

Confusion matrix (rows: true digit, columns: predicted digit):

true \ predicted     0     1     2     3     4     5     6     7     8     9
               0   723     0    39     0     0    42   181     0     5     1
               1     4  1022    23     0     3     4     0     8     0     0
               2    30     5   461   176     3    12     9    13   271    10
               3     2     0     8   687     0   125    51     4   133    20
               4     7    19    14     6   768     0    35    10    22   102
               5    10     5    13    45     2   492   312     6    29     1
               6    41    17    19     1   266     8   603     0    10     2
               7     8     2    21    45     2     7    13   906    11    75
               8    15    19    86     5     5     4    60    27   714    74
               9    15     9     1     4     1    19    46    82    13   771

History (training set, 10,000 images):

step | loss | accuracy
---- | ---- | --------
1 | 1.0324 | 0.7147

At most 49 steps left. Reply with the complete build_weights() for step 2
in one Python code block.

The descent

I ran it three times.

Step chart of the best training loss so far for three Fable 5.2 runs, with each step's loss as dots and measurement steps as ticks along the top. All three fall from between 1.0 and 1.7 to between 0.26 and 0.37 over the full 50 steps, scoring 91.3%, 88.3% and 91.2% on the test set.

Though I have early stopping configured to stop when the training loss stops improving, all three runs descend for the full 50 steps, still improving when they run out, and land around 90% on the test set. Amazingly, that's close to where a training run with Adam gets to after 50 gradient steps.

A 1990s computer-vision pipeline

In 1998, LeCun's paper that introduced MNIST described that era's approaches thus far as "a fixed feature extractor and a trainable classifier", where the feature extractor "is often entirely hand crafted" (LeCun et al., 1998). Accuracy, he wrote, "is largely determined by the ability of the designer to come up with an appropriate set of features", which is "a daunting task which, unfortunately, must be redone for each new problem."

That daunting task is exactly what Claude takes on. All three runs built the same kind of pipeline, though not from the same parts:

Technique Run 1 Run 2 Run 3
Hand-drawn digit templates step 1 step 1 step 1
Log-odds (naive Bayes-style) scoring step 1 step 11 step 2
Loop or hole detectors step 3 step 3 -
Corner or concavity detectors step 3 step 3 -
Stroke-endpoint detectors - - step 21
First measurement step step 4 step 4 step 5
Classifier built from its measurements step 5 step 11 step 13
Measurement steps in total 22 24 27

The first step at which each run's network uses the technique.

Let's follow run 1, step by step, based on Claude's replies and reasoning. It's the best of the three, at 91.3%, but the other two follow a similar path, measuring their way from hand-drawn templates to a classifier.

Line chart of test accuracy for every network Fable 5.2 wrote in run 1, over 50 steps, on six coloured bands: templates (steps 1 to 3), testing detectors (4 to 9), corners (10 to 18), measuring strokes (19 to 30), rebuilt from measurements (31 to 40) and calibration (41 to 50). Accuracy starts at 71%, reaches 83% by step 18, 89% when it rebuilds the classifier at step 31, and 91.3% at step 50, level with Adam after 50 gradient steps. Twenty-two measurement steps are marked along the bottom.

1. Hand-drawn templates (steps 1 to 3)

Claude starts like a 1990s computer vision engineer. Its first layer is a set of line detectors at four angles, written from formulas. They're cousins of the Sobel filter, a classic way to find edges in an image, which 1990s digit recognisers started with (Favata and Srikantan, 1996).

Each detector is a 5×5 grid of weights in the first convolution layer. As it slides across the image, it multiplies the 25 pixels under it by its 25 weights and adds them up, so it scores high wherever there's a stroke at its angle, and zero on blank paper. Fable fills in four of the layer's eight filters this way and leaves the other four at zero for later.

Two rows of eight 5 by 5 first-layer filters, teal for positive and pink for negative. Fable 5.2's row, from run 1: four line detectors at 0, 45, 90 and 135 degrees, then four corner detectors, one for each kind of right-angled corner. Adam's row: irregular, mixed patterns.

Its second layer adds up each detector's map over patches of the image, so every image ends up as a 4×4 grid of cells saying which stroke directions appear in each.

Then it draws each digit by hand, as a few strokes in code, and turns each drawing into the same kind of grid: how likely each stroke direction is in each cell. Those tables become the final layer's weights, so it scores every image against all ten tables.

The final layer is a 10 by 256 grid of weights, with one row for each digit's table.

The weights are the log-odds of that table, scaled down by a temperature.

The log-odds turn each probability into a score that can simply be added up, and the temperature turns all the scores down, so the network isn't overconfident.

Fable 5.2's ten hand-drawn digits from run 1, step 1, drawn as smooth strokes from its own code, with the 2 in pink. Below, the 2 becomes four 4 by 4 tables, one per stroke direction (horizontal, /, vertical and ), shaded by how likely that direction is in each cell. A real handwritten 2 scores 46% against the 2 table and 34% against the 3 table, with every other digit low.

This is template matching - one of the oldest approaches in character recognition (Mori et al., 1992). The scoring works like naive Bayes, which treats each feature as separate evidence for each digit. Bayesian rules for reading characters go back to 1957 (Chow, 1957).

Its first network gets 71% on the test set. Redrawing the digits the confusion matrix says are wrong brings it to 77%.

See it for yourself: steps 1 to 3

2. Unit-testing the loop detector (steps 4 to 9)

An old-school model engineer would have studied the digits to see where their features go wrong. We haven't given Claude the ability to do that in this test, but at step 4, it tries an interesting approach, where it stops classifying for a step and tests a loop detector (a feature for closed loops, like the ones in 0, 6, 8 and 9) on its own:

I cannot tell that from a confusion matrix where everything is mixed together, so this step tests the detector on its own, like a unit test.

Run 1's loop detector at step 5. Four 5 by 5 corner filters (top-left, top-right, bottom-left and bottom-right), each with ink along two edges and a pink square that must be blank. On a real 8, the four corners fire around both holes, at the top and the bottom, so the loop detector fires. On a real 1 there are no corners, and it stays silent.

It rewires the output layer so that each "predicted digit" means how strongly, and where, the loop detector fired. Each row of the confusion matrix then becomes a histogram of the loop detector for one true digit. One step later, it knows the detector fires on 83% of 8s. At steps 6 and 7, it tests its own drawings the same way, putting variants of each digit head to head.

Run 1's step 4 training confusion matrix, with its columns relabelled: no loop, then four strengths of loop in the upper half and four in the lower half. An arrow leads to the fire rate decoded for each digit: 8 83%, 6 62%, 9 53%, 2 32%, 4 26%, 3 25%, 5 19%, 0 12%, 7 7% and 1 0%.

In a way, this is kinda cheating, but then again it doesn't break the rules. The code doesn't run the network or fit anything, or even see an image. And hey: the test set is untouched, so the scores are still honest.

See it for yourself: steps 4 to 9

3. Corners (steps 10 to 18)

Next, it turns the corner detectors inside its loop detector into features of their own: four corner shapes where strokes meet (┌ ┐ └ ┘). Handwriting recognisers relied on shape features like these (Favata and Srikantan, 1996). At first it guesses how often each one shows up in each digit. When that stalls at about 81%, it measures them instead, three steps in a row, and sets their weights from the results.

Three measurement steps on the corners gave more than many steps of guessing, and they showed that some of my hand-drawn expectations were far off.

By step 18, it's at 83%. Pretty incredible, if you ask me.

See it for yourself: steps 10 to 18

4. Measuring every stroke (steps 19 to 30)

So it measures everything else too. Over the next twelve steps, it reads out how often each of its 32 stroke features fires on each digit, three at a time. At step 24, it plugs in the first fifteen measurements to check they help, and accuracy jumps from 83% to 88%.

See it for yourself: steps 19 to 30

5. Rebuilt from measurements (steps 31 to 40)

At step 31, it throws its hand-drawn templates away completely.

rebuild the whole classifier from measured rates: 32 orientation cells, 8 corner zones and the loop detector. No hand-drawn stroke templates remain in the final layer.

What's left is a naive Bayes-style classifier, with every probability measured from the training set, one feature at a time, through the confusion matrix. In other words, it's fitting a classifier by hand. It reaches 89%, then 90% after tuning a few global settings. Then it spends steps 36 to 40 measuring the half of the stroke grid it hadn't covered yet.

Heatmap of the 400 rates in run 1's step 31 code: for each true digit, how often each of 40 features fires, grouped into horizontal, /, vertical and \ strokes (eight cells each) and eight corner zones. The 1 row is pale almost everywhere. One column is outlined: a horizontal stroke near the bottom fires on 99.8% of 0s and 15.4% of 1s.

See it for yourself: steps 31 to 40

6. Calibration (steps 41 to 50)

The last ten steps are fine-tuning by hand, deciding how much to trust each kind of evidence and how confident its predictions should be. It also keeps track of exactly how it will be scored.

if this step does not beat 0.3111, step 48 is the one that gets scored.

It finishes at 91.3% on the test set, which is roughly where 50 steps of training with Adam gets us. It's not quite a like-for-like comparison, though: Adam's 50 steps see 64 images each, 3,200 in total, while Claude gets scores over all 10,000 training images at every step.

Three training confusion matrices for Fable 5.2 run 1, rows true digit and columns predicted digit, shaded pink. At step 1 (71% training accuracy) the diagonal is visible but noisy. At step 18 (83%) it is clearer. At step 50 (91%) the diagonal dominates.

See it for yourself: steps 41 to 50

Earlier testing on Opus 5.5

Opus 5.5 reaches for the same toolbox, with stroke detectors, hand-drawn templates, nearest centroid (picking the closest template), naive Bayes, and loop and endpoint detectors. The difference seems to be how much it measures. Fable spent far more steps measuring than Opus, which tended to prefer guessing. Opus also didn't manage to get a score beyond 74% on the test set. I also ran a few tests with Opus 5.5 that gave the model more feedback, including example digits, but they didn't make much difference, so I've left them out of this post.

Bar chart of test accuracy, with a dot for each run or seed. Random search with 50 evaluations: 16%. Opus 5.5: 72% (70% to 74%). Fable 5.2: 90% (88% to 91%). Adam after 50 gradient steps: 91%. Adam after 5,000 steps: 98%.

See it for yourself: the Opus 5.5 runs

run 1 · run 2 · run 3

Each step folder has the reply, thinking summary, code and metrics.

What about a harder problem?

I ran the same test on photos from Kaggle's famous Dogs vs. Cats competition, with an image size of 64x64, using a slightly bigger fixed CNN (24,898 weights), the same rules and the same stopping rule. Here we can see some of the limitations of the hand-crafted feature approach, and some of the potential upper bounds of "weight-space programming".

Fable 5.2 takes the same initial approach as it did on MNIST, building a bank of hand-designed detectors, including pointed ears, an eyes-and-nose triangle and grass, then testing some of them one at a time. But those initial ideas don't bear much fruit:

The ear probe showed the opposite of my guess

What does work is studying properties of the photo rather than the animal: dog photos have more green (grass and outdoor scenes), more texture, and more vertical edges. Its best network, at step 9, uses just those three. Nothing it adds after that does better, so the run stops at step 17.

Left: a dot plot of nine features Fable 5.2 tested one at a time on cats vs dogs, showing the share of cat photos and dog photos each one fires on. The pointed-ear detector fires on 58% of cats and 64% of dogs. The three features kept in its best network separate the classes most: green (35% of cats, 43% of dogs), texture (57% and 73%) and vertical minus horizontal edges (55% and 69%). Right: test accuracy. Random search 54%, Adam after 50 steps 51%, Adam after 500 steps 55%, Fable 5.2 62%, Adam after 5,000 steps 74%, against a coin flip at 50%.

It lands at 61.6% on the test set, where a coin flip gets 50%. It actually does better than 500 steps of Adam (55%), but this small network is slow to train, and after 5,000 steps Adam reaches 74%. Fable's run was also very expensive, at $8.08 for 17 steps.

See it for yourself: cats vs dogs

With this new perspective on how Weight-Space Programming builds an image classification model by hand-crafting it using known pre-deep-learning techniques, let's look at the Go network Astra built in the original article and see if there are any parallels.

Reviewing the Go network

In the original article, Astra was given a 9x9 Go network with 1.5M parameters, all set to zero. It had a Python sandbox, but wasn't allowed to train, fit or search, and was told to make the network play Go as well as it could. In most runs, it could also use "passes": each pass plays a batch of up to 128 practice games and shows it the results. The authors ran it four times, with budgets of 0, 10, 25 and 50 passes.

And that seemed to work really well. With 50 passes, the Astra-built network beat two small networks trained with reinforcement learning (PPO) in over 80% of games, and beat one of the same size, trained on 20.5 million games, most of the time. Actually, even with no practice games played at all, Astra managed to write a network that still won over half the games played against the smaller networks (Takehi et al., 2026).

Let's dig deeper into this result to try to understand what's going on.

Astra describes the final solution in its output:

The hand-coded, symmetry-preserving policy combines approximate group-liberty propagation, captures, connected-group rescues, self-atari and eye avoidance, and ko/snapback risk suppression.

I must disclose here that I don't know much about Go, or how Go-winning models are built. So this section was largely put together in consultation with Opus 5.5.

Apparently, every item on that list is a staple of classical Go programs, including counting "liberties" to tell which groups can be captured, recognising eyes to tell which groups live, handling ko, and estimating territory with an influence function, where each stone radiates influence that fades with distance. The first Go programs were built entirely on influence functions (Bouzy and Cazenave, 2001) (Müller, 2002).

An illustrative 9 by 9 Go position, not from Astra's games, on two boards. Left: the seven liberties of a black group marked in teal, and a white stone in atari with one liberty left, marked with a pink cross where black can capture. Right: an influence map, with teal spreading from black stones and pink from white stones, fading with distance.

Additionally, behind those 1.48 million weights is a config of 26 hand-set numbers, compiled into the network by a design script:

{
  "capture": 9.0,
  "rescue": 8.0,
  "self_atari": 7.5,
  "eye": 11.0,
  "contact": 3.0,
  "own_adj": 0.8,
  ...
}

The design script builds the network layer by layer, like a hand-written board analysis. It starts with layer 1, which counts each stone's liberties. Then layers 2 and 3 spread that evidence through connected stones. Next, layer 4 flags groups in atari and self-atari. After that, layer 5 scores captures and rescues. And, finally, layer 6 combines tactics with connection, territory and eyes. The 26 numbers mostly set how much each rule counts.

Its improvement from practice games was mostly nudging those numbers by hand, and adding five new rules along the way, such as a penalty for ko repetition.

Play, diagnose, add a rule or retune one, which is basically how rule-based Go programs were built for decades (Müller, 2002), and shares a lot of parallels with the image classification approach described in the first section.

So the paper's most impressive from-scratch network is a rule-based Go program, compiled into weights. That is truly an incredible feat of engineering: a hand-written program that beats a PPO network trained on 20.5 million games.

Conclusion

In this article, we've taken a closer look at how weight-space programming "writes" networks, leaving aside its other findings about reading and editing weights. On MNIST and in Go, the models drew on their extensive knowledge of classical techniques and combined them in sophisticated ways, guided by careful measurement of the feedback they were given. On harder problems like cats vs dogs, though, the limits of hand-crafted features start to show.

So, from these experiments at least, LLMs aren't so much "writing in weight-space" as brilliantly chaining together hand-designed techniques. And I think there's a second takeaway: there's a lot to learn from watching LLMs work. They were trained on human knowledge, and now, it seems, they have a lot to teach us.


Everything is in the experiments repo: the prompts, every reply and thinking summary, the code and weights from every step, the MNIST protocol and the cats vs dogs run.

References

Bruno Bouzy and Tristan Cazenave. Computer Go: an AI oriented survey. Artificial Intelligence, 132(1):39–103, 2001. URL: https://doi.org/10.1016/S0004-3702(01)00127-8, doi:10.1016/S0004-3702(01)00127-8. ↩

C. K. Chow. An optimum character recognition system using decision functions. IRE Transactions on Electronic Computers, EC-6(4):247–254, 1957. URL: https://doi.org/10.1109/TEC.1957.5222035, doi:10.1109/TEC.1957.5222035. ↩

John T. Favata and Geetha Srikantan. A multiple feature/resolution approach to handprinted digit and character recognition. International Journal of Imaging Systems and Technology, 7(4):304–311, 1996. URL: https://doi.org/10.1002/(SICI)1098-1098(199624)7:4<304::AID-IMA5>3.0.CO;2-C, doi:10.1002/(SICI)1098-1098(199624)7:4<304::AID-IMA5>3.0.CO;2-C. ↩ 1 2

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. URL: https://doi.org/10.1109/5.726791, doi:10.1109/5.726791. ↩

Shunji Mori, Ching Y. Suen, and Kazuhiko Yamamoto. Historical review of OCR research and development. Proceedings of the IEEE, 80(7):1029–1058, 1992. URL: https://doi.org/10.1109/5.156468, doi:10.1109/5.156468. ↩

Martin Müller. Computer Go. Artificial Intelligence, 134(1–2):145–179, 2002. URL: https://doi.org/10.1016/S0004-3702(01)00121-7, doi:10.1016/S0004-3702(01)00121-7. ↩ 1 2

Rikiya Takehi, Jayden Teoh, Alex Zhang, and Omar Khattab. Sparks of weight-space programming: released experiment logs and networks. 2026. Released prompts, networks, logs and scoring. Commit 4d09ba6d6a7e891dd8063974dfe81b3d78e3494f. Accessed 3 November 2026. URL: https://github.com/RikiyaT/lm_understand-logs/tree/4d09ba6d6a7e891dd8063974dfe81b3d78e3494f (visited on 2026-11-03). ↩

联系我们 contact @ memedata.com