I replicated Ng's RYS method and found that duplicating 3 specific layers in Qwen2.5-32B boosts reasoning by 17% and duplicating layers 12-14 in Devstral-24B improves logical deduction from 0.22→0.76 on BBH — no training, no weight changes, just routing hidden states through the same circuit twice. Tools included. Two AMD GPUs, one evening.

Duplicate 3 layers. No training. Logical deduction goes from 0.22 → 0.76.

This toolkit finds and exploits "reasoning circuits" hidden inside transformer models. The idea: certain contiguous blocks of layers act as indivisible cognitive units. Duplicate them in the forward pass — same weights, no training, no merging — and the model gets measurably smarter on specific capabilities.

Built on David Ng's RYS method and extended with new findings. Everything here was discovered on two AMD consumer GPUs (RX 7900 XT + RX 6950 XT) in one evening.

Validated on standard benchmarks via lm-evaluation-harness at n=50:

Average improvement: +8% across all metrics. Nothing degraded.

Measured on custom probe suite (BBH-derived + EQ-Bench-style + GSM8K):

Transformers organize themselves during training into functional circuits — multi-layer processing units that perform complete cognitive operations. These circuits are indivisible: duplicating a single layer does almost nothing, but duplicating the right block of 3-4 layers gives the model a second pass through its reasoning pipeline.

Different models have different circuits in different places:

• Devstral-24B (40 layers): reasoning circuit at layers 12-14
• Qwen2.5-32B (64 layers): reasoning circuit at layers 7-9

The boundaries are sharp. Shift the block by one layer in either direction and the improvement disappears or inverts.

Different duplication patterns create distinct cognitive profiles from the same weights:

Same weights on disk. Same VRAM for the base model. Just different routing.

pip install gguf requests tqdm

python sweep.py \
    --model /path/to/model.gguf \
    --llama-server /path/to/llama-server \
    --tmpdir /dev/shm/rys \
    --results pass.jsonl \
    --block-sizes 3 4 5 \
    --stride 1 \
    --start-min 10 --start-max 20 \
    --skip-baseline \
    --port 8099 \
    --server-args --device Vulkan1,Vulkan2

# Duplicate layers 12-14 in Devstral (the result validated above)
python layer_path.py model.gguf improved.gguf \
    -p "0..14,12,13,14,15..39" -v

# Duplicate layers 7-9 in Qwen2.5-32B
python layer_path.py model.gguf improved.gguf \
    -p "0..9,7,8,9,10..63" -v

# Go wild: triple-pass, interleaved, skip layers, whatever you want
python layer_path.py model.gguf experiment.gguf \
    -p "0..16,13,14,15,16,13,14,15,16,17..39" -v

# Start the server with modified model
llama-server -m improved.gguf --port 8089 -ngl 99 --device Vulkan1,Vulkan2

# Run lm-evaluation-harness
lm_eval --model local-chat-completions \
    --model_args model=test,base_url=http://localhost:8089/v1/chat/completions,num_concurrent=1,max_retries=3,tokenized_requests=False \
    --tasks gsm8k_cot,ifeval,mbpp,bbh_cot_fewshot_logical_deduction_five_objects \
    --apply_chat_template --limit 50 \
    --output_path ./eval_results

# Compare runs
python compare_eval.py ./eval_base ./eval_improved

1. For each layer configuration (i, j):

   • GGUF surgery creates a model where layers i..j-1 are physically duplicated
   • The new forward pass: layers 0..j-1 → layers i..j-1 again → layers j..N-1
   • llama-server loads the modified model
   • Three probe suites run: math, EQ, reasoning (BBH-derived)
   • Scores are compared to baseline, results printed live
   • Server killed, modified GGUF deleted, next config

2. The search strategy:

   • Pass 1: Large blocks (8 layers), wide stride → find the hot zone
   • Pass 2: Small blocks (3-5 layers), stride 1 within hot zone → find exact boundaries
   • Pass 3: Try multi-pass, interleaved, and compound configs

Modified GGUFs are written to tmpfs (/dev/shm) and deleted after each test. The base model weights stay on disk.

• Linux with llama.cpp built (CPU, CUDA, Vulkan, or Metal)
• Python 3.10+ with gguf, requests, tqdm
• Enough VRAM/RAM to run the model + a few extra duplicated layers
• Optional: lm-eval for benchmark validation, matplotlib for heatmap plots

Does this use more VRAM? Yes, the duplicated layers are physical copies in the GGUF. For 3 extra layers on a 24B model, expect ~1.5 GiB additional. A llama.cpp forward-pass patch (using pointers instead of copies) would eliminate this — contributions welcome.

Does this slow down inference? Yes, proportionally to the number of extra layers. 3 extra layers on a 40-layer model = ~7.5% slower. The reasoning improvement is worth it.

Will this work on my model? Probably. We've tested on Mistral-architecture (Devstral) and Qwen2 architecture. Ng's original work was on Qwen2-72B. The circuits exist in all transformer models — the question is where they are and how big they are. Run the sweep and find out.

Why not fine-tune instead? This is orthogonal to fine-tuning. You can do both. In fact, Ng's RYS models were later fine-tuned by others and topped the HuggingFace leaderboard. Layer duplication changes the architecture; fine-tuning changes the weights. Stack them.

MIT