LZ-RDO 进入 ARM 的 astc-encoder
LZ-RDO lands in ARM's astc-encoder

原始链接: https://www.jonolick.com/home/lz-rdo-lands-in-arms-astc-encoder

ASTC LZ-RDO 现已纳入 ARM 的 astc-encoder(PR #781)。它通过让 ASTC 块更易压缩来缩小 LZ 压缩包,同时保持固定的 128 位块大小。RDO 有意以质量换取体积,偏好重复位的编码,并使用受限的通用配置、按活动加权的误差容限、移至前端历史、候选生成和简单的 LZ 成本模型。它支持 CLI 和 API,提供 lambda 调优(例如漫反射 20–40,法线约 10)、并行分块、多轮处理、进度和取消。它仅支持 LDR,不减小内存占用,并会增加编码时间。

黑客新闻 最新 | 往期 | 评论 | 提问 | 展示 | 招聘 | 提交 登录 LZ-RDO 进入 ARM 的 astc-encoder ( jonolick.com ) 7 分 由 dverlaeckt80 1 小时前 | 隐藏 | 往期 | 收藏 | 讨论 帮助 指南 | 常见问题 | 列表 | API | 安全 | 法律 | 申请 YC | 联系 搜索:
相关文章

原文
My rate-distortion optimization work for ASTC is now in ARM's astc-encoder (PR #781). Fabian "ryg" Giesen took my original implementation, added a round of optimizations and improvements, and landed it. If you ship ASTC textures inside Oodle, zlib, zstd, LZ4 or LZMA-compressed packages, your downloads can now get smaller with one command-line flag.

​The problem: ASTC looks like noise to LZ

​ASTC is a fixed-rate format. Every block is exactly 128 bits, whether it's 4x4 or 12x12. On the GPU that's what you want, but nobody ships raw textures. They go into packages that get compressed with a general-purpose LZ codec. To an LZ compressor, a normal ASTC bitstream looks almost random: each block's endpoints and weights are fitted to that block's pixels, so neighboring blocks rarely share bytes. That leaves the compressor nothing to match.

RDO fixes this by choosing, among encodings that look nearly as good, the ones that repeat bits the encoder has already emitted. The GPU sees a valid ASTC texture of exactly the same size, and LZ sees matches. It's a quality-for-package-size trade, and a single knob controls it.

​How it works

1. Encode normally, plus a "restricted" encoding

First the image goes through the regular astcenc encode. With LZ-RDO enabled, each block also gets one extra trial encode using the most common configuration: 1 partition, 1 weight plane, endpoints quantized to at most 32 levels, and only the always-available block modes for 2D. Blocks that share a configuration are much more likely to share bits, so this gives the optimizer more chances to find repeats.​

2. Measure where errors will be visible

Raw squared error (PSNR) is the wrong measure when you're adding error on purpose. Errors in smooth areas and slow gradients are easy to spot. Errors in noisy texture or near edges are mostly hidden.

So before encoding, the image is split into tiles (about 256×256, or 32³ for 3D) and each texel gets an activity weight:

  • Low-pass filter the image (Gaussian, σ = 2.2).
  • Take the squared difference from the original. That's the local high-frequency energy.
  • Blur that energy a little (σ = 1.25) so it spreads.
  • Map it to a weight that falls off with the square root of the energy.
Smooth texels keep tight error tolerances. Busy texels get looser ones.

3. ​Move-to-front histories for endpoints and weights

ASTC packs weights from the top of the block downward and the mode and endpoints from the bottom up. That makes splitting a block into "endpoint bits" and "weight bits" a simple bitmask. The optimizer keeps two move-to-front lists of recently used values:

  • Endpoint history: mode, header and endpoint bits.
  • Weight history: weight bits.
The -rdo-endpoints and -rdo-weights options set their sizes. Defaults range from 16/4 at -fastest up to 512/160 at -exhaustive.

4. Build candidates for each block

For every block, the encoder scores:

  • the original encoding;
  • the restricted encoding;
  • endpoint substitution: take endpoints from the history, compute the ideal weights for this block's pixels with those endpoints, and quantize. A quick check on endpoint-axis alignment skips pairs that can't work. The best few candidates are kept.
  • weight splicing: combine every weight entry in the history with each surviving endpoint candidate that has the same weight bit count. If both halves came from the same earlier block, it's one long LZ match.

5. ​Score with a simple LZ cost model

Each candidate's cost is cost = weighted_SSD + λ · estimated_bits
The bit estimate is deliberately simple:

Unmatched bits are billed as literals.

  • A match costs about 10 + log2(mtf_position + 32) bits.
  • If the endpoints and weights match the same earlier block, they're billed as one match.
  • The first two bytes (where the mode lives) are costed from an adaptive histogram, so frequently used modes get cheaper.
  • A partial byte at the endpoint/weight boundary is billed as literal bits.
The code comments say "don't over-tune this," and I mean it. The goal is to point the search in the right direction, not to simulate zstd.

λ is scaled by texel count, so the same value behaves about the same at 4x4 and at 12x12.

6. Multiple passes, in parallel

The image is processed in independent chunks of blocks, so it spreads across threads. Each pass seeds the histories with random blocks from the chunk, and passes alternate between forward and reverse order. Because splicing produces the same bit pattern many different ways, a small hash-table cache of decoded blocks avoids decoding them again during error checks.

Using It

Command line:

astcenc -cl albedo.png albedo.astc 6x6 -thorough -rdo-lambda 20

API:

astcenc_config_init(ASTCENC_PRF_LDR, 6, 6, 1, ASTCENC_PRE_THOROUGH,
                    ASTCENC_FLG_USE_LZ_RDO, &config);
config.lz_rdo_lambda = 20.0f;

Tuning guidance:

  • λ is tuned so that a step of about 10 is a visible difference.
  • Diffuse/albedo: 20–40 is usually fine.
  • Normal maps: stay low, around 10 or even below, depending on your quality bar.
  • -rdo-passes 2 (the -exhaustive default) squeezes out a bit more, at a real cost in encode time.
Progress callbacks and cancellation work across both stages: the base encode reports the first 33%, and the RDO passes report the rest.

​Limitations

  • LDR only. Nothing prevents HDR in principle.
  • In-memory size doesn't change. The gain only shows up after LZ compression on disk or over the wire.
  • It costs encode time. It's meant to be paired with the higher-quality presets.

Results

Thanks

​Huge thanks to Fabian for the optimizations, fixes and for getting this merged, to ARM for taking it upstream, and to Charles, Fabian and Sean for the Oodle Texture work this comes from.

联系我们 contact @ memedata.com