Hamming Codes
ECC RAM corrects errors that occur in memory before it gets to the CPU. The most common form of ECC is the Hamming Code [1] which when it has R redundant bits can correct single bit errors and detect double-bit errors in messages with 2^R-R-1 bits of data. For PC use that means if you want to protect 32bits of data you need R=6 and with 64bits you need R=7. The standard for DDR4 and similar RAM is 72 bits of data width on the bus and Hamming codes to correct single bit errors and detect double bit errors for 65bits of data. The computers we use have 64bits of data so that allows an extra bit that could be an extra parity, I don’t know what if anything is done with this extra bit.
RDIMM vs UDIMM
One point of confusion in such things is the difference between Registered memory AKA RDIMMs [2] and regular PC/laptop memory which is often referred to as UDIMMs. The “register” is just a buffer which due to complex issues that aren’t relevant to this post means that DIMMs can be larger and you can have more DIMMs in a system but latency may be slightly worse. It is technically quite possible to create RDIMMs without ECC (64bits wide instead of 72) but I have never seen a system that used such RAM.
I have used more than a few systems with ECC UDIMMs and I recommend avoiding them if convenient as ECC UDIMMs are expensive on the second hand market while ECC RDIMMs can get very cheap. There are servers with ECC RDIMMs that are very unsuitable for home use (such as dual-CPU 1RU servers which are very noisy) so once they are past the 5 year tax write-off period the server chassis gets sent to ewaste and the RAM goes on the second hand market, the glut of RAM without systems to use it forces the price down.
For the systems most commonly seen there are RDIMM systems with ECC and UDIMM systems without ECC.
Chipkill
If every bit in RAM was independent of every other bit then the basic Hamming code would solve most problems. However multiple bits in the same chip may be affected by the same problem, or one chip on the DIMM might entirely fail. With every RDIMM having 18 or 36 DRAM chips there are 2 or 4 bits per chip. On DIMMs with 36 DRAM chips one chip could fail and have the errors reliably detected with a Hamming code. On DIMMs with 18 DRAM chips one failed chip can’t necessarily be detected with Hamming codes. IBM trademarked the term ChipKill for ECC systems which can cope with a single DRAM chip failing [3]. This is referred to as “Advanced ECC” on Dell and HP servers which require an even number of DIMMs. If anyone knows what coding method is used for “ChipKill” type systems then please let me know.
Systems with advanced ECC also often have features like hot-spare for RAM and RAID-1 type functionality which is interesting but not something most people who read my blog will ever want to use.
DDR5
DDR5 has on-die ECC to deal with the increased error incidence from smaller and faster memory [4], this is specified as 8 bits of error correction per 128 bits of data which implies basic Hamming codes.
The on-die ECC is not a replacement for regular ECC, it’s a mitigation for new problems introduced. My experience of memory errors is that the majority of repeatable errors (where a system would get an error with Memtest86+ or an ECC error report repeatedly) were DIMM seating issues, I could unplug and reinsert the DIMM in question and then the same tests would pass. Those errors would not be affected by on-die ECC.
One thing that concerns me is the possibility of on-die ECC interacting with ECC on the motherboard and reducing it’s effectiveness. I haven’t been able to find out enough about how this works to determine if that’s the case. My concern is that an error of 3+ bits that’s corrected with a basic Hamming code might be more likely to create an error condition that “Advanced ECC” can’t fix than the original error.
Currently the best published research on the effectiveness of ECC on RAM errors is the Google paper published in 2009 which is based on DDR and DDR2 RAM [5]. So I don’t expect that we will see published research about even DDR4 ECC any time soon. I presume that Google and the other cloud providers are still doing such research and providing the information to DRAM vendors under NDA so we have to just hope that the DRAM vendors do what’s required to make things work correctly and allow us to buy products based on that research.
DDR5 EC4 vs EC8
DDR5 supports 2*32bit “subchannels” instead of just supporting 64bit words [6]. For DDR5 ECC RAM there are variants EC4 which has 36bits of data per subchannel and EC8 which has 40 bits. EC8 allows Hamming codes on each subchannel indepdendently. I haven’t found a reference on how exactly EC4 works, it could be reading 64bits at a time (not taking advantage of the subchannels) to use Hamming codes or it could have 1 parity bit for each subchannel and just assume that there’s no need to check Hamming codes unless the subchannel parity fails. EC8 allows full Hamming code checks on 32bits of data and presumably ChipKill on 64bits.
It’s widely claimed that all DDR5 RDIMMs are EC8 and all DDR5 ECC UDIMMs are EC4. A quick search on ebay turned up adverts for EC4 and EC8 RDIMMs and links to apparently reliable sites confirming that some of the RDIMMs are EC4. There are reports of EC8 UDIMMs even though I couldn’t find any advertised. This seems to mirror the situation with DDR4 where non-ECC RDIMMs are apparently available somewhere and ECC UDIMMs are something I’ve used a few times but most people have never seen.
I then searched for information on what servers support. The Dell R760 server supports both EC4 and EC8 RDIMMs but you can’t have both in the same system.
The existence of EC4 DIMMs is wrong. They shouldn’t make substandard gear, the manufacturing price difference between 72 and 80 bit wide DIMMs isn’t going to be great and the end result is some systems with inadequate specs and extra difficulty in upgrading systems with more things to check for compatibility.
Why ECC is Needed
Here’s an interesting article about Mozilla’s claim that 15% of Firefox crashes are due to RAM hardware errors [7], this seems to be based on repeatable errors and therefore won’t count errors where a bit flip happens once a day or less.
Some years ago I reported a BTRFS corruption issue on my desktop PC to the BTRFS developers and one of them stated that the corruption in question didn’t match any pattern expected from a BTRFS bug and recommended that I run Memtest86+. The memory test revealed that I was getting about one memory corruption per 5 hours so if I had used Firefox on that system any crashes probably wouldn’t have been regarded as hardware errors with RAM. Those errors caused filesystem corruption and some data loss, if I hadn’t been using BTRFS that could have gone unnoticed for years.
On another occasion I had a VM I was using for testing software I was developing that had some unexpected errors. After working on it for a day I had shared the errors with a mailing list of other developers who also spent some time investigating it. Eventually I began to suspect a hardware problem, I went on site and when I rebooted the system to run Memtest86+ it didn’t even boot as it had errors that stopped the BIOS from even working correctly. It was strange that the system was apparently working correctly and restarting the KVM VM resulted in the same errors happening in the same code and nothing else on the VM apparently having a problem. It turned out that the system had a motherboard problem that made all but one of the DIMM sockets unusable so I ended up sending it to e-waste. That wasted a day of my time and some hours of other people’s time. Presumably on other occasions developer time is wasted due to hardware errors and no-one even realises.
What Society Needs
We need ECC RAM to be more widely used. Ideally we would have some government action to force this given the ongoing cost to society in corrupted data and lost time due to RAM hardware errors. I think that at minimum we need sufficient taxes on non-ECC RAM (and EC4 RAM for DDR5) to make it more expensive when bought new than ECC RAM.
We need to have greater knowledge of the benefits of ECC RAM among computer experts, people need to recommend that computers be purchased with ECC RAM whenever possible and that systems which can’t have ECC RAM (laptops and phones) shouldn’t be used for storing important data.
We need to avoid silly things like having so many variants of RAM to confuse people and make it needlessly difficult to get ECC RAM working.