0 - https://github.com/ttkb-oss/dedup

A few notes:

* By default it doesn't scan everything. It ignores all files but those in an allow list. The way the allow list is structured, it seems like Hyperspace needs to understand the content of a file. As an end user, I have no idea what the difference between a Text file and a Source Code file would be or how Hyperspace would know. Hyperscan only found 360MB to dedup. Allowing all files increased that to 842MB.

* It doesn't scan files smaller than 100 KB by default. Disabling the size limit along with allowing all files increased that to 1.1GB

* With all files and no size limit it scanned 67,309 of 68,874 files. `dedup` scans 67,426.

* It says 29,522 files are eligible. Eligible means they can be deduped. `dedup` only fines 29,447. There are 76 already deduped files, which is an off-by-one, so I'm not sure what the difference is.

* Scanning files in Hyperspace took around 50s vs `dedup` at 14s

* It seems to scan the file system, then do a duplication calculation, then do the deduplication. I'm not sure why the first shouldn't be done together. I chose to queue any filesystem metadata as it was scanned and in parallel start calculating duplicates. The vast majority of the time files can be mismatched by size, which is available from `fts_read` "for free" while traversing the directory.

* Hyperspace found 1.1GB to save, `dedup` finds 1.04GB and 882MB already saved (from previous deduping)

* I'm not going to buy Hyperspace at this time, so I don't know how long it takes to dedup or if it preserves metadata or deals with strange files. `dedup` took 31s to scan and deduplicate.

* After deduping with `dedup`, Hyperscan thinks there are still 2 files that can be deduped.

* Hyperspace seems to understand it can't dedup files with multiple hard links, empty files, and some of the other things `dedup` also checks for.

* I can't test ACLs or any other attribute preservation like that without paying. `strings` suggests those are handled. HFS Compression is a tricky edge case, but I haven't tested how Hyperspace's scan deals with those.

With a FOSS project this would have been expected, but with a ShareWare-style model? Idk..

Again, I've got no problem with people selling software or closed source models, but I've never understood using this justification. Maybe in this instance he's a well known public figure with published contact info that people will abuse?

Who says you have to deal with support requests if you open source something?

> All his apps are personal itches he scratched and he sells them not to make a profit but to make the barrier of entry high enough to make user feedback manageable.

That makes no sense

Almost anyone who has ever maintained popular open-source software, even if dealing with them means putting up a notice that says "Don't ask support questions" and having to delete angrily posted issues.

My understanding from listening to his explanation is he wants to be able to support users and have an income stream to incentivize that.

As an open-source maintainer of a popular piece of software, I'm very empathetic.

Just keeping everything closed is really missing the point of how trust in infra that handles critical data is built nowadays.

Apps like this can easily bit rot, and more users does often mean more work e.g. answering or filtering emails, finding more edge cases, etc.

From his perspective that means having a income to dedicate time to this. I don't think he's interested in being an "infra" app as you would think of it.

As someone who maintains critical open-source software, I can strongly empathize, even if it’s not an approach I would take.

Making software proprietary and for-pay, especially such a small tool, doesn't just significantly reduce the number of eyeballs this testing is crowdsourced to, but it also disincentivises issue reporting .. why should I spend the time to for free report sth to somebody who is making money off my testing and doesn't even bother to be transparent about how things work exactly (i.e. the source)?

If you really care about the quality of your work then maximizing the eye ball count and incentivise high qualith issue reporting.

Though if you want to maximize income instead, you keep it closed and ask for a subscription.

Quite obvious which option he chose.

Pretty strange thing to say in the context of a closed source app.

App store prices are localized. If the blog post said it costs “$10” or whatever, that doesn’t mean anything to millions of potential customers who live where they don’t use $, and is confusing for millions more that do use $ but don’t know if the price is in their local $ or USD

I ran it over my Postgres development directories that have almost identical files. It saved me about 1.7GB.

The project doesn't have any license associated with it. If you don't mind, can you please license this project with a license of your choice.

As a gesture of thanks, I have attempted to improve the installation step slightly and have created this pull request: https://github.com/ttkb-oss/dedup/pull/6

I see it is "pre-release" and sort of low GH stars (== usage?), so I'm curious about the stability since this type of tool is relatively scary if buggy.

Whenever using it on something sensitive that I can’t back up first for whatever reason, I make checksum files and compare them afterwards. I’ve done this many times on hundreds of GB and haven’t seen corruption. Caveat emptor.

There is one huge caveat I should add to the README - block corruption happens. Having a second copy of a file is a crude form of backup. Cloning causes all instances to use the same block, so if that one instance is corrupted, all clones are. That’s fine for software projects with generated files that can be rebuilt or checked out again, but introduces some risk for files that may not otherwise be replaceable. I keep multiple backups of all that stuff in hardware other than where I’m deduping, so I dedup with abandon.

I’m a nobody with no audience. Maybe some attention here will get some users.

This got me wondering why the filesystem itself doesn't run a similar kind of deduplication process in the background. Presumably, it is at a level of abstraction where it could safely manage these concerns. What could be the downsides of having this happen automatically within APFS?

It might also be a little unintuitive that modifying one byte of a large file would result in a lot disk activity, as the file system would need to duplicate the file again.

[0] https://www.truenas.com/docs/references/zfsdeduplication/

Of course, that is true of most filesystems.

I believe as soon as you change a single bite you get a complete copy that’s your own.

And that’s how this program works. It finds perfect duplicates and then effectively deletes and replaces them with a copy of the existing file so in the background there’s only one copy of the bits on the disk.

As far as I understand, it works like a reflink feature in the modern linux FSs. If so, thats really cool, and thats also a bit better than the zfs's snapshots. Iam newbie on macos, but it looks amazing

However APFS gives you a number of space related foot-guns if you want. You can overcommit partitions, for example.

It also means if you have 30 GB of files on disk that could take up anywhere from a few hundred K to 30 GB of actual data depending on how many dupes you have.

It’s a crazy world, but it provides some nice features.

If we only have two files, A and its duplicate B with some changes as a diff, this works pretty well. Even if the user deletes A, the OS could just apply the diff to the file on disk, unlink A, and assign B to that file.

But if we have A and two different diffs B1 and B2, then try to delete A, it gets a little murkier. Either you do the above process and recalculate the diff for B2 to make it a diff of B1; or you keep the original A floating around on disk, not linked to any file.

Similarly, if you try to modify A, you'd need to recalculate the diffs for all the duplicates. Alternatively, you could do version tracking and have the duplicate's diffs be on a specific version of A. Then every file would have a chain of diffs stretching back to the original content of the file. Complex but could be useful.

It's certainly an interesting concept but might be more trouble than it's worth.

ZFS is essentially an object store database at one layer; the checksum-hash deduplication table is an object like any other (file, metadata, bookmarks, ...). There is one deduplication table per pool, shared among all its datasets/volumes.

On reads, one does not have to consult the dedup table.

The mechanism was fairly easy to add. And for highly-deduplicatable data that is streaming-write-once-into-quiescent-pool-and-never-modify-or-delete-what's-written-into-a-deduplicated-dataset-or-volume, it was a reasonable mechanism.

In other applications, the deduplication table would tend to grow and spread out, requiring extra seeks for practically every new write into a deduplicated dataset or volume, even if it's just to increment or decrement the refcount for a record.

Destroying a deduplicated dataset has to decrement all its refcounts (and remove entries from the table where it's the only reference), and if your table cannot all fit in ram, the additional IOPS onto spinning media hurt, often very badly. People experimenting with deduplication and who wanted to back out after running into performance issues for typical workloads sometimes determined it was much MUCH faster to destroy the entire pool and restore from backups, rather than wait for a "zfs destroy" on a set of deduplicated snapshots/datasets/volumes to complete.

Doing deduplication at this level is nice because you can dedupe across file systems. If you have, say, a thousand systems that all have the same OS files you can save vats of storage. Many times, the only differences will be system specific configurations like host keys and hostnames. No single filesystem could recognize this commonality.

This fails when the deduplication causes you to have fewer replicas of files with intense usage. To take the previous example, if you boot all thousand machines at the same time, you will have a prodigious I/O load on the kernel images.

There were about 1000 copies of the same pre-requisite .NET and VC++ runtimes (each build had one) and we only paid for the cost of storing it once. It was great.

It is worth pointing out though, that on Windows Server this deduplication is a background process; When new duplicate files are created, they genuinely are duplicates and take up extra space, but once in a while the background process comes along and "reclaims" them, much like the Hyperspace app here does.

Because of this (the background sweep process is expensive), it doesn't run all the time and you have to tell it which directories to scan.

If you want "real" de-duplication, where a duplicate file will never get written in the first place, then you need something like ZFS

ZFS is great if you believe you'll exceed some threshold of space while writing. I don't personally plan my volumes with that in mind but rather make sure I have some amount of excess free space.

WinSvr allows you to disable dedupe if you want (don't know why you would) where as ZFS is a one-way street without exporting the data.

Both have pros and cons. I can live with the WinSvr cons while ZFS cons (memory) would be outside of my budget, or would have been at the particular time with the particular system.

You probably don’t want your phone or watch de-duping stuff.

There are tons of knobs you can tweak in macOS but Apple has always been pretty conservative when it comes to what should be default behavior for the vast majority of their users.

Certainly when you duplicate a file using the Finder or use cp -c at the command line, the Copy-on-Write functionality is being used; most users don’t need to know that.

I've been writing a similar thing to dedupe my photo collection and I'm so paranoid of pulling the trigger I just keep writing more tests.

I’ve heard many horror stories of dedupe related corruption or restoration woes though, especially after a ransomware attack.

Any halfway-competent developer can write some code that does a SHA256 hash of all your files and uses the Apple filesystem API's to replace duplicates with shared-clones. I know swift, I could probably do it in an hour or two. Should you trust my bodgy quick script? Heck no.

The author - John Siracusa - has been a professional programmer for decades and is an exceedingly meticulous kind of person. I've been listening to the ATP podcast where they've talked about it, and the app has undergone an absolute ton of testing. Look at the guardrails on the FAQ page https://hypercritical.co/hyperspace/ for an example of some of the extra steps the app takes to keep things safe. Plus you can review all the proposed file changes before you touch anything.

You're not paying for the functionality, but rather the care and safety that goes around it. Personally, I would trust this app over just about any other on the mac.

You are involved. You see the list of duplicates and can review them as carefully as you'd like before hitting the button to write the changes.

> There is no way for Hyperspace to cooperate with all other applications and macOS itself to coordinate a “safe” time for those files to be replaced, nor is there a way for Hyperspace for forcibly take exclusive control of those files.

Maybe not solving the same problem.

After all how many perfect duplicate files do you probably create a month accidentally?

There’s a subscription or buy forever option for people who think that would actually be quite useful to them. But for a ton of people a one time IAP that gives them a limited amount of time to use the program really does make a lot of sense.

And you can always rerun it for free to see if you have enough stuff worth paying for again.

For example, I discovered my time machine backup kicked out the oldest versions of files I didn't know it had a record of and thought I'd long since lost, but it destroyed the names of the directories and obfuscated the contents somewhat. Thousands of numerically named directories, some of which have files I may want to hang onto, but don't know whether I already have them or not, or where they are since it's completely unstructured. Likewise, many of them may just have one of the same build system text file I can obvs toss away.

I wish it were more obvious how to do it with other software. Often there's a learning curve in the way before you can see the value.

however has anyone been able to find out from the website how much the license actually costs?

For each candidate file, you need some "key" that you can use to check if another candidate file is the same. There can be millions of files so the key needs to be small and quick to generate, but at the same time we don't want any false positives.

The obvious answer today is a SHA256 hash of the file's contents; It's very fast, not too large (32 bytes) and the odds of a false positive/collision are low enough that the world will end before you ever encounter one. SHA256 is the de-facto standard for this kind of thing and I'd be very surprised if he'd done anything else.

At that point maybe it’s better to just compare byte by byte? You’ll have to read the whole file to generate the hash and if you just compare the bytes there is no chance of hash collision no matter how small.

Plus if you find a difference in bytes 1290 you can just stop there instead of reading the whole thing to finish the hash.

I don’t think John has said exactly how on ATP (his podcast with Marco and Casey), but knowing him as a longtime listener/reader he’s being very careful. And I think he’s said that on the podcast too.

This is the default for ZFS deduplication and git does something similar with size and far weaker SHA-1. I would add a test for SHA-256 collisions, but no one has seemed to find a working example yet.

0 - https://github.com/ttkb-oss/dedup

Wonder what the distribution is here, on average? I know certain file types tend to cluster in specific ranges.

>maybe it’s better to just compare byte by byte? You’ll have to read the whole file to generate the hash

Definitely, for comparing any two files. But, if you're searching for duplicates across the entire disk, then you're theoretically checking each file multiple times, and each file is checked against multiple times. So, hashing them on first pass could conceivably be more efficient.

>if you just compare the bytes there is no chance of hash collision

You could then compare hashes and, only in the exceedingly rare case of a collision, do a byte-by-byte comparison to rule out false positives.

But, if your first optimization (the file size comparison) really does dramatically reduce the search space, then you'd also dramatically cut down on the number of re-comparisons, meaning you may be better off not hashing after all.

You could probably run the file size check, then based on how many comparisons you'll have to do for each matched set, decide whether hashing or byte-by-byte is optimal.

To have a mere one in a billion chance of getting a SHA-256 collision, you'd need to spend 160 million times more energy than the total annual energy production on our planet (and that's assuming our best bitcoin mining efficiency, actual file hashing needs way more energy).

The probability of a collision is so astronomically small, that if your computer ever observed a SHA-256 collision, it would certainly be due to a CPU or RAM failure (bit flips are within range of probabilities that actually happen).

You can group all files into buckets, and as soon as a bucket is empty, discard it. If in the end there are still files in the same bucket, they are duplicates.

Initially all files are in the same bucket.

You now iterate over differentiators which given two files tell you whether they are maybe equal or definitely not equal. They become more and more costly but also more and more exact. You run the differentiator on all files in a bucket to split the bucket into finer equivalence classes.

For example:

* Differentiator 1 is the file size. It's really cheap, you only look at metadata, not the file contents.

* Differentiator 2 can be a hash over the first file block. Slower since you need to open every file, but still blazingly fast and O(1) in file size.

* Differentiator 3 can be a hash over the whole file. O(N) in file size but so precise that if you use a cryptographic hash then you're very unlikely to have false positives still.

* Differentiator 4 can compare files bit for bit. Whether that is really needed depends on how much you trust collision resistance of your chosen hash function. Don't discard this though. Git got bitten by this.

- compute SHA256 hashes for each file on the source side

- copy files which are not already known to a "canonical copies" folder on the destination (this step uses the hash itself as the file name, which makes it easy to check if I had a copy from the same file earlier)

- mirror the source directory structure to the destination

- create hardlinks in the destination directory structure for each source file; these should use the original file name but point to the canonical copy.

Then I got too scared to actually use it :)

In any case, a good design is to ask the kernel to do the dedupe step after user space has found duplicates. The kernel can double-check for you that they are really identical before doing the dedupe. This is available on Linux as the ioctl BTRFS_IOC_FILE_EXTENT_SAME.

Of course, engineering being what it is, it's possible that only one of these has hardware support and thus might end up actually being faster in realtime.

Also, use the length of the file for a fast check.

If we took the first 1024 bytes of each file as the lookup key, then our key size would be 1024 bytes. If you have 1 million files on your disk, then that's 128MB of ram just to store all the keys. That's not a big deal these days, but it's also annoying if you have a bunch of files that all start with the same 1024 bytes -- e.g. perhaps all the photoshop documents start with the same header. You'd need a 2-stage comparison, where you first match the key (1024 bytes) and then do a full comparison to see if it really matches.

Far more efficient - and less work - If you just use a SHA256 of the file's contents. That gets you a much smaller 32 byte key, and you don't need to bother with 2-stage comparisons.

I don't think it would be far more efficient to do hash the entire contents though. If you have a million files storing a terabyte of data, the 2 stage comparison would read at most 1GB (1 million * 1KB) of data, and less for smaller files. If you do a comparison of the whole hashed contents, you have to read the entire 1TB. There are a hundred confounding variables, for sure. I don't think you could confidently estimate which would be more efficient without a lot of experimenting.

... okay, so as long as you always feed chunks of data into your hash in the same deterministic order, it doesn't matter for the sake of correctness what that order is or even if you process some bytes multiple times. You could hash the first 1kB, then the second-through-last disk blocks, then the entire first disk block again (double-hashing the first 1kB) and it would still tell you whether two files are identical.

If you're reading from an SSD and seek times don't matter, it's in fact probable that on average a lot of files are going to differ near the start and end (file formats with a header and/or footer) more than in the middle, so maybe a good strategy is to use the first 32k and the last 32k, and then if they're still identical, continue with the middle blocks.

In memory, per-file, you can keep something like

  - the length
  - h(block[0:4])
  - h(block[0:4] | block[-5:])
  - h(block[0:4] | block[-5:] | block[4:32])
  - h(block[0:4] | block[-5:] | block[4:128])
  - ...
  - h(block[0:4] | block[-5:] | block[4:])

A further refinement would be to have per-file-format hashing strategies... but then hashes wouldn't be comparable between different formats, so if you had 1M pngs, 1M zips, and 1M png-but-also-zip quine files, it gets weird. Probably not worth it to go down this road.

You misunderstood the article, as it's basically doing the opposite of what you said.

This tool finds duplicate data that is specifically not duplicated via copy-on-write, and then turns it into a copy-on-write copy.

libcopyfile also supports cloning via two flags: COPYFILE_CLONE and COPYFILE_CLONE_FORCE. The former clones if supported (same volume and filesystem supports it) and falls back to actual copy if not. The force variant fails if cloning isn't supported.

> Finally, at WWDC 2017, Apple announced Apple File System (APFS) for macOS (after secretly test-converting everyone’s iPhones to APFS and then reverting them back to HFS+ as part of an earlier iOS 10.x update in one of the most audacious technological gambits in history).

How can you revert a FS change like that if it goes south? You'd certainly exercise the code well but also it seems like you wouldn't be able to back out of it if something was wrong.

https://asciiwwdc.com/2017/sessions/715

Let’s say for simplification we have three metadata regions that report all the entirety of what the file system might be tracking, things like file names, time stamps, where the blocks actually live on disk, and that we also have two regions labeled file data, and if you recall during the conversion process the goal is to only replace the metadata and not touch the file data.

We want that to stay exactly where it is as if nothing had happened to it.

So the first thing that we’re going to do is identify exactly where the metadata is, and as we’re walking through it we’ll start writing it into the free space of the HFS+ volume.

And what this gives us is crash protection and the ability to recover in the event that conversion doesn’t actually succeed.

Now the metadata is identified.

We’ll then start to write it out to disk, and at this point, if we were doing a dry-run conversion, we’d end here.

If we’re completing the process, we will write the new superblock on top of the old one, and now we have an APFS volume.

Run the real thing, throw away the results, report all problems back to the mothership so you have a high chance of catching them all even on their multi-hundred million device fleet.

I modify A_0. Does this modify A_1 as well or just kind of reify the new state of A_0 while leaving A_1 untouched?

But if you have the same 500MB of node_modules in each of your dozen projects, this might actually durably save some space.

I'm not sure if this is what you intended, but just to be sure: writing changes to a cloned file doesn't immediately duplicate the entire file again in order to write those changes — they're actually written out-of-line, and the identical blocks are only stored once. From [the docs](^1) posted in a sibling comment:

> Modifications to the data are written elsewhere, and both files continue to share the unmodified blocks. You can use this behavior, for example, to reduce storage space required for document revisions and copies. The figure below shows a file named “My file” and its copy “My file copy” that have two blocks in common and one block that varies between them. On file systems like HFS Plus, they’d each need three on-disk blocks, but on an Apple File System volume, the two common blocks are shared.

[^1]: https://developer.apple.com/documentation/foundation/file_sy...

So APFS supports it, but there is no way to control what an app is going to do, and after it’s done it, no way to know what APFS has done.

I don't understand what makes you think there's a significant risk of corruption. Are you talking about the risk of something modifying a file while the dedupe is happening? Or do you think there's risk associated with just having deduplicated files on disk?

I then tried again including my user home folder (731K files, 127K folders, 2755 eligible files) to hopefully catch more savings and I only ended up at 1.3GB of savings (300MB more than just what was in the NodeJS folders.)

I tried to scan System and Library but it refused to do so because of permission issues.

I think the fact that I use pnpm for my package manager has made my disk space usage already pretty near optimal.

Oh well. Neat idea. But the current price is too high to justify this. Also I would want it as a background process that runs once a month or something.

True

> and dedupes automatically

Also true.

But the way you put them after each other, makes it sound like npm does de-duplication, and since pnpm tries to be a drop-in replacement for npm, so does pnpm.

So for clarification: npm doesn't do de-duplication across all your projects, and that in particular was of the more useful features that pnpm brought to the ecosystem when it first arrived.

In general, you don’t want to mess with that.

When I ran it on my home folder with 165GB of data it only found 1.3GB of savings. This isn't that significant to me and it isn't really worth paying for.

BTW I highly recommend the free "disk-inventory-x" utility for MacOS space management.

If you read the rest of the comment he only saved another 30% running his entire user home directory through it.

So this is not a linear trend based on space used.

When I run it on my home folder (Roughly 500GB of data) I find 124 MB of duplicated files.

At this stage I'd like it to tell me what those files are - The dupes are probably dumb ones that I can simply go delete by hand, but I can understand why he'd want people to pay up first, as by simply telling me what the dupes are he's proved the app's value :-)

You misunderstood my comment. I ran it on my home folder which contains 165GB of data and it found 1.3GB is savings. That isn't significant for me to care about because I currently have 225GB free of my 512GB drive.

BTW I highly recommend the free "disk-inventory-x" utility for MacOS space management.

You wrote: but it only found 1GB of savings on a 8.1GB folder.

It’s quite a saving and that’s what everyone understood from your comment.

His comment is pretty understandable if you've done frontend work in javascript.

Node_modules is so ripe for duplicate content that some tools explicitly call out that they're disk efficient (It's literally in the tagline for PNPM "Fast, disk space efficient package manager": https://github.com/pnpm/pnpm)

So he got ok results (~13% savings) on possibly the best target content available in a user's home directory.

Then he got results so bad it's utterly not worth doing on the rest (0.10% - not 10%, literally 1/10 of a single percent).

---

Deduplication isn't super simple, isn't always obviously better, and can require other system resources in unexpected ways (ex - lots of CPU and RAM). It's a cool tech to fiddle with on a NAS, and I'm generally a fan of modern CoW filesystems (incl APFS).

But I want to be really clear - this is people picking spare change out of the couch style savings. Penny wise, pound foolish. The only people who are likely to actually save anything buying this app probably already know it, and have a large set of real options available. Everyone else is falling into the "download more ram" trap.

For reference, from the comment they’re talking about:

> I then tried again including my user home folder (731K files, 127K folders, 2755 eligible files) to hopefully catch more savings and I only ended up at 1.3GB of savings (300MB more than just what was in the NodeJS folders.)

Ex - On my non-apple machines, 8GB is trivial. I load them up with the astoundingly cheap NVMe drives in the multiple terabyte range (2TB for ~$100, 4TB for ~$250) and I have a cheap NAS.

So that "big win" is roughly 40 cents of hardware costs on the direct laptop hardware. Hardly worth the time and effort involved, even if the risk is zero (and I don't trust it to be zero).

If it's just "storage" and I don't need it fast (the perfect case for this type of optimization) I throw it on my NAS where it's cheaper still... Ex - it's not 40 cents saved, it's ~10.

---

At least for me, 8GB is no longer much of a win. It's a rounding error on the last LLM model I downloaded.

And I'd suggest that basically anyone who has the ability to not buy extortionately priced drives soldered onto a mainboard is not really winning much here either.

I picked up a quarter off the ground on my walk last night. That's a bigger win.

You do realize that this software is only available on macOS, and only works because of Apple's APFS filesystem? You're essentially complaining that medicine is only a win for people who are sick.

It’s a free app because you don’t have to buy it to run it. It will tell you how much space it can save you for free. So you don’t have to waste $20 to find out it only would’ve been 2kb.

But that means the parts you actually have to buy are in app purchases, which are always hidden on the store pages.

Here’s a question though: how does this work with transparently compressed files on APFS?

In my past experience, using reflinks is fine and using transparent compression is fine, but combining them leads to hard-to-debug file corruption.

The idea is not new, of course, and I've written one of these (for Linux, with hardlinks) years ago but in the end just deleted all the duplicate files in my mp3 collection and didn't touch the rest of the files on the disk, because not a lot of size was reclaimed.

I wonder for whom this really saves a lot of space. (I saw someone mentioning node_modules, had to chuckle there).

But today I learned about this APFS feature, nice.

My main motivation was for the packages of Python virtual envs, where I often have similar packages installed, and even if versions are different, many files would still match. Some of the packages are quite huge, e.g. Numpy, PyTorch, TensorFlow, etc. I got quite some disk space savings from this.

https://github.com/albertz/system-tools/blob/master/bin/merg...

Each worktree I usually work on is several gigs of (mostly) identical files.

Unfortunately the source files are often deep in a compressed git pack file, so you can't de-duplicate that.

(Of course, the bigger problem is the build artefacts on each branch, which are like 12G per debug/release per product, but they often diverge for boring reasons.)

There is also ".git/objects/info/alternates", accessed via "--shared"/"--reference" option of "git clone", that allows only sharing of object storage and not branches etc... but it is has caveats, and I've only used it in some special circumstances.

Am I missing something, or isn't it a "file de-duplicator" with a nice UI/UX? Sounds pretty simple to describe, and tells you why it's useful with just two words.

In times where documentation is often an afterthought, and technical details get hidden away from users all the time ("Ooops some error occurred") this should be celebrated.

> Q: Are clone files the same thing as symbolic links or hard links?

> A: No. Symbolic links ("symlinks") and hard links are ways to make two entries in the file system that share the same data. This might sound like the same thing as the space-saving clones used by Hyperspace, but there’s one important difference. With symlinks and hard links, a change to one of the files affects all the files.

> The space-saving clones made by Hyperspace are different. Changes to one clone file do not affect other files. Cloned files should look and behave exactly the same as they did before they were converted into clones.

[1] https://hypercritical.co/hyperspace/

[0] https://hypercritical.co/hyperspace/#how-it-works

Once you open one of the reference handles and modify the contents, the copy-on-write process is invoked by the filesystem, and the underlying data is copied into a new, separate file with your new changes, breaking the link.

Comparing with a hardlink, there is no copy-on-write, so any changes made to the contents when editing the file opened from one reference would also show up if you open the other hardlinks to the same file contents.

With APFS Clones, the contents start off identical, but can be changed independently. If you change a small part of a file, those block(s) will need to be created, but the existing blocks will continue to be shared with the clone.

I know that internally it isn't actually "removing" anything, and that it uses fancy new technology from Apple. But in order to explain the project to strangers, I think my tagline gets the point across pretty well.

The duplicates aren't removed, though. Nothing changes from the POV of users or software that use those files, and you can continue to make changes to them independently.

That’s why Copy-on-write clones are completely different than hardlinks.

Symbolic links, hard links, ref links are all part of the file system interface, not the implementation.

Also, I get a data security itch having a random piece of software from the internet scan every file on an HD, particularly on a work machine where some lawyers might care about what's reading your hard drive. It would be nice if it was open source, so you could see what it's doing.

> It would be nice if it was open source

> I get a data security itch having a random piece of software from the internet scan every file on an HD

With the source it would be easy for others to create freebie versions, with or without respecting license restrictions or security.

I am not arguing anything, except pondering how software economics and security issues are full of unresolved holes, and the world isn't getting default fairer or safer.

--

The app was a great idea, indeed. I am now surprised Apple doesn't automatically reclaim storage like this. Kudos to the author.

Edit: this might not work with the payment option actually. I don't think you can IAP without the internet.

> Q: Does Hyperspace preserve file metadata during reclamation?

> A: When Hyperspace replaces a file with a space-saving clone, it attempts to preserve all metadata associated with that file. This includes the creation date, modification date, permissions, ownership, Finder labels, Finder comments, whether or not the file name extension is visible, and even resource forks. If the attempt to preserve any of these piece of metadata fails, then the file is not replaced.

When you asked about the filesystem, I assumed you were asking about which filesystem feature was being used, since hyperspace itself is not provided by the filesystem.

Someone else mentioned[0] fclones, which can do this task of finding and replacing duplicates with reflinks on more than just macOS, if you were looking for a userspace tool.

[0]: https://news.ycombinator.com/item?id=43173713

You only get CoW on APFS if you copy a file with certain APIs or tools.

If you have a program that does it manually, you copied a duplicate to somewhere on your desk from some other source, or your files already existed on the file system when you converted to APFS because you’ve been carrying them for a long time then you’d have duplicates.

APFS doesn’t look for duplicates at any point. It just keeps track of those that it knows are duplicates because of copy operations.

No word about alternate data streams. I'll pass for now.. Although it's nice to see how much duplicates you have

Q: Does Hyperspace preserve file metadata during reclamation?

A: When Hyperspace replaces a file with a space-saving clone, it attempts to preserve all metadata associated with that file. This includes the creation date, modification date, permissions, ownership, Finder labels, Finder comments, whether or not the file name extension is visible, and even resource forks. If the attempt to preserve any of these piece of metadata fails, then the file is not replaced.

If you find some piece of file metadata that is not preserved, please let us know.

Q: How does Hyperspace handle resource forks?

A: Hyperspace considers the contents of a file’s resource fork to be part of the file’s data. Two files are considered identical only if their data and resource forks are identical to each other.

When a file is replaced by a space-saving clone during reclamation, its resource fork is preserved.

He points out its dangerous but could be worth it cause space savings.

I wonder if the implementation is using a hash only or does an additional step to actually compare the contents to avoid hash collision issues.

It's not open source, so we'll never know. He chose a pay model instead.

Also, some files might not be identical but have identical blocks. Something that could be explored too. Other filesystems have that either in their tooling or do it online or both.

    sudo tmutil listlocalsnapshots /
    sudo tmutil deletelocalsnapshots 

I probably wouldn't have risked it, either.

Back in the MS-DOS days, when the RAM was sparse, there was a class of so-called "memory optimization" programs. They all inevitably found at least few KB to be reclaimed through their magic even if the same optimizer was run back to back with itself and allowed to "optimize" things. That is, on each run they always find extra memory to be freed. They ultimately did nothing but claim they did the work. Must've sold pretty well nonetheless.

QEMM worked by remapping stuff into extended memory - in a time that most software wasn't interested in using it. It worked as advertised.

Quarterdeck made good stuff all around. Desq and DesqView/X were amazing multitaskers. Way snappier than Windows and ran on little to nothing.

> If some eligible files were found, the amount of disk space that can be reclaimed is shown next to the “Potential Savings” label. To proceed any further, you will have to make a purchase. Once the app’s full functionality is unlocked, a “Review Files” button will become available after a successful scan. This will open the Review Window.

I half remember this being discussed on ATP; the logic being that if you have the list of files, you will just go and de-dupe them yourself.

If you can do that, you can check for duplicates yourself anyway. It's not like there aren't already dozens of great apps that dedupe.

But then I ran this command and saved over 20GB:

  brew install fclones
  cd ~
  fclones group . | fclones dedupe

Edit: Apparently both is possible in the end: https://hypercritical.co/hyperspace/#purchase

I worry about that with Procreate. It feels like it's priced too low to be sustainable.

This app though? No chance. Parent comment says “if you want to support the app’s development” but not all apps need to be “developed” continuously, least of all system utilities.

And because it has no Internet access yet (and because I prompted it to use a workaround like this in that circumstance), the first thing it asked me to do (after hallucinating the functionality first, and then catching itself) was run `curl https://hypercritical.co/hyperspace/ | sed 's/<[^>]*>//g' | grep -v "^$" | clip`

("clip" is a bash function I wrote to pipe things onto the clipboard or spit them back out in a cross-platform linux/mac way)

    clip() {
      if command -v pbcopy > /dev/null; then
        [ -t 0 ] && pbpaste || pbcopy;
      else
        if command -v xclip > /dev/null; then
          [ -t 0 ] && xclip -o -selection clipboard || xclip -selection clipboard;
        else
          echo "clip function error: Neither pbcopy/pbpaste nor xclip are available." >&2;
          return 1;
        fi;
      fi
    }

When doing something with any risk potential I first ask the model for potential risks with the output, and then I manually read the code.

I also "recreated" this tool with Sonnet 3.7. The initial bash script worked (but was slow), and after a few iterations we landed on an fclones one-liner. I hadn't heard of fclones before, but works great! Saved a bunch of disk space today.

Since it's the kind of thing you will likely only need every couple of years, $10 each time feels fair.

If putting all your data online or into an SSD makes more sense, then this app isn't for you and that's okay too.

TL;DR - $49 for a lifetime subscription, or $19/year or $9/month.

It could definitely be easier to find.

The author talked about being very conservative on launch; skipping directories like the Photo library or others apps that actively manage data or looking across user directories. He stumbled into writing this app because he noticed the duplicated data of shared Photo libraries between different users on the same machine. That use case isn't even supported in this version. He said he plans future development to safely dedup more data--making a one time purchase less sustainable for them.

  rmlint -c sh:handler=reflink .

[1]: https://github.com/sahib/rmlint

macOS 15 was released in September 2024, this feels far too soon to deprecate older versions.

The problem is SwiftUI. It's very new, still barely usable on the Mac, but they are adding lots of new features every macOS release.

If you want to support older versions of macOS you can't use the nice stuff they just released. Eg. pointerStyle() is a brand new macOS 15 API that is very useful.

They are working on it, and making it better every year. I've started using it for small projects and it's pretty neat how fast you can work with it -- but not everything can be done yet.

Since they are still adding pretty basic stuff every year, it really hurts if you target older versions. AppKit is so mature that for most people it doesn't matter if you can't use new features introduced in the last 3 years. For SwiftUI it still makes a big difference.

I used to be a hardcore Apple/Mac guy, but I'm kind of giving up on the ecosystem. Even the dev tools are keeping everyone on the treadmill.

That being said, it's not quite an apples to apples comparison, because SwiftUI or UIKit can work with basically an infinite number of rows, whereas HTML will eventually get to a point where it won't load.

I'd argue there's a lot more to iced than just being a quick toolkit. the Elm Architecture really shines for GUI apps

If we are to believe ChatGPT itself: "The ChatGPT macOS desktop app is built using Electron, which means it is primarily written in JavaScript, HTML, and CSS"

For me on the German store it looks like this:

    Unlock for One Year 22,99 €
    Unlock for One Month 9,99 €
    Lifetime Unlock 59,99 €

Of course the Copy-on-Write clone functionality has been available since APFS became the default file system in 2019.

There's value in convenience. I wouldn't pay for a yearly license (that price seems more than fair for a "version lifetime" price to me?) but seeing as this tool will probably need constant maintenance as Apple tweaks and changes APFS over time, combined with the mandatory Apple taxes for publishing software like this, it's not too awful.

Which really means up until the dev gets bored, which can be as short as 18 months.

I wouldn't mind something like this versioned to OS. 20$ for the current OS, and ten dollars for every significant update.

That's why we see so many more subscription-based apps these days, application development is an ongoing process with ongoing costs, so it needs to have ongoing income. But the traditional buy-it-once app pricing doesn't enable that long-term development and support. The app store supports subscriptions though, so now we get way more subscription-based apps.

I really think Siracusa came up with a clever pricing scheme here, given his want to use the app store for distribution.

Every time I need to purchase something via Apple, it becomes a quest. Enter password, validate card, welcome to endless login loop. Reboot. Click purchase, enter password, confirm OTP on another device, then nothing happens, purchase button is active, clicks ignored. Reboot. Click "Get", program begins downloading, wait 30s, app cannot be download, go to Settings to verify account. Sure. Account is perfectly fine in Settings. Reboot. Click "Get". Finally program installed. Click in-app purchase. Enter password again. Choose Apple Pay. Proceed with purchase. You need to verify your account. Account is fine in Settings. Reboot. Click purchase. Cannot be completed at this time. Wait couple of hours, try again. Purchase successful.

All. The. Time. For years. On almost all of the devices which I upgrade annually.

I'm going to have to clone the drive, then use terminal to chmod the /usr dirs to extract the files I want (mostly personal music production).

I immediately ordered a Mac mini, but since I didn't want a 256GB drive and 16GB RAM, I'm still waiting for it to arrive from China.

Also, the M1 MB Pro was the most expensive* and worst computer I have ever owned. I wish I had just bought an air. No tactile volume controls. As a musician that is the worst.

(*company I worked for a while in school paid $12k for a Mac 2 FX. Lol.)

Again, I don't think you're doing anything wrong, and I don't doubt your experience. But I really think something's fundamentally wrong somewhere, because what you're dealing with is not normal. It's not the common experience others are tolerating.

I can't complain though because I have this account for like couple decades and loosing that account would be painful. Apple did ban my account twice on the grounds that it's a US account while I'm not physically located there. I was able to revert the ban by explaining that I've got a US legal entity (account, banking card, etc) and thus I beg to continue using it. Not taking chances for the third time, so I silently endure.

I don't believe that to be true, I have been having issues with Apple bugs for the last 7-8 years. Totally unnecessary friction due to features I do not want and do not use.

While it's not very user-friendly software (command line, after all), duperemove has proven quite useful for my filesystems.

> clone: reflink-capable filesystems only. Try to clone both files with the FIDEDUPERANGE ioctl(3p) (or BTRFS_IOC_FILE_EXTENT_SAME on older kernels). This will free up duplicate extents while preserving the metadata of both. Needs at least kernel 4.2.

Then again, this app was written with SwiftUI, which hasn't received some handy features before macOS 12 and is still way behind AppKit.

When I see an app that's not compatible with the second most recent macOS, I assume the dev either didn't know better or they were too lazy to write workarounds / shims for the latest-and-greatest shiny stuff.

Anyway, congrats to Siracusa on the release, great idea, etc. etc.

The author is a "household" name in the macOS / Apple scene for a long time even before the podcast. If someone is spending all their life blogging about all things Apple on outlets like ArsTechnica and is consistently putting out new content on podcasts for decades they will naturally have a better distribution.

How many years did you spend on building up your marketing and distribution reach?

For example if you have multiple node_modules, or app installs, or source photos/videos (ones you don't edit), or music archives, then hardlinks work just fine.