Dedupd
Local daemon that deduplicates Hugging Face models across tools
1 repository
ActivededupdRust · Always-on local daemon that deduplicates Hugging Face models across Ollama, LM Studio, MLX and llama.cpp using one content-addressed store, and archives them for offline use. | Release v0.1.0 | GitHub | MIT | 1 open issues | 0 open pull requests | 0 | Health 100Good |
|---|
Health
Health 100Good1 repository weighed: 0 at attention, 0 to watch
Nothing needs attention.
Issues
1 needing triageNeeds triage, waits for a maintainer to look at itdedupd #1renovate[bot] (automation)idle
Releases
Up to date- descoped/dedupdv0.1.0dedupd 0.1.0 by oranheim
Release notes
The first release of dedupd: the daemon, the
dedupdcommand and the macOS app, in one disk image signed with Developer ID and notarised by Apple.dedupd keeps each model file once, in a content-addressed store, and gives Ollama, Bionic and LM Studio, and the Hugging Face cache (
hf,huggingface_hub,mlx_lm, llama.cpp) a copy-on-write clone of it. See the README and the documentation.Requirements
- macOS 26 on Apple silicon.
- APFS, with the dedupd store (
~/.dedupd) and the apps' model folders on the same volume.
Install
- Open
dedupd-0.1.0.dmgand drag Dedupd to Applications. - Open Dedupd. It runs in the menu bar.
- Install the background service with the
dedupdcommand the app carries, which copies itself to~/.dedupd/binand registers a launchd agent:
The Install button on the app's Daemon screen runs the same command; it has been tested only against a simulated agent so far./Applications/Dedupd.app/Contents/Helpers/dedupd install
The daemon takes the models the apps already hold into the store; the apps' files are not changed. Quitting the app leaves the daemon running.
Included
- Download once for several apps, adoption of existing models, and replacement of duplicate copies by clones.
- The Hugging Face Hub API and the Ollama registry API on
127.0.0.1:8642, answered from the store. - App profiles: which app takes which format, by which method and at what cost in disk space. 4-bit and 8-bit MLX models are placed into Ollama as clones with a rewritten index.
- An archive disk: a second copy of every stored file, and archive and restore per model.
- The app: Library, Discover, Downloads, Storage and Daemon screens, and the menu bar panel.
Not included yet
- Updates for the app. A new version is installed by replacing the app; the Daemon screen then offers to update the daemon.
- Linux, ModelScope and the catalogue of model families; see the roadmap.
Checksum
92c7a5c1c1d0a0e787ad2deab50602ffd167bf0681e88e940d0888e9f5919872 dedupd-0.1.0.dmg
Latest commits
Up to dateDrop the Licence section from the site's home page; the footer names the licence0 additions and 5 deletions in 1 file
descoped/dedupdoranheim ecb787c
Drop the note on model licences from the READMEs0 additions and 4 deletions in 2 files
descoped/dedupdoranheim b06cfb3
Drop the affiliation line from the READMEs; keep the note on model licences2 additions and 4 deletions in 2 files
descoped/dedupdoranheim 957f5ac
Remember when Ollama refuses a model, and say so when a job places nothing426 additions and 19 deletions in 11 files
descoped/dedupdoranheim a9e70cc
Give the app an icon made from the monogram64 additions and 130 deletions in 18 files
descoped/dedupdoranheim 4f32dd8
Count the site's visitors with Plausible11 additions and 0 deletions in 1 file
descoped/dedupdoranheim 8cf273b
Lay the site out for phones142 additions and 5 deletions in 3 files
descoped/dedupdoranheim d4878e7
Make the download button download the latest disk image92 additions and 9 deletions in 4 files
descoped/dedupdoranheim fb3e184
Give the site a download button and a footer81 additions and 2 deletions in 3 files
descoped/dedupdoranheim c0f358e
Set the download link while the page is rendered55 additions and 37 deletions in 6 files
descoped/dedupdoranheim 1d609dd