Skip to content
Descoped Source Code Portalv2.2.13
PollingSign in

Dedupd

Local daemon that deduplicates Hugging Face models across tools

1 repository

Repositories of Dedupd, sortable by every column
ActivededupdRust · Always-on local daemon that deduplicates Hugging Face models across Ollama, LM Studio, MLX and llama.cpp using one content-addressed store, and archives them for offline use.
Release v0.1.0GitHubMIT1 open issues0 open pull requests0Health 100Good

Health

Health 100Good

1 repository weighed: 0 at attention, 0 to watch

Nothing needs attention.

Issues

1 needing triage
  1. Dependency Dashboard

    Needs triage, waits for a maintainer to look at itdedupd #1renovate[bot] (automation)idle

All 1 open

Releases

Up to date
  1. descoped/dedupdv0.1.0dedupd 0.1.0 by oranheim
    Release notes

    The first release of dedupd: the daemon, the dedupd command and the macOS app, in one disk image signed with Developer ID and notarised by Apple.

    dedupd keeps each model file once, in a content-addressed store, and gives Ollama, Bionic and LM Studio, and the Hugging Face cache (hf, huggingface_hub, mlx_lm, llama.cpp) a copy-on-write clone of it. See the README and the documentation.

    Requirements

    • macOS 26 on Apple silicon.
    • APFS, with the dedupd store (~/.dedupd) and the apps' model folders on the same volume.

    Install

    1. Open dedupd-0.1.0.dmg and drag Dedupd to Applications.
    2. Open Dedupd. It runs in the menu bar.
    3. Install the background service with the dedupd command the app carries, which copies itself to ~/.dedupd/bin and registers a launchd agent:
      /Applications/Dedupd.app/Contents/Helpers/dedupd install
      
      The Install button on the app's Daemon screen runs the same command; it has been tested only against a simulated agent so far.

    The daemon takes the models the apps already hold into the store; the apps' files are not changed. Quitting the app leaves the daemon running.

    Included

    • Download once for several apps, adoption of existing models, and replacement of duplicate copies by clones.
    • The Hugging Face Hub API and the Ollama registry API on 127.0.0.1:8642, answered from the store.
    • App profiles: which app takes which format, by which method and at what cost in disk space. 4-bit and 8-bit MLX models are placed into Ollama as clones with a rewritten index.
    • An archive disk: a second copy of every stored file, and archive and restore per model.
    • The app: Library, Discover, Downloads, Storage and Daemon screens, and the menu bar panel.

    Not included yet

    • Updates for the app. A new version is installed by replacing the app; the Daemon screen then offers to update the daemon.
    • Linux, ModelScope and the catalogue of model families; see the roadmap.

    Checksum

    92c7a5c1c1d0a0e787ad2deab50602ffd167bf0681e88e940d0888e9f5919872  dedupd-0.1.0.dmg
    

Latest commits

Up to date
  1. Drop the Licence section from the site's home page; the footer names the licence0 additions and 5 deletions in 1 file

    descoped/dedupdoranheim ecb787c

  2. Drop the note on model licences from the READMEs0 additions and 4 deletions in 2 files

    descoped/dedupdoranheim b06cfb3

  3. Drop the affiliation line from the READMEs; keep the note on model licences2 additions and 4 deletions in 2 files

    descoped/dedupdoranheim 957f5ac

  4. Remember when Ollama refuses a model, and say so when a job places nothing426 additions and 19 deletions in 11 files

    descoped/dedupdoranheim a9e70cc

  5. Give the app an icon made from the monogram64 additions and 130 deletions in 18 files

    descoped/dedupdoranheim 4f32dd8

  6. Count the site's visitors with Plausible11 additions and 0 deletions in 1 file

    descoped/dedupdoranheim 8cf273b

  7. Lay the site out for phones142 additions and 5 deletions in 3 files

    descoped/dedupdoranheim d4878e7

  8. Make the download button download the latest disk image92 additions and 9 deletions in 4 files

    descoped/dedupdoranheim fb3e184

  9. Give the site a download button and a footer81 additions and 2 deletions in 3 files

    descoped/dedupdoranheim c0f358e

  10. Set the download link while the page is rendered55 additions and 37 deletions in 6 files

    descoped/dedupdoranheim 1d609dd

Commit log of the group

Insights of this product