OpenAI builds and deploys advanced AI models like GPT-4o for autonomous agents and workflows.
Shoehorn
0Popularity
About Shoehorn
Shoehorn solves a specific problem: fitting large language models into the memory you actually have on your machine. Instead of using preset quantizations that waste headroom or fail to fit, it calculates a per-tensor mixed-precision assignment that uses nearly all available memory, often within 13KB of the budget. The tool works with any GGUF model and requires llama.cpp as a backend. It is designed for programmers running local LLMs, researchers with limited GPU memory, and data analysts on consumer hardware who need local inference without cloud dependency. Shoehorn is command-line only and currently offers no GPU acceleration for some configurations. It is open-source and can be installed via Homebrew with a single command.
Key features
- Per-tensor mixed-precision memory optimization
- Nearly full utilization of available memory (within 13KB of budget)
- Works with any GGUF model
- One-command install via Homebrew
- Command-line interface only
- Open-source and free to use
- Local inference without cloud dependency
- Requires llama.cpp as backend
Use cases
- Running large models on limited hardware
- Maximizing model quality per memory budget
- Checking which models fit your machine
Pros
- Calculates per-tensor mixed-precision assignments to maximize memory usage, often within 13KB of the available budget.
- Works with any GGUF model and integrates with llama.cpp as the inference backend.
- Provides a local web app for model selection and fitting, simplifying the user experience.
- Open-source and MIT-licensed, allowing for community contributions and transparency.
- Offers a browser-based tool to preview which models fit specific hardware configurations.
Cons
- Command-line only for advanced users, with limited GPU acceleration in some configurations.
- Requires manual installation of llama.cpp as a backend for full functionality.
Frequently asked questions about Shoehorn
What does Shoehorn do?
Shoehorn calculates a per-tensor mixed-precision assignment to fit large language models into the available memory on a machine, maximizing usage while ensuring the model loads successfully.
Who is Shoehorn designed for?
It is designed for programmers running local LLMs, researchers with limited GPU memory, and data analysts using consumer hardware who need local inference without cloud dependency.
How do I install Shoehorn?
Shoehorn can be installed via Homebrew with a single command, or built from source using Cargo. It requires llama.cpp to be installed on your system.
Does Shoehorn support GPU acceleration?
Shoehorn supports GPU acceleration in some configurations, but not all. The tool primarily focuses on memory optimization for local inference.
What models does Shoehorn work with?
Shoehorn works with any GGUF model and outputs a standard GGUF v3 file that can be used with llama.cpp and other downstream tools.
How do I get started with Shoehorn?
Install Shoehorn via Homebrew, ensure llama.cpp is on your PATH, then run 'shoehorn ui' to open the local web app, select a model, and fit it to your hardware.
