OpenAI builds and deploys advanced AI models like GPT-4o for autonomous agents and workflows.
Ferrum
0Popularity
About Ferrum
Ferrum is an open-source Rust-based tool for running and serving large language models locally on supported hardware. It provides a single binary for local inference on Apple Silicon using Metal and on NVIDIA GPUs using CUDA, without requiring Python, PyTorch, or vLLM at runtime. The tool offers both an interactive CLI for direct model interaction and an HTTP server exposing an OpenAI-compatible API for broader client integration. Ferrum supports explicit model selection, continuous batching, and server-side controls like paged KV cache and session management. It targets local and private serving scenarios, enabling users to inspect models before download, run interactive sessions, and deploy models behind familiar HTTP endpoints. The project emphasizes release validation and hardware-specific optimizations, with documented support for models such as Qwen3.5, Qwen3, and Llama 3.1 variants.
Key features
- Rust-native local LLM inference
- Metal acceleration for Apple Silicon
- CUDA acceleration for NVIDIA GPUs
- OpenAI-compatible HTTP API
- Interactive CLI for direct model interaction
- Continuous batching and paged KV cache
- Session and prefix caching support
- Model inspection before download
Use cases
- Local model experimentation and testing
- Private LLM serving behind OpenAI-compatible endpoints
- Interactive chat sessions with multi-turn context
Pros
- Rust-native implementation without Python runtime dependency
- Supports both Apple Silicon Metal and NVIDIA CUDA backends
- OpenAI-compatible API for seamless client integration
- Single binary for both CLI and server operations
- Explicit model selection with no silent defaults
Cons
- Limited to specific hardware: Apple Silicon Metal or NVIDIA sm89 GPUs
- CUDA support requires NVIDIA driver 551.78 or newer on Windows
- Linux CUDA requires NVIDIA sm89 GPU, CUDA 12.4, and NCCL runtimes
- No free hosted service; requires local hardware
Frequently asked questions about Ferrum
What is Ferrum?
Ferrum is an open-source Rust-based command-line tool for running and serving large language models locally on supported Metal and CUDA hardware. It provides a single binary for local inference without requiring Python, PyTorch, or vLLM at runtime.
Does Ferrum require Python?
No, Ferrum does not require a Python runtime. Linux CUDA uses the host's NVIDIA driver, CUDA 12.4, and NCCL runtimes, while the Windows installer includes CUDA and VC runtimes separately.
Which models are officially supported by Ferrum?
Ferrum v0.8 formally supports models such as Qwen3.5 4B and 35B-A3B, Qwen3 30B-A3B, and Llama 3.1 8B dense. Other models may be available as development evidence before entering the support matrix.
Can existing OpenAI clients connect to Ferrum?
Yes, Ferrum implements OpenAI-compatible Chat Completions and stateless Responses endpoints, allowing existing OpenAI clients to connect to it with minimal changes.
Is Ferrum a hosted AI service?
No, Ferrum is inference software that runs on hardware you control. Model licenses and data-handling choices remain entirely under the user's control.
How do I get started with Ferrum?
Install Ferrum using the provided one-command installer for your system, verify the release checksums, and then use commands like 'ferrum doctor' to inspect models or 'ferrum run' to start an interactive session.
