Model Gallery
Discover and install AI models from our curated collection
Find Your Perfect Model
Filter by Model Type
Browse by Tags
hy4-preview
# Hy4-preview GGUF
Three GGUF builds of Hy4-Preview: https://huggingface.co/tencent/Hy4-preview
**Language / 语言:** English · 中文
**Neither file runs on stock llama.cpp.** The `hyv4` architecture is not upstream. Apply the
patches in `hy4-preview-patch/`
## English
### 1. What these are
**`Hy4-preview-Q4_K_M.gguf`** — a conventional Q4_K_M. Most tensors are Q4_K; `ffn_down_exps`
gets Q6_K on 37 layers via llama.cpp's own logic. Use this unless you are memory-constrained.
**`Hy4-preview-UD-IQ1_M.gguf`** - mixed precision with UD-IQ1_M strategy at ~2.44 bpw, roughly **half the size** for the
same model. The routed-expert `gate`/`up` projections run at 1.75 bpw (IQ1_M) and 2.0625 bpw (IQ2_XXS).
**`Hy4-preview-STQ1_0.gguf`** — mixed precision with MIX-STQ1_0 strategy at ~2.38 bpw, roughly **half the size** for the
same model. The routed-expert `gate`/`up` projections run at 1.3125 bpw (STQ1_0) on 29 layers and
2.0625 bpw (IQ2_XXS) on the other 48. See section 3.
### 2. Running them
Build a patched llama.cpp
```bash
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout 0cea36222
...
Repository: localai
glm-5.3
# GLM-5.3
GLM-5.3 uses the same base model as GLM-5.2 — every gain comes from post-training. Compared with GLM-5.2, it is much better at complex coding and long-horizon tasks:
+ Stronger Coding: GLM-5.3 is the most capable open-weights model for coding, with a 50% improvement over GLM-5.2 on our in-house Z.ai Code Bench. It also achieve open-source SOTA on public benchmarks including Terminal Bench 3.0 and Agents' Last Exam.
+ Emergent Cyber Capability: As we scaled post-training, cyber capability developed faster than we expected. GLM-5.3 is state of the art on CyberGym for vulnerability discovery, and its gains are largest further up the exploitation chain, where it more than doubles GLM-5.2 on exploitation benchmarks.
## Benchmark
### Serve GLM-5.3 Locally
GLM-5.3 supports deployment with the following frameworks. Feel free to try them out:
- SGLang — see cookbook
- vLLM — see recipes
- TokenSpeed — see here
- Transformers — see transformers docs
- KTransformers — see tutorial
- Unsloth — see guide
- For deployment on the `Ascend NPU` platform, inference frameworks such as vLLM-Ascend, xLLM and SGLang are supported — see here.
### Note
...
Repository: localaiLicense: other
qwen3.5-9b-defiant-fable
Qwen3.5 9B Defiant Fable in the plain NEO-imatrix Q4_K_M GGUF format.
This fallback offers the same multimodal reasoning, coding, and creative
capabilities without enabling multi-token prediction.
Repository: localaiLicense: apache-2.0
deepseek-v4-flash-0731
# DeepSeek-V4-Flash-0731
Technical Report👁️
## Introduction
**DeepSeek-V4-Flash-0731** is the official release of **DeepSeek-V4-Flash**, superseding the preview version, with substantially enhanced agentic capabilities. It has the same model structure as DeepSeek-V4-Flash-DSpark, i.e. it comes with a speculative decoding module attached.
DeepSeek-V4-Flash-0731 outperforms DeepSeek-V4-Pro (Preview) on benchmarks listed below despite its far smaller activated parameter count, and is broadly competitive with the strongest proprietary models available.
Notes:
1. For the Code Agent tasks among the public benchmarks above, DeepSeek-V4-Flash-0731 is evaluated with the minimal mode of DeepSeek Harness (to be released) as the agent framework, using the `max` reasoning effort level with `temperature = 1.0, top_p = 0.95`.
2. † DSBench-FullStack is an internal full-stack development test set; DSBench-Hard is an internal test set of difficult coding-agent problems.
## Chat Template
...
Repository: localaiLicense: mit
glm-5.2
# GLM-5.2
👋 Join our WeChat or Discord community.
📖 Check out the GLM-5.2 blog and GLM-5 Technical report.
📍 Use GLM-5.2 API services on Z.ai API Platform.
🔜 Try GLM-5.2 here.
[Paper]
[GitHub]
## Introduction
We're introducing GLM-5.2, our latest flagship model for long-horizon tasks. It marks a substantial leap in long-horizon task capability over its predecessor GLM-5.1 and, for the first time, delivers that capability on a **solid 1M-token context**. GLM-5.2's new capabilities include:
- **Solid 1M Context:** A solid 1M-token context that stably sustains long-horizon work
- **Advanced Coding with Flexible Effort**: Stronger coding capabilities with multiple thinking effort levels to balance performance and latency
- **Improved Architecture**: We propose IndexShare, which reuses the same indexer across every four sparse attention layers, reducing per-token FLOPs by 2.9× at a 1M context length. We also improve GLM-5.2’s MTP layer for speculative decoding, increasing the acceptance length by up to 20%
- **Pure Open**: An MIT open-source license — no regional limits, technical access without borders
## Benchmark
## Serve GLM-5.2 Locally
...
Repository: localaiLicense: mit
gemma-4-12b-agentic-fable5-composer2.5-v2-3.5x-tau2
Hugging Face |
GitHub |
Launch Blog |
Documentation
License: Apache 2.0 | Authors: Google DeepMind
> [!Note]
> This model card is for the Gemma 4 12B Unified model, which is part of the Gemma 4 family of open models. Built with the same multimodal functionality as Gemma 4 E2B and E4B (text, audio, image, and video inputs), it brings native audio and vision understanding directly to local environments without the need for separate encoders. This unified approach to multimodality makes the model encoder-free, offering a deployment size that is perfect for consumer devices and streamlined local execution.
Gemma is a family of open models built by Google DeepMind. Gemma 4 models are multimodal, handling text and image input (with audio supported on E2B, E4B, and 12B) and generating text output. This release includes open-weights models in both pre-trained and instruction-tuned variants. Gemma 4 features a context window of up to 256K tokens and maintains multilingual support in over 140 languages.
...
Repository: localaiLicense: apache-2.0
gemma-4-12b-coder-fable5-composer2.5-v1
Hugging Face |
GitHub |
Launch Blog |
Documentation
License: Apache 2.0 | Authors: Google DeepMind
> [!Note]
> This model card is for the Gemma 4 12B Unified model, which is part of the Gemma 4 family of open models. Built with the same multimodal functionality as Gemma 4 E2B and E4B (text, audio, image, and video inputs), it brings native audio and vision understanding directly to local environments without the need for separate encoders. This unified approach to multimodality makes the model encoder-free, offering a deployment size that is perfect for consumer devices and streamlined local execution.
Gemma is a family of open models built by Google DeepMind. Gemma 4 models are multimodal, handling text and image input (with audio supported on E2B, E4B, and 12B) and generating text output. This release includes open-weights models in both pre-trained and instruction-tuned variants. Gemma 4 features a context window of up to 256K tokens and maintains multilingual support in over 140 languages.
...
Repository: localaiLicense: gemma
qwen3.6-35b-a3b-claude-4.6-opus-reasoning-distilled
# 🔥 Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled
A reasoning SFT fine-tune of `Qwen/Qwen3.6-35B-A3B` on chain-of-thought (CoT) distillation mostly sourced from Claude Opus 4.6. The goal is to preserve Qwen3.6's strong agentic coding and reasoning base while nudging the model toward structured Claude Opus-style reasoning traces and more stable long-form problem solving.
The training path is text-only. The Qwen3.6 base architecture includes a vision encoder, but this fine-tuning run did not train on image or video examples.
- **Developed by:** @hesamation
- **Base model:** `Qwen/Qwen3.6-35B-A3B`
- **License:** apache-2.0
This fine-tuning run is inspired by Jackrong/Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled, including the notebook/training workflow style and Claude Opus reasoning-distillation direction.
[](https://x.com/Hesamation) [](https://discord.gg/vtJykN3t)
## Benchmark Results
The MMLU-Pro pass used 70 total questions per model: `--limit 5` across 14 MMLU-Pro subjects. Treat this as a smoke/comparative check, not a release-quality full benchmark.
...
Repository: localaiLicense: apache-2.0
nanbeige4.1-3b-q8
Nanbeige4.1-3B is built upon Nanbeige4-3B-Base and represents an enhanced iteration of our previous reasoning model, Nanbeige4-3B-Thinking-2511, achieved through further post-training optimization with supervised fine-tuning (SFT) and reinforcement learning (RL). As a highly competitive open-source model at a small parameter scale, Nanbeige4.1-3B illustrates that compact models can simultaneously achieve robust reasoning, preference alignment, and effective agentic behaviors.
Key features:
Strong Reasoning: Capable of solving complex, multi-step problems through sustained and coherent reasoning within a single forward pass, reliably producing correct answers on benchmarks like LiveCodeBench-Pro, IMO-Answer-Bench, and AIME 2026 I.
Robust Preference Alignment: Outperforms same-scale models (e.g., Qwen3-4B-2507, Nanbeige4-3B-2511) and larger models (e.g., Qwen3-30B-A3B, Qwen3-32B) on Arena-Hard-v2 and Multi-Challenge.
Agentic Capability: First general small model to natively support deep-search tasks and sustain complex problem-solving with >500 rounds of tool invocations; excels in benchmarks like xBench-DeepSearch (75), Browse-Comp (39), and others.
Repository: localaiLicense: apache-2.0
nanbeige4.1-3b-q4
Nanbeige4.1-3B is built upon Nanbeige4-3B-Base and represents an enhanced iteration of our previous reasoning model, Nanbeige4-3B-Thinking-2511, achieved through further post-training optimization with supervised fine-tuning (SFT) and reinforcement learning (RL). As a highly competitive open-source model at a small parameter scale, Nanbeige4.1-3B illustrates that compact models can simultaneously achieve robust reasoning, preference alignment, and effective agentic behaviors.
Key features:
Strong Reasoning: Capable of solving complex, multi-step problems through sustained and coherent reasoning within a single forward pass, reliably producing correct answers on benchmarks like LiveCodeBench-Pro, IMO-Answer-Bench, and AIME 2026 I.
Robust Preference Alignment: Outperforms same-scale models (e.g., Qwen3-4B-2507, Nanbeige4-3B-2511) and larger models (e.g., Qwen3-30B-A3B, Qwen3-32B) on Arena-Hard-v2 and Multi-Challenge.
Agentic Capability: First general small model to natively support deep-search tasks and sustain complex problem-solving with >500 rounds of tool invocations; excels in benchmarks like xBench-DeepSearch (75), Browse-Comp (39), and others.
Repository: localaiLicense: apache-2.0
silero-vad-sherpa
Silero VAD served through the sherpa-onnx backend. Uses the same ONNX weights as the dedicated silero-vad backend, loaded through sherpa-onnx's C VAD API. Pairs with the sherpa-onnx ASR entries for round-trip audio pipelines.
Repository: localaiLicense: mit
deepseek-v4-flash-spark-exl3-vllm-cpp
DeepSeek V4 Flash's Spark and GB10-oriented REAP-K216 EXL3 checkpoint,
served by vllm.cpp. It needs CUDA and roughly 100 GiB for its large
rank-sliced checkpoint.
The repository is pinned to its latest recorded revision. vllm.cpp's
existing runtime evidence measured the older 22f28d32b9b29b4352eaa380ff8c2c170b2847ab
revision; this entry does not claim that the newer revision has passed the
same end-to-end gate.
Repository: localai
qwen3.6-27b-nvfp4-vllm-cpp
Qwen3.6-27B in NVFP4, served by vllm.cpp: LocalAI's own C++ port of vLLM, with
no Python at inference time. This is the reference text-generation checkpoint
the engine is gated on, token-for-token identical to vLLM's own greedy output
over the 235-prompt correctness battery, and measured at or above vLLM's
throughput at every concurrency from 1 to 32.
The weights are PINNED to revision 890bdef7. That pin is load-bearing, not
housekeeping: the same repository name was later re-quantized to FP8 W8A8
throughout, so an unpinned copy of this entry serves entirely different
weights with no error and none of the measured behaviour above.
Needs a Blackwell-class NVIDIA GPU (NVFP4 has no kernel on older
architectures) and roughly 25 GB of weights plus KV cache. Tool calling and
the thinking split are parsed inside the engine.
Repository: localaiLicense: apache-2.0
qwen3.6-27b-nvfp4-mtp-vllm-cpp
Qwen3.6-27B NVFP4 on vllm.cpp with MTP speculative decoding enabled.
MTP (Multi-Token Prediction) drafts from a head that ships inside the target
checkpoint's own mtp.* tensors, so there is no second model to download and
no extra weights to manage. The verifier accepts roughly 85% of drafted
tokens on prose and 92% on code, worth about 1.5x to 1.6x the decode
throughput of the same weights with speculation off, and it holds that lead
at concurrency 2, 4 and 8.
Same weights and same revision pin as qwen3.6-27b-nvfp4-vllm-cpp; install
that entry instead if you would rather not spend the extra memory. The
speculative state (a doubled recurrent-state slot plus the draft cache and
head) costs roughly 3.6 GB on top of the base footprint.
MTP here is depth 1 by construction: the engine refuses
num_speculative_tokens above 1 for this method.
Repository: localaiLicense: apache-2.0
qwen3.6-27b-nvfp4-dflash-vllm-cpp
Qwen3.6-27B NVFP4 on vllm.cpp with DFlash block-diffusion speculative
decoding: the fastest configuration of this model the engine ships.
Where MTP drafts one token at a time, DFlash drafts a whole 16-token block in
a single non-autoregressive pass from a separate 3.5 GB drafter, then the
target verifies the block in one step. At concurrency 1 that measures 2.9x
the throughput of the same weights with speculation off, and at or above
vLLM's own DFlash-on decode.
Both checkpoints are installed for you: the target as a revision-pinned
snapshot, the drafter into models/Qwen3.6-27B-DFlash, which is where the
backend looks when speculative_config.model names it. The drafter shares the
target's embed_tokens and lm_head, so the two are not independently
swappable.
Needs a Blackwell-class NVIDIA GPU and roughly 28 GB of weights in total.
Repository: localaiLicense: apache-2.0
minimax-h3-ref2va-q4
MiniMax-H3 served by vllm.cpp, LocalAI's own C++ port of vLLM. It generates
video AND audio jointly, so a clip comes back as an MP4 with a real
soundtrack rather than a silent render.
This is the Q4_K_M quantisation of the Ref2VA partition, the one that takes
REFERENCE conditioning: a reference image, a reference clip, or reference
audio, prepended as their own blocks so the subject or style carries into the
generated video. For plain text-to-video or first/last-frame conditioning use
minimax-h3-fl2va-q4 instead - the two partitions are separate checkpoints and
each refuses the other's tasks.
Use this Q4_K_M build, NOT the NVFP4 Ref2VA weights: NVFP4 renders a
multicolour patch grid, and it took three investigations upstream to
establish that the fault is the quantisation rather than the reference path.
On Q4_K_M the same code renders coherently.
Roughly 40 GB of weights across five files, plus the two VAE configs that
carry the latent statistics. The default canvas is 1344x768 at 124 frames and
24 fps, about 5.2 seconds. Generation is slow - roughly 176 s per denoise
step at that canvas on a 20-SM device, so the 50-step default is a multi-hour
job. Muxing the finished frames needs ffmpeg on the host.
Repository: localaiLicense: other
ace-step-turbo
ACE-Step 1.5 Turbo is a music generation model that can create music from text descriptions,
lyrics, or audio samples. Supports both simple text-to-music and advanced music generation
with metadata like BPM, key scale, and time signature.
Repository: localaiLicense: mit
speechbrain-ecapa-tdnn
Speaker (voice) recognition with SpeechBrain's ECAPA-TDNN trained
on VoxCeleb. 192-d L2-normalised embeddings, ~1.9% Equal Error
Rate on VoxCeleb1-O. APACHE 2.0 — commercial-safe.
The checkpoint is auto-downloaded from HuggingFace on first
LoadModel (no separate weight file in gallery `files:`). Points at
the upstream SpeechBrain HF repo directly — same bytes every
deployment.
Repository: localaiLicense: apache-2.0
rfdetr-cpp-nano
RF-DETR Nano object detection model, served via the native rfdetr.cpp backend (ggml + purego, no Python).
Q8_0 quantization is the recommended default for CPU: same accuracy as F16/F32, ~20MB on disk, fastest CPU latency.
Pure C++/ggml runtime; no Python dependencies. Drop-in for the /v1/detection endpoint.
Repository: localaiLicense: apache-2.0
edgetam
EdgeTAM is an ultra-efficient variant of the Segment Anything Model (SAM) for image segmentation.
It uses a RepViT backbone and is only ~16MB quantized (Q4_0), making it ideal for edge deployment.
Supports point-prompted and box-prompted image segmentation via the /v1/detection endpoint.
Powered by sam3.cpp (C/C++ with GGML).
Repository: localaiLicense: apache-2.0
fast-math-qwen3-14b
By applying SFT and GRPO on difficult math problems, we enhanced the performance of DeepSeek-R1-Distill-Qwen-14B and developed Fast-Math-R1-14B, which achieves approx. 30% faster inference on average, while maintaining accuracy.
In addition, we trained and open-sourced Fast-Math-Qwen3-14B, an efficiency-optimized version of Qwen3-14B`, following the same approach.
Compared to Qwen3-14B, this model enables approx. 65% faster inference on average, with minimal loss in performance.
Technical details can be found in our github repository.
Note: This model likely inherits the ability to perform inference in TIR mode from the original model. However, all of our experiments were conducted in CoT mode, and its performance in TIR mode has not been evaluated.
Repository: localaiLicense: apache-2.0
Page
1
of many