Guides / Local LLMs for 16GB devices: discrete GPU vs Mac · 中文版

Local LLMs for 16GB devices: discrete GPU vs Mac

For people with a 16GB discrete GPU or a 16GB-memory Mac: the configurations here passed this site's 16GB estimate filter, which is not a guarantee they run; work out which kind of 16GB you have, then pick by run config.

Editorial picks

First work out whether your 16GB is discrete GPU VRAM or Mac unified memory — they are not the same budget; then read each card's "Run config" rather than the model name: MLX packs run only on Apple Silicon. The configurations here passed this site's 16GB estimate filter, which is not a guarantee they run; actual needs also depend on the runtime, context length and what else is using memory. Open "Show how to run" to copy a command; every text model is at /guide/text/.

  • Coding agent; what passed is the MLX 4-bit pack, Apple Silicon only Ornith-1.0-9B (DeepReinforce)

    Fits the budget via lmstudio-community/Ornith-1.0-9B-MLX-4bit (4bit): min ~5.4GB (estimate) — the name and the deploy command below belong to that variant, not to the canonical repo.

    What passed the filter is the pack lmstudio-community/Ornith-1.0-9B-MLX-4bit: this site's rough estimate is about 5.4GB and the card suggests planning for 7GB — not a measurement. Discrete-GPU users should look at the GGUF packs on the detail page, which this site has not estimated; community feedback puts the Q4_K_M build at 6–16GB of VRAM — a community statement. The material states MIT, commercial use allowed, tool calling. Community feedback is split: usable inside an agent framework, less stable than Qwen 3.5/3.6 for standalone coding; try it on a small scale first.

Within the 16GB budget

20 configurations whose structured footprint estimate (min VRAM ≤ 16GB) fits the budget. 174 other entries in this domain are unknown or over budget and are not listed here — see the full text list. A VRAM estimate is not a unified-memory budget: on a Mac the same number does not by itself promise it runs in 16GB of shared memory.

  • LFM2.5-2.6B (LiquidAI)

    Apple Silicon 用 4-bit 量化 LFM，始终思考模式

    Params
    2.6B
    Commercial
    需自查

    This tier is met by lmstudio-community/LFM2.5-2.6B-MLX-4bit (4bit)

    VRAM
    ~1.6GB 4-bit(estimate)

    VRAM and command below belong to this config

    Details
    Show how to run 
    mlx_lm.generate --model lmstudio-community/LFM2.5-2.6B-MLX-4bit
    License
    LFM Open License v1.0
    Context
    128k
    China access
    需代理
    Other forms
    原生 · GGUF·liquidai · GGUF·lmstudio · 4bit · 8bit

  • Qwen3.8-27B-MTPLX-Optimized-Speed (Youssofal)

    Mac 上 Qwen3.8-27B 4-bit量化

    Params
    27B
    Commercial
    可商用

    Run config Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed

    VRAM
    ~16GB 4-bit(estimate)
    Details
    Show how to run 
    mtplx serve --model Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed
    License
    apache-2.0
    Context
    262k
    China access
    需代理

  • Qwen3.6-27B-MTPLX (Youssofal)

    Qwen3.6-27B 的 MLX 推理加速版供苹果芯片

    Params
    27B
    Commercial
    可商用

    Run config Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed

    VRAM
    ~16GB 4-bit(estimate)
    Details
    Show how to run (+2 more) 
    brew install youssofal/mtplx/mtplx

    2 more way(s) to run on the detail page.

    License
    Apache-2.0
    China access
    需代理

  • SuperGemma4-26B (Jiunsong)

    Gemma4 26B 无审查 MLX, 本地代理加速

    Params
    26B (4-bit)
    Commercial
    限制商用

    Run config Jiunsong/supergemma4-26b-uncensored-mlx-4bit-v2

    VRAM
    ~16GB 4-bit(estimate)
    Details
    Show how to run (+1 more) 
    mlx_lm.server --model Jiunsong/supergemma4-26b-uncensored-mlx-4bit-v2

    1 more way(s) to run on the detail page.

    License
    gemma
    China access
    需代理

  • Ornith-1.0-9B (DeepReinforce)

    MLX 8-bit量化，为Apple Silicon优化

    Params
    9B
    Commercial
    可商用

    This tier is met by lmstudio-community/Ornith-1.0-9B-MLX-4bit (4bit)

    VRAM
    ~5.4GB 4-bit(estimate)

    VRAM and command below belong to this config

    Details
    Show how to run 
    mlx_lm.generate --model lmstudio-community/Ornith-1.0-9B-MLX-4bit
    License
    mit
    Context
    256k
    China access
    需代理
    Other forms
    原生 · GGUF·deepreinforce-ai · GGUF·lmstudio · 4bit · 8bit · GGUF·protolabsai

  • Bonsai-8B-mlx-1bit (PrismML)

    Apple Silicon 的 1-bit LLM，端侧极低内存推理

    Params
    8B
    Commercial
    可商用

    Run config prism-ml/Bonsai-8B-mlx-1bit

    VRAM
    ~1.2GB 1-bit(estimate)
    Details
    Show how to run 
    pip install mlx-lm; pip install mlx@git+https://github.com/PrismML-Eng/mlx.git@prism; from mlx_lm import load; load('prism-ml/Bonsai-8B-mlx-1bit')
    License
    Apache-2.0
    Context
    65k
    China access
    需代理

  • Meta-Llama-3-8B-Instruct-4bit (mlx-community)

    Llama-3-8B 4-bit版，M系列芯片本地推理

    Params
    8B
    Commercial
    限制商用

    Run config mlx-community/Meta-Llama-3-8B-Instruct-4bit

    VRAM
    ~4.8GB 4-bit(estimate)
    Details
    Show how to run (+1 more) 
    pip install mlx-lm && mlx_lm.generate --model mlx-community/Meta-Llama-3-8B-Instruct-4bit --prompt "hello"

    1 more way(s) to run on the detail page.

    License
    llama3
    Context
    8k
    China access
    需代理

  • Qwen3.8-27B (Qwen)

    Mac本地Qwen3.8-27B-4bit量化

    Params
    27B
    Commercial
    可商用

    Run config Youssofal/Qwen3.8-27B-MTPLX-Bare-Speed

    VRAM
    ~16GB 4-bit(estimate)
    Details
    Show how to run 
    mlx_lm.generate --model Youssofal/Qwen3.8-27B-MTPLX-Bare-Speed
    License
    Apache-2.0
    Context
    262k
    China access
    需代理

  • Qwen3.8-27B-Escha-W2 (Escha)

    2-bit 量化 27B 模型，24GB 显卡 64k 上下文

    Params
    27B
    Commercial
    可商用

    Run config EschaLabs/Qwen3.8-27B-Escha-W2

    VRAM
    ~9.2GB 2-bit(estimate)
    Details
    No command indexed

    No command indexed for this config. See the detail page or the official repo; commands from other configs are not borrowed.

    License
    Apache-2.0
    Context
    128k
    China access
    需代理

  • BitCPM-CANN-8B (OpenBMB)

    昇腾原生1.58-bit三元LLM，推理内存降6x，保持95.7%性能

    Params
    8B
    Commercial
    可商用

    Run config openbmb/BitCPM-CANN-8B

    VRAM
    ~2.2GB ternary(estimate)
    Details
    Show how to run (+1 more) 
    transformers: AutoModelForCausalLM.from_pretrained('openbmb/BitCPM-CANN-8B')

    1 more way(s) to run on the detail page.

    License
    apache-2.0
    China access
    需代理

  • gear-manual-qwen3-4b (Raindog)

    面向硬件音乐设备的本地助手，基于Qwen3-4B微调

    Params
    4B
    Commercial
    不可商用

    Run config zak-raindog/gear-manual-qwen3-4b

    VRAM
    ~2.4GB 4-bit(estimate)
    Details
    Show how to run 
    mlx_lm.generate --model zak-raindog/gear-manual-qwen3-4b --prompt "如何使用EP-133?"
    License
    cc-by-nc-4.0
    China access
    需代理

  • Ornith-1.0-35B-MLX (deepreinforce-ai)

    Apple Silicon 上 Claude-Code 式本地编码代理

    Params
    35B (3B 活跃)
    Commercial
    需自查

    Run config nathansutton/Ornith-1.0-35B-UD-Q2_K_XL-MLX

    VRAM
    ~12GB 2-bit(estimate)
    Details
    Show how to run 
    uvx --from git+https://github.com/nathansutton/mlxcc chad
    China access
    需代理

  • Bonsai-27B (Prism ML)

    多模态 27B 模型，纯 Q1_0 量化，语言部分仅 3.8 GB

    Params
    27B
    Commercial
    可商用

    Run config prism-ml/Bonsai-27B-gguf

    VRAM
    ~4.1GB 1-bit(estimate)
    Details
    Show how to run (+1 more) 
    ollama run hf.co/prism-ml/Bonsai-27B-gguf

    1 more way(s) to run on the detail page.

    License
    Apache-2.0
    Context
    262K
    China access
    需代理
    Other forms
    GGUF·prism-ml · GGUF·lmstudio

  • Maple-Preview (DeepGrove)

    20B-A1B 三元推理模型，M4 218 tok/s

    Params
    20B-A1B
    Commercial
    可商用

    Run config deepgrove/maple-preview

    VRAM
    ~5.4GB ternary(estimate)
    Details
    No command indexed

    No command indexed for this config. See the detail page or the official repo; commands from other configs are not borrowed.

    License
    MIT
    Context
    131k
    China access
    需代理

  • Qwen3.8-27B-Atlassian-Q8-mlx (LeanZero)

    Atlassian 开发专用 27B MLX 模型

    Params
    27B
    Commercial
    可商用

    This tier is met by Mihai-LeanZero/Qwen3.8-27B-Atlassian-Q4-mlx (MLX·mihai-leanzero)

    VRAM
    ~16GB 4-bit(estimate)

    VRAM and command below belong to this config

    Details
    Show how to run 
    mlx_lm.generate --model Mihai-LeanZero/Qwen3.8-27B-Atlassian-Q4-mlx --prompt "Which Forge module adds a panel to the Jira issue view?"
    License
    Apache-2.0
    Context
    128k
    China access
    需代理
    Other forms
    MLX·mihai-leanzero · MLX·mihai-leanzero

  • Granite-4.2-3B-MLX-8bit (LM Studio)

    IBM Granite 3B MLX 量化，苹果芯片推理

    Params
    3B
    Commercial
    可商用

    Run config lmstudio-community/granite-4.2-3b-MLX-8bit

    VRAM
    ~3.3GB 8-bit(estimate)
    Details
    Show how to run 
    mlx_lm.generate --model lmstudio-community/granite-4.2-3b-MLX-8bit
    License
    apache-2.0
    China access
    需代理
    Other forms
    8bit · GGUF · 4bit

  • Qwen3.5-9B-Atlassian-Q4-mlx (LeanZero)

    面向 Atlassian Forge 的 6-bit MLX 模型

    Params
    9B
    Commercial
    可商用

    Run config Mihai-LeanZero/Qwen3.5-9B-Atlassian-Q4-mlx

    VRAM
    ~5.4GB 4-bit(estimate)
    Details
    Show how to run 
    mlx_lm.generate --model Mihai-LeanZero/Qwen3.5-9B-Atlassian-Q4-mlx --prompt "Which Forge module adds a panel to the Jira issue view?"
    License
    Apache-2.0
    Context
    32k
    China access
    需代理
    Other forms
    MLX·mihai-leanzero · MLX·mihai-leanzero

  • Qwen3.6-35B-A3B-Escha-W2 (EschaLabs)

    2-bit Qwen3.6-35B-A3B，12.3GB，单卡本地推理

    Params
    35B (3B active)
    Commercial
    可商用

    Run config EschaLabs/Qwen3.6-35B-A3B-Escha-W2

    VRAM
    ~12GB 2-bit(estimate)
    Details
    No command indexed

    No command indexed for this config. See the detail page or the official repo; commands from other configs are not borrowed.

    License
    Apache-2.0
    Context
    128k
    China access
    需代理

  • Bonsai-27B-mlx-1bit (Prism ML)

    1-bit 27B 模型, 可在手机和笔记本本地运行推理

    Params
    27B
    Commercial
    可商用

    Run config prism-ml/Bonsai-27B-mlx-1bit

    VRAM
    ~4.1GB 1-bit(estimate)
    Details
    Show how to run 
    mlx_lm.generate --model prism-ml/Bonsai-27B-mlx-1bit
    License
    apache-2.0
    Context
    262K
    China access
    需代理

  • POCKET-KR-MLX (FINAL-Bench)

    35B 韩语模型，在 iPhone/Mac 上用 MLX 本地运行

    Params
    35B
    Commercial
    可商用

    Run config FINAL-Bench/POCKET-KR-MLX

    VRAM
    ~12GB 2-bit(estimate)
    Details
    Show how to run 
    mlx_lm.generate --model FINAL-Bench/POCKET-KR-MLX
    License
    Apache-2.0
    China access
    需代理

Common pitfalls

  • 独立编码任务稳定性不如 Qwen 3.5/3.6，容易出现 flaky 行为 — deepreinforce-ai/Ornith-1.0-9B
  • 实际 bug 发现能力弱于同等规模模型，与基准评测表现有落差 — deepreinforce-ai/Ornith-1.0-9B
  • 无工具支持的纯对话场景下幻觉倾向明显 — deepreinforce-ai/Ornith-1.0-9B
  • 在某些配置下会无限重复输出，难以正常完成代码编辑 — EschaLabs/Qwen3.6-35B-A3B-Escha-W2
  • 遇到不熟悉的代码容易出 eager error，几次调用就能破坏企业级代码库 — EschaLabs/Qwen3.6-35B-A3B-Escha-W2
  • 不擅长探索代码库和制定计划，需搭配能消化全量上下文的模型 — EschaLabs/Qwen3.6-35B-A3B-Escha-W2
  • 比 Claude 更挑剔/不稳定，需要较多人工引导 — EschaLabs/Qwen3.6-35B-A3B-Escha-W2
  • 输出非常冗长（very verbose） — EschaLabs/Qwen3.6-35B-A3B-Escha-W2
  • 在同尺寸开源模型中价格偏高 — EschaLabs/Qwen3.6-35B-A3B-Escha-W2
  • 存在使用错误版本（未固定 layers）导致体验差的情况 — EschaLabs/Qwen3.6-35B-A3B-Escha-W2
  • benchmark 在知识和技能维度上明显弱于部分竞品 — EschaLabs/Qwen3.6-35B-A3B-Escha-W2
  • 非 Google 官方发布,为个人社区微调,无官方维护承诺 — Jiunsong/supergemma4-26b-uncensored-mlx-4bit-v2
  • 24GB 统一内存为最低舒适门槛,推荐 32GB+ — Jiunsong/supergemma4-26b-uncensored-mlx-4bit-v2
  • 基准测试数据来自创作者自己的模型卡,需独立验证,不宜全信 — Jiunsong/supergemma4-26b-uncensored-mlx-4bit-v2
  • 仅支持文本任务,不具备多模态能力 — Jiunsong/supergemma4-26b-uncensored-mlx-4bit-v2
  • 需要自行配置 MLX-LM 或 llama.cpp 等本地推理框架 — Jiunsong/supergemma4-26b-uncensored-mlx-4bit-v2
  • LM Studio 的 MLX 后端在反复加载/卸载模型后可能出现性能下降，需重启解决 (1) — mlx-community/Meta-Llama-3-8B-Instruct-4bit
  • LM Studio 的 MLX 后端偶有生成时进入无限循环的问题 (1) — mlx-community/Meta-Llama-3-8B-Instruct-4bit
  • 不擅长精确理解和修改已有代码，容易偏离预期改动 — nathansutton/Ornith-1.0-35B-UD-Q2_K_XL-MLX
  • 在代码移植/对齐场景下可能出现幻觉式修改，把正确的代码改坏 — nathansutton/Ornith-1.0-35B-UD-Q2_K_XL-MLX
  • 部分用户反馈其调试能力（找 bug）表现一般，虽然基准评测分数高 — nathansutton/Ornith-1.0-35B-UD-Q2_K_XL-MLX
  • 基准测试分数随模型规模或量化位宽降低而递减，1.58bit量化在部分任务上相比全精度仍有能力损失 — openbmb/BitCPM-CANN-8B
  • 日常对话质量不及同内存占用的 Qwen 3.6 27B 和 Gemma 4 — prism-ml/Bonsai-27B-gguf
  • 手机端实际运行速度偏慢 — prism-ml/Bonsai-27B-mlx-1bit
  • 底模是 Qwen，核心创新在压缩效率而非模型本身 — prism-ml/Bonsai-27B-mlx-1bit
  • 三元版本在质量对比中不敌同体量传统量化 Q2_K_XL — prism-ml/Bonsai-27B-mlx-1bit
  • 部分用户实测认为其回答质量明显不如 Gemma-4-E2B 等传统精度的 8B 模型 — prism-ml/Bonsai-8B-mlx-1bit
  • MLX 版本的每权重实际比特数为 1.25 bits，高于官方宣称的 1.125 bits，存在额外打包开销 — prism-ml/Bonsai-8B-mlx-1bit
  • 官方声称的「end-to-end 1-bit，无高精度 escape hatch」表述容易引起误解 — prism-ml/Bonsai-8B-mlx-1bit
  • 受 llama.cpp MTP 机制限制:不支持 -np > 1 并行,不支持 --mmproj — Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed
  • 依赖 llama.cpp PR #22673,对推理框架版本有硬性要求 — Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed
  • agentic 编码强但探索/理解整个代码库、制定计划偏弱,这类任务需要搭配能吞下全上下文的模型 — Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed
  • 使用体验比 Claude 更「较劲」,需要逐步人工引导才出彩 — Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed
  • 多模态与 grounded 任务排名靠后:34 个候选模型里排第 24,54.1/100 — Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed
  • IQ4_XS 量化有已知坑:此前 llama.cpp 的 bug 让 IQ4_XS 量化体积偏大,16GB 显存场景建议用 llama-quantize 默认参数、不要加 --pure — Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed
  • The configurations here passed this site's 16GB estimate filter, which is not a guarantee they run. The estimate is a rough figure from parameter count and weight format, not a measurement; actual needs also depend on the runtime, context length and what else is using memory, so leave headroom per the card's suggested VRAM.
  • Discrete VRAM and Mac unified memory are not the same budget: a Mac's 16GB is shared with the system, other apps and the model, and an MLX pack's estimate describes the weights only, so it cannot by itself mean "a 16GB Mac runs it".
  • A model missing from this table is in one of two situations: this site has no trustworthy material on its weight format (the card shows "VRAM: no estimate"), or the estimate exceeds 16GB. Neither means "cannot run". If the repo has a lower-bit quant that this site has not indexed, the table will not guess on its behalf.

FAQ

Does a Mac with 16GB of memory count as 16GB of VRAM?
Not the same thing. This table filters on a site estimate ≤ 16GB, a rough figure for putting the weights into discrete GPU memory; a Mac's unified memory is shared with the system, other apps and the model's context, so the part available to the model is clearly below 16GB. A row with MLX shows that an Apple Silicon weight pack exists, but the same number does not mean a 16GB Mac runs it — leave headroom per the suggested VRAM and try it yourself.
Why are some familiar models missing from the table?
Only two reasons: this site has no trustworthy material on the weight format (the detail page shows "VRAM: no estimate"), or the site estimate exceeds 16GB. Neither is a verdict of "cannot run". For every text-generation model (including unestimated and over-budget ones) see /guide/text/; for just the 16GB tier see /guide/text/16gb/.
What does "This tier is met by" followed by another repo name mean on a card?
The native repo itself did not pass the filter (no estimate, or over budget); it is that specific quant pack (GGUF / MLX) that passed this site's 16GB estimate filter. The name, VRAM estimate and run command then all belong to that pack — download by that ID, and do not apply the number to the native repo's weights.
I have an NVIDIA / AMD discrete GPU — what can I pick here?
Skip rows whose config ID contains MLX (Apple Silicon-only weights; rows whose run command is mlx_lm or mtplx are the same), look at GGUF or native-weight rows, then open "Show how to run" to check whether a command is indexed; rows marked "No command indexed" need the detail page or the model repo for how to run. Most rows here are currently MLX packs, so discrete GPUs have fewer directly usable rows than Macs — not because they cannot run these models, but because many GGUF packs have no site estimate yet (one repo carries several quant levels and this site does not pick one to guess).