Guides / Local LLMs for 16GB devices: discrete GPU vs Mac · 中文版

Local LLMs for 16GB devices: discrete GPU vs Mac

For people with a 16GB discrete GPU or a 16GB-memory Mac: the configurations here passed this site's 16GB estimate filter, which is not a guarantee they run; work out which kind of 16GB you have, then pick by run config.

Editorial picks

First work out whether your 16GB is discrete GPU VRAM or Mac unified memory — they are not the same budget; then read each card's "Run config" rather than the model name: MLX packs run only on Apple Silicon. The configurations here passed this site's 16GB estimate filter, which is not a guarantee they run; actual needs also depend on the runtime, context length and what else is using memory. Open "Show how to run" to copy a command; every text model is at /guide/text/.

  • Coding agent; what passed is the MLX 4-bit pack, Apple Silicon only Ornith-1.0-9B (DeepReinforce)

    Configuration that passed the filter: lmstudio-community/Ornith-1.0-9B-MLX-4bit (4bit), roughly ~5.4GB by this site's estimate, not a guarantee that it runs — the name and the run command belong to this package, not to the original repo.

    What passed the filter is the pack lmstudio-community/Ornith-1.0-9B-MLX-4bit: this site's rough estimate is about 5.4GB and the card suggests planning for 7GB — not a measurement. Discrete-GPU users should look at the GGUF packs on the detail page, which this site has not estimated; community feedback puts the Q4_K_M build at 6–16GB of VRAM — a community statement. The material states MIT, commercial use allowed, tool calling. Community feedback is split: usable inside an agent framework, less stable than Qwen 3.5/3.6 for standalone coding; try it on a small scale first.

Within the 16GB budget

24 configurations passed this site's 16GB estimate filter (a rough estimate, not a guarantee that they run). 200 other models for this task have no estimate or exceed the budget and are not listed here — see the full text list. Discrete-GPU VRAM and Mac unified memory are not the same budget: the same estimate does not by itself mean it runs on a Mac with 16GB of shared memory.

  • Mac 上 Qwen3.8-27B 4-bit量化

    Params
    27B
    Commercial
    可商用

    Run config Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed

    VRAM
    ~16GB 4-bit(estimate)
    Details
    Show how to run
    mtplx serve --model Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed
    License
    apache-2.0
    Context
    262k
    China access
    需代理
    Other forms
    原生·youssofal · 原生·youssofal
  • 2.6B 混合架构代理模型,面向端侧工具调用与 128K 长上下文任务

    Params
    2.6B
    Commercial
    需自查

    This tier is met by lmstudio-community/LFM2.5-2.6B-MLX-4bit (4bit)

    VRAM
    ~1.6GB 4-bit(estimate)

    VRAM and command below belong to this config

    Details
    Show how to run
    mlx_lm.generate --model lmstudio-community/LFM2.5-2.6B-MLX-4bit
    License
    LFM Open License v1.0
    Context
    128k
    China access
    需代理
    Other forms
    原生 · GGUF·liquidai · GGUF·lmstudio · 4bit · 8bit
  • Qwen3.6-27B 的 MLX 推理加速版供苹果芯片

    Params
    27B
    Commercial
    可商用

    Run config Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed

    VRAM
    ~16GB 4-bit(estimate)
    Details
    Show how to run
    mtplx serve --model Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed --port 8000
    License
    Apache-2.0
    China access
    需代理
  • Gemma4 26B 无审查 MLX, 本地代理加速

    Params
    26B (4-bit)
    Commercial
    限制商用

    Run config Jiunsong/supergemma4-26b-uncensored-mlx-4bit-v2

    VRAM
    ~16GB 4-bit(estimate)
    Details
    Show how to run
    mlx_lm.server --model Jiunsong/supergemma4-26b-uncensored-mlx-4bit-v2
    License
    gemma
    China access
    需代理
  • 自改进编码代理模型,9B 单卡部署

    Params
    9B
    Commercial
    可商用

    This tier is met by lmstudio-community/Ornith-1.0-9B-MLX-4bit (4bit)

    VRAM
    ~5.4GB 4-bit(estimate)

    VRAM and command below belong to this config

    Details
    Show how to run
    mlx_lm.generate --model lmstudio-community/Ornith-1.0-9B-MLX-4bit
    License
    mit
    Context
    256k
    China access
    需代理
    Other forms
    原生 · GGUF·deepreinforce-ai · GGUF·lmstudio · 4bit · 8bit · GGUF·protolabsai
  • Apple Silicon 的 1-bit LLM,端侧极低内存推理

    Params
    8B
    Commercial
    可商用

    Run config prism-ml/Bonsai-8B-mlx-1bit

    VRAM
    ~1.2GB 1-bit(estimate)
    Details
    No quick-run command

    Run material is on file for this config, but none of it passes the quick-run command check. Follow the config on the detail page or the source repo ↗; commands from other configs are not borrowed.

    License
    Apache-2.0
    Context
    65k
    China access
    需代理
  • Llama-3-8B 4-bit版,M系列芯片本地推理

    Params
    8B
    Commercial
    限制商用

    Run config mlx-community/Meta-Llama-3-8B-Instruct-4bit

    VRAM
    ~4.8GB 4-bit(estimate)
    Details
    Show how to run
    pip install mlx-lm && mlx_lm.generate --model mlx-community/Meta-Llama-3-8B-Instruct-4bit --prompt "hello"
    License
    llama3
    Context
    8k
    China access
    需代理
  • Mac本地Qwen3.8-27B-4bit量化

    Params
    27B
    Commercial
    可商用

    Run config Youssofal/Qwen3.8-27B-MTPLX-Bare-Speed

    VRAM
    ~16GB 4-bit(estimate)
    Details
    Show how to run
    mlx_lm.generate --model Youssofal/Qwen3.8-27B-MTPLX-Bare-Speed
    License
    Apache-2.0
    Context
    262k
    China access
    需代理
  • 2-bit 量化 27B 模型,24GB 显卡 64k 上下文

    Params
    27B
    Commercial
    可商用

    Run config EschaLabs/Qwen3.8-27B-Escha-W2

    VRAM
    ~9.2GB 2-bit(estimate)
    Details
    No run material indexed

    No run material indexed for this config. Follow the config on the detail page or the source repo ↗; commands from other configs are not borrowed.

    License
    Apache-2.0
    Context
    128k
    China access
    需代理
  • 苹果端侧 4-bit MoE 文本生成

    Params
    13.5B
    Commercial
    可商用

    Run config monkiey/StarSupernova

    VRAM
    ~8.1GB 4-bit(estimate)
    Details
    Show how to run
    mlx_lm.generate --model monkiey/StarSupernova
    License
    Apache-2.0
    Context
    131k
    China access
    需代理
  • 决策函数:输入状态+问题+选项,输出校准概率

    Params
    2B
    Commercial
    可商用

    Run config chaoliangUNSW/Jev-Style-Qwen3.5-2B-Decision-MLX-bf16

    VRAM
    ~4.2GB BF16(estimate)
    Details
    Show how to run
    mlx_lm.generate --model chaoliangUNSW/Jev-Style-Qwen3.5-2B-Decision-MLX-bf16
    License
    Apache-2.0
    China access
    需代理
  • Qwen3.8-27B 3-bit 量化,面向长时程 agent

    Params
    27B
    Commercial
    可商用

    Run config orcarouter/OrcaSAQ-2-27B

    VRAM
    ~13GB 3-bit(estimate)
    Details
    No run material indexed

    No run material indexed for this config. Follow the config on the detail page or the source repo ↗; commands from other configs are not borrowed.

    License
    apache-2.0
    Context
    262K
    China access
    需代理
  • 昇腾原生1.58-bit三元LLM,推理内存降6x,保持95.7%性能

    Params
    8B
    Commercial
    可商用

    Run config openbmb/BitCPM-CANN-8B

    VRAM
    ~2.2GB ternary(estimate)
    Details
    No quick-run command

    Run material is on file for this config, but none of it passes the quick-run command check. Follow the config on the detail page or the source repo ↗; commands from other configs are not borrowed.

    License
    apache-2.0
    China access
    需代理
  • 面向硬件音乐设备的本地助手,基于Qwen3-4B微调

    Params
    4B
    Commercial
    不可商用

    Run config zak-raindog/gear-manual-qwen3-4b

    VRAM
    ~2.4GB 4-bit(estimate)
    Details
    Show how to run
    mlx_lm.generate --model zak-raindog/gear-manual-qwen3-4b --prompt "如何使用EP-133?"
    License
    cc-by-nc-4.0
    China access
    需代理
  • Apple Silicon 上 Claude-Code 式本地编码代理

    Params
    35B (3B 活跃)
    Commercial
    需自查

    Run config nathansutton/Ornith-1.0-35B-UD-Q2_K_XL-MLX

    VRAM
    ~12GB 2-bit(estimate)
    Details
    No quick-run command

    Run material is on file for this config, but none of it passes the quick-run command check. Follow the config on the detail page or the source repo ↗; commands from other configs are not borrowed.

    China access
    需代理
  • 27B 1-bit 量化模型,面向笔记本与手机本地推理

    Params
    27B
    Commercial
    可商用

    Run config prism-ml/Bonsai-27B-gguf

    VRAM
    ~4.1GB 1-bit(estimate)
    Details
    Show how to run (+1 more)
    ollama run hf.co/prism-ml/Bonsai-27B-gguf

    1 more way(s) to run on the detail page.

    License
    Apache-2.0
    Context
    262K
    China access
    需代理
    Other forms
    GGUF·prism-ml · GGUF·lmstudio
  • 20B-A1B 三元推理模型,M4 218 tok/s

    Params
    20B-A1B
    Commercial
    可商用

    Run config deepgrove/maple-preview

    VRAM
    ~5.4GB ternary(estimate)
    Details
    No run material indexed

    No run material indexed for this config. Follow the config on the detail page or the source repo ↗; commands from other configs are not borrowed.

    License
    MIT
    Context
    131k
    China access
    需代理
  • Atlassian 开发专用 27B MLX 模型

    Params
    27B
    Commercial
    可商用

    This tier is met by Mihai-LeanZero/Qwen3.8-27B-Atlassian-Q4-mlx (MLX·mihai-leanzero)

    VRAM
    ~16GB 4-bit(estimate)

    VRAM and command below belong to this config

    Details
    Show how to run
    mlx_lm.generate --model Mihai-LeanZero/Qwen3.8-27B-Atlassian-Q4-mlx --prompt "Which Forge module adds a panel to the Jira issue view?"
    License
    Apache-2.0
    Context
    128k
    China access
    需代理
    Other forms
    MLX·mihai-leanzero · MLX·mihai-leanzero
  • IBM Granite 3B MLX 量化,苹果芯片推理

    Params
    3B
    Commercial
    可商用

    Run config lmstudio-community/granite-4.2-3b-MLX-8bit

    VRAM
    ~3.3GB 8-bit(estimate)
    Details
    Show how to run
    mlx_lm.generate --model lmstudio-community/granite-4.2-3b-MLX-8bit
    License
    apache-2.0
    China access
    需代理
    Other forms
    8bit · GGUF · 4bit
  • Forge/Jira 开发辅助 4-bit MLX

    Params
    9B
    Commercial
    可商用

    Run config Mihai-LeanZero/Qwen3.5-9B-Atlassian-Q4-mlx

    VRAM
    ~5.4GB 4-bit(estimate)
    Details
    Show how to run
    mlx_lm.generate --model Mihai-LeanZero/Qwen3.5-9B-Atlassian-Q4-mlx --prompt "Which Forge module adds a panel to the Jira issue view?"
    License
    Apache-2.0
    Context
    32k
    China access
    需代理
    Other forms
    MLX·mihai-leanzero · MLX·mihai-leanzero
  • 2-bit Qwen3.6-35B-A3B,12.3GB,单卡本地推理

    Params
    35B (3B active)
    Commercial
    可商用

    Run config EschaLabs/Qwen3.6-35B-A3B-Escha-W2

    VRAM
    ~12GB 2-bit(estimate)
    Details
    No run material indexed

    No run material indexed for this config. Follow the config on the detail page or the source repo ↗; commands from other configs are not borrowed.

    License
    Apache-2.0
    Context
    128k
    China access
    需代理
  • 7.89GB 27B GGUF,工具调用/Agent

    Params
    27B
    Commercial
    可商用

    Run config ConwayResearch/Underdog-Saluki-27B-1.0

    VRAM
    ~9.2GB 2-bit(estimate)
    Details
    Show how to run (+1 more)
    ollama run hf.co/ConwayResearch/Underdog-Saluki-27B-1.0

    1 more way(s) to run on the detail page.

    License
    apache-2.0
    Context
    32k
    China access
    需代理
  • 1-bit 27B 模型, 可在手机和笔记本本地运行推理

    Params
    27B
    Commercial
    可商用

    Run config prism-ml/Bonsai-27B-mlx-1bit

    VRAM
    ~4.1GB 1-bit(estimate)
    Details
    Show how to run
    mlx_lm.generate --model prism-ml/Bonsai-27B-mlx-1bit
    License
    apache-2.0
    Context
    262K
    China access
    需代理
  • 35B 韩语模型,在 iPhone/Mac 上用 MLX 本地运行

    Params
    35B
    Commercial
    可商用

    Run config FINAL-Bench/POCKET-KR-MLX

    VRAM
    ~12GB 2-bit(estimate)
    Details
    Show how to run
    mlx_lm.generate --model FINAL-Bench/POCKET-KR-MLX
    License
    Apache-2.0
    China access
    需代理

Common pitfalls

  • 独立编码任务稳定性不如 Qwen 3.5/3.6,容易出现 flaky 行为 — deepreinforce-ai/Ornith-1.0-9B
  • 实际 bug 发现能力弱于同等规模模型,与基准评测表现有落差 — deepreinforce-ai/Ornith-1.0-9B
  • 无工具支持的纯对话场景下幻觉倾向明显 — deepreinforce-ai/Ornith-1.0-9B
  • 在某些配置下会无限重复输出,难以正常完成代码编辑 — EschaLabs/Qwen3.6-35B-A3B-Escha-W2
  • 遇到不熟悉的代码容易出 eager error,几次调用就能破坏企业级代码库 — EschaLabs/Qwen3.6-35B-A3B-Escha-W2
  • 不擅长探索代码库和制定计划,需搭配能消化全量上下文的模型 — EschaLabs/Qwen3.6-35B-A3B-Escha-W2
  • 比 Claude 更挑剔/不稳定,需要较多人工引导 — EschaLabs/Qwen3.6-35B-A3B-Escha-W2
  • 输出非常冗长(very verbose) — EschaLabs/Qwen3.6-35B-A3B-Escha-W2
  • 在同尺寸开源模型中价格偏高 — EschaLabs/Qwen3.6-35B-A3B-Escha-W2
  • 存在使用错误版本(未固定 layers)导致体验差的情况 — EschaLabs/Qwen3.6-35B-A3B-Escha-W2
  • benchmark 在知识和技能维度上明显弱于部分竞品 — EschaLabs/Qwen3.6-35B-A3B-Escha-W2
  • 非 Google 官方发布,为个人社区微调,无官方维护承诺 — Jiunsong/supergemma4-26b-uncensored-mlx-4bit-v2
  • 24GB 统一内存为最低舒适门槛,推荐 32GB+ — Jiunsong/supergemma4-26b-uncensored-mlx-4bit-v2
  • 基准测试数据来自创作者自己的模型卡,需独立验证,不宜全信 — Jiunsong/supergemma4-26b-uncensored-mlx-4bit-v2
  • 仅支持文本任务,不具备多模态能力 — Jiunsong/supergemma4-26b-uncensored-mlx-4bit-v2
  • 需要自行配置 MLX-LM 或 llama.cpp 等本地推理框架 — Jiunsong/supergemma4-26b-uncensored-mlx-4bit-v2
  • LM Studio 的 MLX 后端在反复加载/卸载模型后可能出现性能下降,需重启解决 (1) — mlx-community/Meta-Llama-3-8B-Instruct-4bit
  • LM Studio 的 MLX 后端偶有生成时进入无限循环的问题 (1) — mlx-community/Meta-Llama-3-8B-Instruct-4bit
  • 不擅长精确理解和修改已有代码,容易偏离预期改动 — nathansutton/Ornith-1.0-35B-UD-Q2_K_XL-MLX
  • 在代码移植/对齐场景下可能出现幻觉式修改,把正确的代码改坏 — nathansutton/Ornith-1.0-35B-UD-Q2_K_XL-MLX
  • 部分用户反馈其调试能力(找 bug)表现一般,虽然基准评测分数高 — nathansutton/Ornith-1.0-35B-UD-Q2_K_XL-MLX
  • 基准测试分数随模型规模或量化位宽降低而递减,1.58bit量化在部分任务上相比全精度仍有能力损失 — openbmb/BitCPM-CANN-8B
  • 日常对话质量不及同内存占用的 Qwen 3.6 27B 和 Gemma 4 — prism-ml/Bonsai-27B-gguf
  • 手机端实际运行速度偏慢 — prism-ml/Bonsai-27B-mlx-1bit
  • 底模是 Qwen,核心创新在压缩效率而非模型本身 — prism-ml/Bonsai-27B-mlx-1bit
  • 三元版本在质量对比中不敌同体量传统量化 Q2_K_XL — prism-ml/Bonsai-27B-mlx-1bit
  • 部分用户实测认为其回答质量明显不如 Gemma-4-E2B 等传统精度的 8B 模型 — prism-ml/Bonsai-8B-mlx-1bit
  • MLX 版本的每权重实际比特数为 1.25 bits,高于官方宣称的 1.125 bits,存在额外打包开销 — prism-ml/Bonsai-8B-mlx-1bit
  • 官方声称的「end-to-end 1-bit,无高精度 escape hatch」表述容易引起误解 — prism-ml/Bonsai-8B-mlx-1bit
  • 受 llama.cpp MTP 机制限制:不支持 -np > 1 并行,不支持 --mmproj — Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed
  • 依赖 llama.cpp PR #22673,对推理框架版本有硬性要求 — Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed
  • agentic 编码强但探索/理解整个代码库、制定计划偏弱,这类任务需要搭配能吞下全上下文的模型 — Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed
  • 使用体验比 Claude 更「较劲」,需要逐步人工引导才出彩 — Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed
  • 多模态与 grounded 任务排名靠后:34 个候选模型里排第 24,54.1/100 — Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed
  • IQ4_XS 量化有已知坑:此前 llama.cpp 的 bug 让 IQ4_XS 量化体积偏大,16GB 显存场景建议用 llama-quantize 默认参数、不要加 --pure — Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed
  • The configurations here passed this site's 16GB estimate filter, which is not a guarantee they run. The estimate is a rough figure from parameter count and weight format, not a measurement; actual needs also depend on the runtime, context length and what else is using memory, so leave headroom per the card's suggested VRAM.
  • Discrete VRAM and Mac unified memory are not the same budget: a Mac's 16GB is shared with the system, other apps and the model, and an MLX pack's estimate describes the weights only, so it cannot by itself mean "a 16GB Mac runs it".
  • A model missing from this table is in one of two situations: this site has no trustworthy material on its weight format (the card shows "VRAM: no estimate"), or the estimate exceeds 16GB. Neither means "cannot run". If the repo has a lower-bit quant that this site has not indexed, the table will not guess on its behalf.

FAQ

Does a Mac with 16GB of memory count as 16GB of VRAM?
Not the same thing. This table filters on a site estimate ≤ 16GB, a rough figure for putting the weights into discrete GPU memory; a Mac's unified memory is shared with the system, other apps and the model's context, so the part available to the model is clearly below 16GB. A row with MLX shows that an Apple Silicon weight pack exists, but the same number does not mean a 16GB Mac runs it — leave headroom per the suggested VRAM and try it yourself.
Why are some familiar models missing from the table?
Only two reasons: this site has no trustworthy material on the weight format (the detail page shows "VRAM: no estimate"), or the site estimate exceeds 16GB. Neither is a verdict of "cannot run". For every text-generation model (including unestimated and over-budget ones) see /guide/text/; for just the 16GB tier see /guide/text/16gb/.
What does "This tier is met by" followed by another repo name mean on a card?
The native repo itself did not pass the filter (no estimate, or over budget); it is that specific quant pack (GGUF / MLX) that passed this site's 16GB estimate filter. The name, VRAM estimate and run command then all belong to that pack — download by that ID, and do not apply the number to the native repo's weights.
I have an NVIDIA / AMD discrete GPU — what can I pick here?
Skip rows whose config ID contains MLX (Apple Silicon-only weights; rows whose run command is mlx_lm or mtplx are the same), look at GGUF or native-weight rows, then open "Show how to run" to check whether a command is indexed; rows marked "No command indexed" need the detail page or the model repo for how to run. Most rows here are currently MLX packs, so discrete GPUs have fewer directly usable rows than Macs — not because they cannot run these models, but because many GGUF packs have no site estimate yet (one repo carries several quant levels and this site does not pick one to guess).