三元1.58-bit量化8B模型，苹果芯片本地推理
- Apple Silicon
- 部署
-
- py mlx_lm.load('prism-ml/Ternary-Bonsai-8B-mlx-2bit')
- py mlx-lm generate --model prism-ml/Ternary-Bonsai-8B-mlx-2bit
本期 (2)
Llama-3-8B 4-bit版，M系列芯片本地推理
- Apple Silicon
- 部署
-
- pip pip install mlx-lm && mlx_lm.generate --model mlx-community/Meta-Llama-3-8B-Instruct-4bit --prompt "hello"
- py from mlx_lm import load, generate; model, tokenizer = load('mlx-community/Meta-Llama-3-8B-Instruct-4bit'); generate(model, tokenizer, prompt='hello')