Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
seydor
on April 18, 2024
|
parent
|
context
|
favorite
| on:
Meta Llama 3
how much vram does the 8B model use?
cjbprime
on April 18, 2024
|
next
[–]
In general you can swap B for GB (and use the q8 quantization), so 8GB VRAM can probably just about work.
lolinder
on April 18, 2024
|
parent
|
next
[–]
If you want to not quantize at all, you need to double it for fp16—16GB.
cjbprime
on April 19, 2024
|
root
|
parent
|
next
[–]
Yes, but I think it's standard to do inference at q8, not fp16.
derac
on April 18, 2024
|
prev
[–]
You can use 5 bits per parameter with negligible loss of capability as a general rule. 4 bits for a tiny bit worse results. This is subject to changes in how good quantization is in general and on the specific model.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search: