Because you can just quantise the 70B model to 3-4 bits and it'll perform better... | Hacker News

Hacker Timesnew | past | comments | ask | show | jobs | submit

		logicchains on July 24, 2024 \| parent \| context \| favorite \| on: Large Enough Because you can just quantise the 70B model to 3-4 bits and it'll perform better than a 30B model but be a similar size.

novok on July 24, 2024 [–]

A 70B 4bit model does not fit in a 24GB VRAM card, 30B models are the sweet spot for that size of card at 20GB, with 4GB left for the system to still function.

Consider applying for YC's Summer 2026 batch! Applications are open till May 4
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact