The Tesla V100 16GB only has 16GB VRAM. Claude 3.5 Sonnet needs ~106.5GB (estimate). Time to rent or upgrade.
Estimate · Q4_K_M · 4k context · batch 1. For dense 7B–70B this sits ~5–8% above the Q4_K_M GGUF file. Not peak runtime VRAM and not a measured bench.
Need more VRAM?
Rent cloud GPUs with 106.5GB+ VRAM to run Claude 3.5 Sonnet, or upgrade hardware.
This GPU is short by ~90.5GB VRAM for Claude 3.5 Sonnet at Q4.
We may earn a commission from cloud and hardware partners at no extra cost to you.
MSRP: $8,000