this post was submitted on 07 Feb 2024
190 points (95.2% liked)

Technology

59438 readers
3092 users here now

This is a most excellent place for technology news and articles.


Our Rules


  1. Follow the lemmy.world rules.
  2. Only tech related content.
  3. Be excellent to each another!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, to ask if your bot can be added please contact us.
  9. Check for duplicates before posting, duplicates may be removed

Approved Bots


founded 1 year ago
MODERATORS
 

Abacus.ai:

We recently released Smaug-72B-v0.1 which has taken first place on the Open LLM Leaderboard by HuggingFace. It is the first open-source model to have an average score more than 80.

you are viewing a single comment's thread
view the rest of the comments
[–] simple@lemm.ee 45 points 9 months ago (5 children)

I'm afraid to even ask for the minimum specs on this thing, open source models have gotten so big lately

[–] General_Effort@lemmy.world 18 points 9 months ago

CUDA 11.4 and above are recommended (this is for GPU users, flash-attention users, etc.) To run Qwen-72B-Chat in bf16/fp16, at least 144GB GPU memory is required (e.g., 2xA100-80G or 5xV100-32G). To run it in int4, at least 48GB GPU memory is requred (e.g., 1xA100-80G or 2xV100-32G).

It's derived from Qwen-72B, so same specs. Q2 clocks it in at only ~30GB.

[–] SinningStromgald@lemmy.world 11 points 9 months ago

Just a data center or two. Easy peasy dirt cheapy.

[–] girsaysdoom@sh.itjust.works 5 points 9 months ago (2 children)

I think I read somewhere that you'll basically need 130 GB of RAM to load this model. You could probably get some used server hardware for less than $600 to run this.

[–] cm0002@lemmy.world 15 points 9 months ago (2 children)

Oh if only it were so simple lmao, you need ~130GB of VRAM, aka the graphics card RAM. So you would need about 9 consumer grade 16GB graphics cards and you'll probably need Nvidia because of fucking CUDA so we're talking about thousands of dollars. Probably approaching 10k

Ofc you can get cards with more VRAM per card, but not in the consumer segment so even more $$$$$$

[–] kakes@sh.itjust.works 9 points 9 months ago

Afaik you can substitute VRAM with RAM at the cost of speed. Not exactly sure how that speed loss correlates to the sheer size of these models, though. I have to imagine it would run insanely slow on a CPU.

[–] girsaysdoom@sh.itjust.works 0 points 9 months ago (1 children)

I'm pretty sure you can load the model using RAM like another poster said. Here's a used server under $600 that could theoretically run it: ebay.

[–] brick@lemm.ee 5 points 9 months ago

You would want to look for an R730, which can be had for not too much more. The 20 series was the “end of an era” and the 30 series was the beginning of the next era. Most importantly for this application, R30s use DDR4 whereas R20s use DDR3.

RAM speed matters a lot for ML applications and DDR4 is about 2x as fast as DDR3 in all relevant measurements.

If you’re going to offload any part of these models to CPU, which you 99.99% will have to do for a model of this size with this class of hardware, skip the 20s and go to the 30s.

[–] ArchAengelus@lemmy.dbzer0.com 10 points 9 months ago

Unless you’re getting used datacenter grade hardware for next to free, I doubt this. You need 130 gb of VRAM on your GPUs

[–] L_Acacia@lemmy.one 3 points 9 months ago

Around 48gb of VRAM if you want to run it in 4bits

[–] TheChurn@kbin.social 0 points 9 months ago

Every billion parameters needs about 2 GB of VRAM - if using bfloat16 representation. 16 bits per parameter, 8 bits per byte -> 2 bytes per parameter.

1 billion parameters ~ 2 Billion bytes ~ 2 GB.

From the name, this model has 72 Billion parameters, so ~144 GB of VRAM