[u-u] Specialized hardware?

[D. Hugh Redelmeier]

> From: Iain Calder <icalder-fcd99 at torfree.net>
> 
> arocker wrote:
> >
> > Does anyone know how well crypto mining rigs might be suited for
> > conversion into AI machines?
> >
> > My suspicion is that crypto mining wouldn't require the graphics cards
> > that AI uses for vector calculations
> 

> Competitive pressure pushed most crypto-mining off graphics cards,
> over a decade ago.  A few currencies chose algorithms specifically
> designed to not be acceleratable by a custom pipeline.  Rigs for
> *those* currencies do use graphics cards.

Adding to this (but I'm no expert):

What mainstream LLMs require is a lot of memory for weights. They also 
need lots of computing with high bandwidth access to the weights. That 
means that normal graphics cards are not very useful, especially old ones.

VRAM is expensive and in short supply.

On the other hand, small models have their uses and some can fit in quite 
small machines.

Oh, and there is a big difference between "training" and "inferencing" 
(using) an LLM.  Cutting edge models are trained by enormous amounts of 
hardware.  Beyond all but a handful of companies.

That being said, one could use lesser hardware if one didn't need to 
produce competitive results.

You can slowly run "frontier" models on the right microcomputers.  For 
example, the Nvidia Sparx (128G of RAM), a Mac Mini Pro with maxed-out 
memory (128G, I think), or various AMD-based models like the FrameWork 
desktop (again, typically with 128G of RAM).  Each of these has "unified 
memory", i.e. the integrated GPU can freely access main memory at high 
bandwidth.

This is a large rabbit hole, so I'll turn back here.