Loading...

Oh, that's for sure, but yeah, you're right. So I'm really wanting to run my own local model, at least play with it, right? And that's kind of why I'm being, that's why I'm buying this. I mean, I guess I made up my mind. That's probably why, my wife's laughing at me. Oh, okay. I mean, I'm buying that system. But on that system, from what I've been able to figure out, I mean, I can run it, I could easily run a 7 billion model. I could probably run 13 million quantized pretty decently. Anything above that may be slow. I'd never get to 70 billion on this setup. However, as I was doing some research on it, really, I mean, if I really wanted to go up and make it a real, real big powerhouse, I mean, all I'd have to do is change out my VRAM GPU. I mean, it's going to come with, what is it, what does it have, the 4070, I think, is coming with it. Oh, it's a 5070, yeah, the 5070 is coming on it, but you can always upgrade it. Of course, that's the expensive part, right? If I went, let's say I just instantly wanted to upgrade it, and I went with the RTX 4090, it's going to cost between $1,200 and $1,800 just for that card. Of course, the RAM upgrades, I mean, that's cheap. I can throw a couple hundred dollars of it at it and go from from 32 to 64 gig RAM really quickly So that's probably the first thing I do it. Oh, yeah, dude, like the video card you're paying for the video card You're getting the rest of the computer for free right now It's a no-brainer And I don't worry about running bigger models, man I run a really small model on an orange pi 5 with 16 gigs of RAM And it's got a neural processor built into it but it runs, it runs a little slow but it runs the models no problem. I just run small models that are purpose-built. You don't need to run a giant model unless you're doing giant things. You know, for scripting I've got a model that knows PHP, Python, Bash, a couple of other languages that I can't remember. But it's only, I think it's a three billion, You don't need a huge model. You just get purpose-built models, that's all.