课程: Large Language Models on AWS: Building and Deploying Open-Source LLMs

免费学习该课程！

今天就开通帐号，24,700 门业界名师课程任您挑！

GGUF quantized llama.cpp end-to-end demo

GGUF quantized llama.cpp end-to-end demo

课程: Large Language Models on AWS: Building and Deploying Open-Source LLMs

开始一个月试用 购买团队方案

GGUF quantized llama.cpp end-to-end demo

“

- [Instructor] It can be a little bit confusing about how to use a research model in a local environment. You may have heard about someone fine tuning some kind of foundation model, but then you were like, "Wait, I can't run this." And even tools like, Ollama or Llamafile don't have access to it. So, what do you do? Well, you need to use llama.cpp. So first up here, what are we going to do? We're going to make sure that we have UV installed. So, the way you would do this is you would actually run this command right here, which is uv installer. We can go ahead and run it. You can see, oh, it's already installed. It's a very tiny utility that is going to help us with a lot of stuff. Next step, we also would want to make sure we're cloning llama.cpp. In this case, if we go ahead and we say git remote -v, we can see it's also there, right? So we would make in terms of our CP architecture, also set GGML_CUDA flag. And I also say, "Hey, go ahead and spawn a bunch of threads here." And what…

内容