Hosting LLMs

 LOCAL LLM

Using AI to set up a Linux environment for hosting a Large Language Model. 

I knew it could be done, seemed like a daunting undertaking. I also knew things are moving fast, and hardware for such things was becoming more prevelatnt. 

I bought a machine. A Framework desktop with the latest AMD processor with 128GB LPDDR5x-8000 memory. When it arrived it took about a day to install some drives, get Ubuntu installed and logged in with a SSH connection. Using with help from Gemini and Perplexity I had Ollama installed, some small models downloaded, and started chatting with Open WebUI and AnythingLLM. 

Then I discovered Donato Capitella and his repository at GitHub. He also does YouTube videos about all things tech, and has been diving deep into LLMs. He's benchmarked many models on the Framework computer that I bought for this project, so he's done much of the work of optimizing the hardware, and of course shared it at the link above.

I have been pushing the limits of LLMs for months now, so rather than follow his guide I decided to test how much further one could get with the help of AI, letting an LLM do all the driving. I have Ubuntu installed as mentioned, but the guide above assumes a Fedora installation. He refers anyone interested in Ubuntu to a repository by one Pablo Ross. I uploaded the markdown files from there to a Google NotebookLM, then had it generate a guide. 

Started up Claude, uploaded this guide, then used this prompt, "Use this document as a guide for deploying LLMs on my machine. Help, step by step. When i execute the commands, I want you to analyze the output then guide the setup through the end, when we have a working LLM locally."

And so we began. When it was done, later that day, I was querying a small model, using the resources optimized using the suggestions based on Donato's guide.

Only a year ago it would have taken me months to get that far. AI can truly be useful for such projects. I deliberately used the commands suggested by Claude, looking into the reasons for them, but refrained from deviating from the process put forth. It shows that anyone with limited knowledge and experience can deploy LLMs, and probably other such projects much faster than was possible before OpenAI made it relatively popular.