Llama Corp

llamacorp

Welcome to this ”to-the-point” tutorial on how to quantize any Large Language Model (LLM) available on Hugging Face using llama.cpp. Whether you're a data scientist, a machine learning engineer, or simply an AI enthusiast, this guide is designed to clarify the process of model quantization and make it easy. By the end of this tutorial, you'll have a clear understanding of how to efficiently compress LLMs without significant loss in performance, enabling their deployment on resource-constrained environments. You can also use these model on your fav local setups using Ollama! GitGub Repo is here: https://github.com/mickymultani/QuantizeLLMs What is Model Quantization? Before we dive into the technicalities, let's briefly discuss what model quantization is and why it's important. Model quantization is a technique used to reduce the precision of the numbers used in a model's weights and activations. This process significantly reduces the model size and speeds up inference times, making it possible to deploy state-of-the-art models on devices with limited memory and computational power, such as mobile phones and embedded systems. Introducing llama.cpp llama.cpp is a powerful tool that facilitates the quantization of LLMs. It supports various quantization methods, making it highly versatile for different use cases. The tool is designed to work seamlessly with models from the Hugging Face Hub, which hosts a wide range of pre-trained models across various languages and domains. Setting Up Your Workspace This tutorial is designed with flexibility in mind, catering to both cloud-based environments and local setups. I personally conducted these steps on Google Colab, utilizing the NVIDIA Tesla T4 GPU, which provides a robust platform for model quantization and testing. However, you should not feel limited to this setup. The beauty of llama.cpp and the techniques covered in this guide is their adaptability to various environments, including local machines with GPU support. For instance, if you're using a MacBook with Apple Silicon, you can follow along and leverage the GPU support for model quantization, thanks to the cross-platform compatibility of the tools and libraries we are using. Setting Up on Google Colab Google Colab provides a convenient, cloud-based environment with access to powerful GPUs like the T4. If you choose Colab for this tutorial, make sure to select a GPU runtime by going to . This ensures that your notebook has access to the necessary computational resources. Runtime > Change runtime type > T4 GPU Running on MacBook with Apple Silicon For those opting to run the quantization process on a MacBook with Apple Silicon, ensure that you have the necessary development tools and libraries installed. While the setup might slightly differ from the Linux-based Colab environment, Python's ecosystem and the compilation of llama.cpp with CUDA support (or the equivalent for your specific GPU architecture) are well-documented, ensuring a smooth setup process. Setting Up Hugging Face Authentication Regardless of your platform, access to models from the Hugging Face Hub requires authentication. To seamlessly integrate this in your workflow, especially when working in collaborative or cloud-based environments like Google Colab, it's advisable to set up your Hugging Face authentication token securely. On Google Colab, you can safely store your Hugging Face token by using Colab's "Secrets" feature. This can be done by clicking on the "Key" icon in the sidebar, selecting "Secrets", and adding a new secret with the name and your Hugging Face token as the value. This method ensures that your token remains secure and is not exposed in your notebook's code. HF_TOKEN For local setups, consider setting the environment variable in your shell or utilizing the Hugging Face CLI to log in, thereby ensuring that your scripts have the necessary permissions to download and upload models to the Hub without hard-coding your credentials. HF_TOKEN Step-by-Step Tutorial Now, let's walk through the process of quantizing a model using llama.cpp. For this tutorial, we'll quantize the " " model from Hugging Face, but the steps can be applied to any model of your choice. google/gemma-2b-it 1. Setting Up Your Environment First, we need to clone the llama.cpp repository and install the necessary requirements: !git clone https://github.com/ggerganov/llama.cpp
!cd llama.cpp && LLAMA_CUBLAS=1 make && pip install -r requirements.txt This command clones the llama.cpp repository and compiles the necessary binaries with CUDA support for GPU acceleration. It also installs Python dependencies required for the process. 2. Downloading the Model Next, we download the model from Hugging Face Hub using the function. This function ensures that we have a local copy of the model for quantization: snapshot_download from huggingface_hub import snapshot_download

model_name = "google/gemma-2b-it"
base_model = "./original_model/"
snapshot_download(repo_id=model_name, local_dir=base_model, local_dir_use_symlinks=False) 3. Preparing the Model for Quantization Before quantizing, we convert the downloaded model to a format compatible with llama.cpp ( ). This step involves specifying the desired precision ( for half-precision floating point) and the output file: gguf f16 !mkdir ./quantized_model/
!python llama.cpp/convert-hf-to-gguf.py ./original_model/ --outtype f16 --outfile ./quantized_model/FP16.gguf 4. Quantizing the Model With the model in the correct format, we proceed to the quantization step. Here, we're using a quantization method called , which is specified in the list. This method quantizes the model to 4-bit precision with knowledge distillation and mapping techniques for better performance: q4_k_m methods import os

methods = ["q4_k_m"]
quantized_path = "./quantized_model/"

for m in methods:
    qtype = f"{quantized_path}/{m.upper()}.gguf"
    os.system("./llama.cpp/quantize "+quantized_path+"/FP16.gguf "+qtype+" "+m) 5. Testing the Quantized Model After quantization, it's important to test the model to ensure it performs as expected. The following command runs the quantized model with a sample prompt from a file, allowing you to assess its output quality: ! ./llama.cpp/main -m ./quantized_model/Q4_K_M.gguf -n 90 --repeat_penalty 1.0 --color -i -r "User: " -f llama.cpp/prompts/chat-with-bob.txt 6. Sharing Your Quantized Model Finally, if you wish to share your quantized model with the community, you can upload it to Hugging Face Hub using the and functions. This step involves creating a new repository for your quantized model and uploading the file: HfApi upload_file .gguf from huggingface_hub import HfApi, create_repo, upload_file

model_path = "./quantized_model/Q4_K_M.gguf"
repo_name = "gemma-2b-it-GGUF-quantized"
repo_url = create_repo(repo_name, private=False)

api = HfApi()
api.upload_file(
    path_or_fileobj=model_path,
    path_in_repo="Q4_K_M.gguf",
    repo_id="yourusername/gemma-2b-it-GGUF-quantized",
    repo_type="model",
) Make sure to replace with your actual Hugging Face username! "yourusername" Wrapping Up Congratulations! You've just learned how to quantize a large language model using llama.cpp. This process not only helps in deploying models to resource-constrained environments but also in reducing computational costs for inference. By sharing your quantized models, you contribute to a growing ecosystem of efficient AI models accessible to a broader audience. Quantization is a powerful tool in the machine learning practitioner's toolkit. Happy quantizing!

Quantizing Large Language Models With llama.cpp: A Clean Guide for 2024

11 Best AI Chat Tools for Developers in 2024

Improving Text-to-SQL with a Fine-Tuned 7B LLM for DB Interactions

How to Run Your Own Local LLM: Updated for 2024 - Version 2

After my latest post about how to build your own RAG and run it locally, today, we're taking it a step further by not only implementing the conversational abilities of large language models but also adding listening and speaking capabilities. The idea is straightforward: we are going to create a voice assistant reminiscent of Jarvis or Friday from the iconic Iron Man movies, which can operate offline on your computer. Since this is an introductory tutorial, I will implement it in Python and keep it simple enough for beginners. Lastly, I will provide some guidance on how to scale the application. Techstack First, you should set up a virtual Python environment. You have several options for this, including pyenv, virtualenv, poetry, and others that serve a similar purpose. Personally, I'll use Poetry for this tutorial due to my personal preferences. Here are several crucial libraries you'll need to install: : For a visually appealing console output. rich : A robust tool for speech-to-text conversion. openai-whisper : A cutting-edge library for text-to-speech synthesis, ensuring high-quality audio output. suno-bark : A straightforward library for interfacing with Large Language Models (LLMs). langchain , , and : Essential for audio recording and playback. sounddevice pyaudio speechrecognition For a detailed list of dependencies, refer to the link . here The most critical component here is the Large Language Model (LLM) backend, for which we will use Ollama. is widely recognized as a popular tool for running and serving LLMs offline. If Ollama is new to you, I recommend checking out my previous article on offline RAG: Basically, you just need to download the Ollama application, pull your preferred model, and run it. Ollama "Build Your Own RAG and Run It Locally: Langchain + Ollama + Streamlit." Architecture Okay, if everything has been set up, let's proceed to the next step. Below is the overall architecture of our application, which fundamentally comprises 3 main components: : Utilizing , we convert spoken language into text. Whisper's training on diverse datasets ensures its proficiency across various languages and dialects. Speech Recognition OpenAI's Whisper : For the conversational capabilities, we'll employ the Langchain interface for the model, which is served using Ollama. This setup promises a seamless and engaging conversational flow. Conversational Chain Llama-2 : The transformation of text to speech is achieved through , a state-of-the-art model from Suno AI, renowned for its lifelike speech production. Speech Synthesizer Bark The workflow is straightforward: record speech, transcribe to text, generate a response using an LLM, and vocalize the response using Bark. Sequence diagram for voice assistant with Whisper, Ollama, and Bark. Implementation The implementation begins with crafting a based on Bark, incorporating methods for synthesizing speech from text and handling longer text inputs seamlessly as follows: TextToSpeechService import nltk
import torch
import warnings
import numpy as np
from transformers import AutoProcessor, BarkModel

warnings.filterwarnings(
    "ignore",
    message="torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.",
)


class TextToSpeechService:
    def __init__(self, device: str = "cuda" if torch.cuda.is_available() else "cpu"):
        """
        Initializes the TextToSpeechService class.
        Args:
            device (str, optional): The device to be used for the model, either "cuda" if a GPU is available or "cpu".
            Defaults to "cuda" if available, otherwise "cpu".
        """
        self.device = device
        self.processor = AutoProcessor.from_pretrained("suno/bark-small")
        self.model = BarkModel.from_pretrained("suno/bark-small")
        self.model.to(self.device)

    def synthesize(self, text: str, voice_preset: str = "v2/en_speaker_1"):
        """
        Synthesizes audio from the given text using the specified voice preset.
        Args:
            text (str): The input text to be synthesized.
            voice_preset (str, optional): The voice preset to be used for the synthesis. Defaults to "v2/en_speaker_1".
        Returns:
            tuple: A tuple containing the sample rate and the generated audio array.
        """
        inputs = self.processor(text, voice_preset=voice_preset, return_tensors="pt")
        inputs = {k: v.to(self.device) for k, v in inputs.items()}

        with torch.no_grad():
            audio_array = self.model.generate(**inputs, pad_token_id=10000)

        audio_array = audio_array.cpu().numpy().squeeze()
        sample_rate = self.model.generation_config.sample_rate
        return sample_rate, audio_array

    def long_form_synthesize(self, text: str, voice_preset: str = "v2/en_speaker_1"):
        """
        Synthesizes audio from the given long-form text using the specified voice preset.
        Args:
            text (str): The input text to be synthesized.
            voice_preset (str, optional): The voice preset to be used for the synthesis. Defaults to "v2/en_speaker_1".
        Returns:
            tuple: A tuple containing the sample rate and the generated audio array.
        """
        pieces = []
        sentences = nltk.sent_tokenize(text)
        silence = np.zeros(int(0.25 * self.model.generation_config.sample_rate))

        for sent in sentences:
            sample_rate, audio_array = self.synthesize(sent, voice_preset)
            pieces += [audio_array, silence.copy()]

        return self.model.generation_config.sample_rate, np.concatenate(pieces) : The class takes an optional parameter, which specifies the device to be used for the model (either if a GPU is available, or ). It loads the Bark model and the corresponding processor from the pre-trained model. You can also use the large version by specifying for the model loader. Initialization ( __init__ ) device cuda cpu suno/bark-small suno/bark : This method takes a input and a parameter, which specifies the voice to be used for the synthesis. You can check out other value . It uses the to prepare the input text and the voice preset, and then generates the audio array using the method. The generated audio array is converted to a NumPy array and the sample rate is returned along with the audio array. Synthesize ( synthesize ) text voice_preset voice_preset here processor model.generate() : This method is used for synthesizing longer text inputs. It first tokenizes the input text into sentences using the function. For each sentence, it calls the method to generate the audio array. It then concatenates the generated audio arrays, with a short silence (0.25 seconds) added between each sentence. Long-form Synthesize ( long_form_synthesize ) nltk.sent_tokenize synthesize Now that we have the set up, we need to prepare the Ollama server for the large language model (LLM) serving. To do this, you'll need to follow these steps: TextToSpeechService : Run the following command to download the latest Llama-2 model from the Ollama repository: Pull the latest Llama-2 model ollama pull llama2 . : If the server is not yet started, execute the following command to start it: . Start the Ollama server ollama serve Once you've completed these steps, your application will be able to use the Ollama server and the Llama-2 model to generate responses to user input. Next, we'll move to the main application logic. First, we need to initialize the following components: : We'll use the Rich library to create a better interactive console for the user within the terminal. Rich Console : We'll initialize a Whisper speech recognition model which is a state-of-the-art open-source speech recognition system developed by OpenAI. We'll use the base English model ( ) for transcribing user input. Whisper Speech-to-Text base.en : We'll initialize a Bark text-to-speech synthesizer instance, which was implemented above. Bark Text-to-Speech : We'll use the built-in from the Langchain library which provides a template for managing the conversational flow. We'll configure it to use the Llama-2 language model with the Ollama backend. Conversational Chain ConversationalChain import time
import threading
import numpy as np
import whisper
import sounddevice as sd
from queue import Queue
from rich.console import Console
from langchain.memory import ConversationBufferMemory
from langchain.chains import ConversationChain
from langchain.prompts import PromptTemplate
from langchain_community.llms import Ollama
from tts import TextToSpeechService

console = Console()
stt = whisper.load_model("base.en")
tts = TextToSpeechService()

template = """
You are a helpful and friendly AI assistant. You are polite, respectful, and aim to provide concise responses of less 
than 20 words.
The conversation transcript is as follows:
{history}
And here is the user's follow-up: {input}
Your response:
"""
PROMPT = PromptTemplate(input_variables=["history", "input"], template=template)
chain = ConversationChain(
    prompt=PROMPT,
    verbose=False,
    memory=ConversationBufferMemory(ai_prefix="Assistant:"),
    llm=Ollama(),
) Now, let's define the necessary functions: : This function runs in a separate thread to capture audio data from the user's microphone using the . The callback function is called whenever new audio data is available, and it puts the data into a for further processing. record_audio sounddevice.RawInputStream data_queue : This function utilizes the Whisper instance to transcribe the audio data from the into text. transcribe data_queue : This function feeds the current conversation context to the Llama-2 language model (via the Langchain ) and retrieves the generated text response. get_llm_response ConversationalChain : This function takes the audio waveform generated by the Bark text-to-speech engine and plays it back to the user using a sound playback library (e.g., ). play_audio sounddevice def record_audio(stop_event, data_queue):
    """
    Captures audio data from the user's microphone and adds it to a queue for further processing.
    Args:
        stop_event (threading.Event): An event that, when set, signals the function to stop recording.
        data_queue (queue.Queue): A queue to which the recorded audio data will be added.
    Returns:
        None
    """
    def callback(indata, frames, time, status):
        if status:
            console.print(status)
        data_queue.put(bytes(indata))

    with sd.RawInputStream(
        samplerate=16000, dtype="int16", channels=1, callback=callback
    ):
        while not stop_event.is_set():
            time.sleep(0.1)

def transcribe(audio_np: np.ndarray) -> str:
    """
    Transcribes the given audio data using the Whisper speech recognition model.
    Args:
        audio_np (numpy.ndarray): The audio data to be transcribed.
    Returns:
        str: The transcribed text.
    """
    result = stt.transcribe(audio_np, fp16=False)  # Set fp16=True if using a GPU
    text = result["text"].strip()
    return text

def get_llm_response(text: str) -> str:
    """
    Generates a response to the given text using the Llama-2 language model.
    Args:
        text (str): The input text to be processed.
    Returns:
        str: The generated response.
    """
    response = chain.predict(input=text)
    if response.startswith("Assistant:"):
        response = response[len("Assistant:") :].strip()
    return response

def play_audio(sample_rate, audio_array):
    """
    Plays the given audio data using the sounddevice library.
    Args:
        sample_rate (int): The sample rate of the audio data.
        audio_array (numpy.ndarray): The audio data to be played.
    Returns:
        None
    """
    sd.play(audio_array, sample_rate)
    sd.wait() Then, we define the main application loop. The main application loop guides the user through the conversational interaction as follows: The user is prompted to press Enter to start recording their input. Once the user presses Enter, the function is called in a separate thread to capture the user's audio input. record_audio When the user presses Enter again to stop the recording, the audio data is transcribed using the function. transcribe The transcribed text is then passed to the function, which generates a response using the Llama-2 language model. get_llm_response The generated response is printed to the console and played back to the user using the function. play_audio if __name__ == "__main__":
    console.print("[cyan]Assistant started! Press Ctrl+C to exit.")

    try:
        while True:
            console.input(
                "Press Enter to start recording, then press Enter again to stop."
            )

            data_queue = Queue()  # type: ignore[var-annotated]
            stop_event = threading.Event()
            recording_thread = threading.Thread(
                target=record_audio,
                args=(stop_event, data_queue),
            )
            recording_thread.start()

            input()
            stop_event.set()
            recording_thread.join()

            audio_data = b"".join(list(data_queue.queue))
            audio_np = (
                np.frombuffer(audio_data, dtype=np.int16).astype(np.float32) / 32768.0
            )

            if audio_np.size > 0:
                with console.status("Transcribing...", spinner="earth"):
                    text = transcribe(audio_np)
                console.print(f"[yellow]You: {text}")

                with console.status("Generating response...", spinner="earth"):
                    response = get_llm_response(text)
                    sample_rate, audio_array = tts.long_form_synthesize(response)

                console.print(f"[cyan]Assistant: {response}")
                play_audio(sample_rate, audio_array)
            else:
                console.print(
                    "[red]No audio recorded. Please ensure your microphone is working."
                )

    except KeyboardInterrupt:
        console.print("\n[red]Exiting...")

    console.print("[blue]Session ended.") Result Once everything is put down together, we can run the application as shown in the video above. The application runs quite slowly on my MacBook because the Bark model is large, even in its smaller version. Therefore, I have slightly sped up the video. For those with a CUDA-enabled computer, it might run faster. Here are the key features of our application: : Users can start and stop recording their voice input, and the assistant responds by playing back the generated audio. Voice-based interaction The assistant maintains the context of the conversation, enabling more coherent and relevant responses. The use of the Llama-2 language model allows the assistant to provide concise and focused responses. Conversational context: For those aiming to elevate this application to a production-ready status, the following enhancements are recommended: : Incorporate optimized versions of the models, such as whisper.cpp, llama.cpp, and bark.cpp, which are designed to boost performance, especially on lower-end computers. Performance Optimization : Implement a system that allows users to customize the bot's persona and prompt, enabling the creation of different types of assistants (e.g., personal, professional, or domain-specific). Customizable Bot Prompts : Develop a user-friendly GUI to enhance the overall user experience, making the application more accessible and visually appealing. Graphical User Interface (GUI) : Expand the application to support multimodal interactions, such as the ability to generate and display images, diagrams, or other visual content in addition to the voice-based responses. Multimodal Capabilities Finally, we have completed our simple voice assistant application, full code can be found at: . This combination of speech recognition, language modeling, and text-to-speech technologies demonstrates how we can build something that sounds difficult but can actually run on your computer. Let's enjoy coding, and don't forget to subscribe to so you don't miss the latest in AI and programming articles. https://github.com/vndee/local-talking-llm my blog Also published here

How to Build Your Own Voice Assistant and Run it Locally Using Whisper + Ollama + Bark

কীভাবে আপনার নিজের ভয়েস সহকারী তৈরি করবেন এবং হুইস্পার + ওল্লামা + বার্ক ব্যবহার করে স্থানীয়ভাবে এটি চালাবেন

So erstellen Sie Ihren eigenen Sprachassistenten und führen ihn lokal mit Whisper + Ollama + Bark aus

Cómo crear su propio asistente de voz y ejecutarlo localmente usando Whisper + Ollama + Bark

Comment créer votre propre assistant vocal et l'exécuter localement à l'aide de Whisper + Ollama + Bark

अपना खुद का वॉयस असिस्टेंट कैसे बनाएं और इसे व्हिस्पर + ओलामा + बार्क का उपयोग करके स्थानीय रूप से कैसे चलाएं

独自の音声アシスタントを構築し、Whisper + Ollama + Bark を使用してローカルで実行する方法

나만의 음성 어시스턴트를 구축하고 Whisper + Ollama + Bark를 사용하여 로컬에서 실행하는 방법

Como construir seu próprio assistente de voz e executá-lo localmente usando Whisper + Ollama + Bark

Как создать собственный голосовой помощник и запустить его локально с помощью Whisper + Ollama + Bark

Kendi Sesli Asistanınızı Oluşturma ve Whisper + Ollama + Bark Kullanarak Yerel Olarak Çalıştırma

Cách xây dựng trợ lý giọng nói của riêng bạn và chạy nó cục bộ bằng Whisper + Ollama + Bark

如何构建您自己的语音助手并使用 Whisper + Ollama + Bark 在本地运行

Intro Running open-source AI models locally on our own computers gives us privacy, endless possibilities of tinkering, and freedom from large corporations. It is almost a matter of free speech. For us GPU-poor, however, having our own AI computer seems to be a pricey dream. ? $3200, ouch! ? $1850; that hurts even if you can get one. ? Starting at $1200, still too much. Macbook M3 Max Nvidia 4090 Microsoft Surface Laptop 6 What if I tell you that you can get a useful AI computer for $300? Interested? You do need to supply your own monitor, keyboard, and mouse. And you need a bit of tinkering around the Linux operating system, drivers, middleware, and configurations. To clarify, we are NOT talking about “training” or “fine-tuning” large generative AI models. We will focus on how to run open-source LLM (large language models such as 7B) locally, as well as generating images using . LLama 2 Stable Diffusion Now, let’s continue. What Makes a Good (and Cheap) AI Computer? Let’s assume one of the main use cases for a home AI computer is running or LLM inference. This task actually does not need a GPU at all since it can all be done in CPU. is an open-source software that enables very fast LLM inference using normal CPU. It was originally designed for the Macbook with an Apple M-series CPU, but it does work on Intel/AMD CPUs as well. large language models llama.cpp However, you do need the following for a faster inference speed. Otherwise, you will be like watching hair grow on your palm while the LLM spits out one token at a time. Fast CPU to memory bandwidth Faster DRAM (at least DDR4, DDR5 will be even better) A lot of memory (like 16GB minimal), especially if you want to run larger models (beyond 7B) For image generation with Stable Diffusion, you do need GPU power. However, you don’t have to have a very fancy GPU for that. You can leverage the integrated GPU already in your home computers: All Macs with M1/M2/M3 CPU, which integrates CPU, GPU, and high-speed memory (they are really good, but due to price are excluded from this particular article) AMD APU (e.g., Ryzen 7 5700U), which integrates CPU and GPU for budget-friendly mini-PCs. This will be the focus of this article. Intel CPU (e.g., Core i5-1135G7), which also integrates CPU and GPU. They are slightly above the $300 budget for the entire mini-PC, but readers are welcome to explore them further on their own. And the $300 AI Computer Is? An AMD-based Mini PC with the following specs usually sells for less than $300. I don’t want to endorse any particular brand, so you can search yourself: 5800H (8C/16T, up to 4.4GHz) AMD Ryzen 7 16GB RAM DDR4 (32GB recommended) 512GB NVME M.2 SSD I splurged a bit and opted for the $400 model with 32GB RAM and 1TB SSD (everything else equal). The main reason is that I do research on open-source LLMs and would like to run bigger models, in addition to running Stable Difusion. But you should be able to do almost everything in this article with the $300 computer. Prep 1: Allocate Enough iGPU Memory For AMD APUs like the , memory is shared between CPU and iGPU (integrated GPU). In my case, I have 32GB RAM total, but the default allocation for iGPU was only 3GB! This varies from computer to computer and is configured in BIOS during manufacturing. Ryzen 7 5800H You need to change that depending on your main use case: If you only need to run LLM inference, you can skip this entire prep step. Since LLM inference will only need to use CPU, and you should save most RAM for the CPU so you can run larger LLM models. If you need to run , especially (1024x1024), you need to allocate as much RAM for the iGPU as the system allows (typically half of total RAM) Stable Diffusion SDXL In my case, I want to run both Stable Diffusion XL and LLM inference on the same mini PC. Therefore, I would like to allocate 16GB (out of 32GB total) for the GPU. You can achieve this by changing the settings in BIOS. Typically, there is an upper limit, and the default setting might be much lower than the upper limit. On my computer, the upper limit was 16GB, or half of the total RAM available. Good BIOS If your computer’s BIOS supports such settings, go ahead and change to your desired number. My BIOS has no such setting. Poor BIOS: Use the Universal AMD tool If your BIOS does not have this setting, then please follow the nice instruction “Unlocking GPU Memory Allocation on AMD Ryzen™ APU?”  by Winston Ma. I tried it and it worked well, so now I have 16GB VRAM. https://winstonhyypia.medium.com/amd-apu-how-to-modify-the-dedicated-gpu-memory-e27b75905056 Prep 2: Install Drivers & Middleware Align the Stars AMD’s (Radeon Open Compute platform), comparable to Nvidia’s , is a suite of drivers and middleware to enable developers to utilize the power of ADM’s GPUs. And typically, AI applications need ROCm to get GPU acceleration. ROCm CUDA In order to install and make AMD’s ROCm work, you have to make sure that the versions of GPU hardware, Linux distro, kernel, python, HIP driver, ROCm library, and pytorch are compatible. If you want the least pain and maximum possibility of first-time success, stick with the recommended and verified combinations. Prerequisite Please check the following link to get the compatible Linux OS and kernel versions, and install them. Initially, I made the mistake of just installing my favorite Linux OS and default Linux kernel, and it was a big pain to walk backward to resolve compatibility issues. You can avoid this pain by just using the officially supported combinations. https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html ROCm Installation https://rocm.docs.amd.com/en/latest/deploy/linux/os-native/install.html If the entire installation finishes well, you can type in , and something like this will show (I only snipped the most relevant parts in highlighted yellow): rocminfo ROCk module is loaded ===================== HSA System Attributes ===================== Runtime Version:         1.1 System Timestamp Freq.:  1000.000000MHz Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) Machine Model:           LARGE System Endianness:       LITTLE Mwaitx:                  DISABLED DMAbuf Support:          YES ========== HSA Agents ========== ******* Agent 1 ******* Name:                    AMD Ryzen 7 5800H with Radeon Graphics Uuid:                    CPU-XX Marketing Name:          AMD Ryzen 7 5800H with Radeon Graphics Vendor Name:             CPU Pool Info: Pool 1 Segment:                 GLOBAL; FLAGS: COARSE GRAINED Size:                    16777216(0x1000000) KB Python Environment Python dependency can be quite tricky, so it is good practice to set up a proper environment. You can use either or for this purpose. conda venv source venv/bin/activate conda activate llm Pytorch https://pytorch.org/get-started/locally/ pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm5.7 HSA Overwrite The following is specific to APU’s with integrated graphics. Even though they are not officially supported by ROCm, the following proved to work. export HSA_OVERRIDE_GFX_VERSION=9.0.0 How to Verify Now, after all the complicated steps, let’s test if ROCm is working with Torch. And you can see that ROCm is “pretending” to be CUDA for the purpose of Pytorch. https://rocm.docs.amd.com/en/latest/how_to/pytorch_install/pytorch_install.html#test-the-pytorch-installation python3 -c 'import torch' 2> /dev/null && echo 'Success' || echo 'Failure' Success python3 -c 'import torch; print(torch.cuda.is_available())' True LLM Inference Let’s start with something easy for our newly configured $300 AI computer: running a large language model locally. We can choose one of the popular open-source modes: with 7B parameters that is optimized for chat. LLaMA 2 In addition, you can also try small LLMs from , , , and . More good quality LLMs can be found here on the very useful “chatbot arena leaderboard” by UC Berkeley’s . Mistral QWen Zephyr Vicuna LMSYS labs https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard Llama.cpp We will be using , which is initially optimized for CPU and later supports GPU as well. In my experience, LLM inference works well on the CPU, and there is little to gain with a modest GPU such as the ones integrated inside the $300 AI machine. llama.cpp https://github.com/ggerganov/llama.cpp First, you need to install and . And then follow the steps to compile and install llama.cpp. wget git sudo apt-get install build-essential git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp make Download Model Weights In order to run the LLMs on our inexpensive machine instead of cloud servers with expensive GPUs, we need to use a “compressed” version of the models so they can fit into the RAM space. For a simple example, a LLaMA-2 7B model has 7B parameters, each represented by float16 (2 bytes). Float 16: 14B bytes or 14GB which will not fit into our 8GB RAM. Quantized to 4-bit: 3.5B bytes or 3.5GB which can now fit into our 8GB RAM. Also, the file format should be . So. in our example, you need to download the weights in this file: gguf https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/blob/main/llama-2-7b-chat.Q4_0.gguf Test on AMD Mini PC First, we tested it on the AMD mini PC, and we achieved about 10 tokens per second. This is actually quite decent, and you can carry on a chat with the LLM without too much waiting. System config: AMD Ryzen 5800H 32GB RAM Command line instruction: ./main -m models/llama-2-7b-chat.Q4_0.gguf --color -ins -n 512 --mlock llama_print_timings:        load time =     661.10 ms llama_print_timings:      sample time =     234.73 ms /   500 runs   (    0.47 ms per token,  2130.14 tokens per second) llama_print_timings: prompt eval time =    1307.11 ms /    32 tokens (   40.85 ms per token,    24.48 tokens per second) llama_print_timings:        eval time =   50090.22 ms /   501 runs   (   99.98 ms per token,    10.00 tokens per second) llama_print_timings:       total time =   64114.27 ms Test on Intel Mini PC Next, we tested on an Intel mini PC, and we achieved about 1.5 tokens per second. This is a bit too slow for a fruitful chat session. It is not a fair comparison, since the Intel N5105 is clearly weaker than AMD 5800H. But that is the only Intel mini PC in my possession. If you use the more powerful Intel CPU (e.g., Core i5-1135G7) you should get comparable results. Please report your findings in the comments below. System config: 11th Gen 4 Cores N5105 (Up to 2.9Ghz) 4 Cores and 4 Threads 16GB RAM (2GB VRAM for iGPU) ./main -m models/llama-2-7b-chat.Q4_0.gguf -ins --color -n 512 --mlock llama_print_timings:        load time = 14490.05 ms llama_print_timings:      sample time =   171.53 ms /    97 runs   (    1.77 ms per token,   565.49 tokens per second) llama_print_timings: prompt eval time = 21234.29 ms /    33 tokens (  643.46 ms per token,     1.55 tokens per second) llama_print_timings:        eval time = 75754.03 ms /    98 runs   (  773.00 ms per token,     1.29 tokens per second) Stable Diffusion Installation https://github.com/AUTOMATIC1111/stable-diffusion-webui And pay attention to this page as well, in regards to AMD ROCm https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Install-and-Run-on-AMD-GPUs Quick Start export HSA_OVERRIDE_GFX_VERSION=9.0.0 source venv/bin/activate ./webui.sh --upcast-sampling --skip-torch-cuda-test --precision full --no-half Stable Diffusion 1.5 Test ./webui.sh --upcast-sampling --skip-torch-cuda-test --precision full --no-half Test 1 Prompt: “horse in forest” Steps: 20, Sampler: DDIM, CFG scale: 7, Seed: 519288240, Size: 512x512, Model hash: 6ce0161689, Model: v1-5-pruned-emaonly, Version: v1.6.0 Time taken: 1 min. 8.3 sec. Stable Diffusion XL 1.0 Test SDXL (max resolution 1024x1024) recommends at least 12GB VRAM, so you definitely need to get the Prep 1 step done to allocate 16GB VRAM for iGPU. So, this task is only possible with the $400 mini PC. ./webui.sh --upcast-sampling Test 1: Prompt: “horse in forest” Steps: 20, Sampler: DDIM, CFG scale: 7, Seed: 1102941451, Size: 1024x768, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Version: v1.6.0 Time taken: 7 min. 41 sec Test 2: Prompt: “young taylor swift in red hoodie riding a horse in forest” Negative prompt: deformities, deformity, deformed eyes, deformed teeth, deformed fingers, deformed face, deformed hands, deformed Steps: 20, Sampler: DDIM, CFG scale: 7, Seed: 2960206663, Size: 1024x1024, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Version: v1.6.0 Time taken: 6 min. 12.3 sec. Windows 11 and AMD/directml Although this article focuses on Linux operating systems, you can get Stable Diffusion working in Windows too. Here are my experiments: https://github.com/lshqqytiger/stable-diffusion-webui-directml First, you need to install Python 3.10.6. Add Python 3.10.6 director to PATH. Important: The Python path has to be the top path https://realpython.com/add-python-to-path/ Install git and git clone repo Run webui-user.bat from the file explorer Test 1: Prompt: “horse in forest” Settings: DPM++ 2M Karras, 512x512, sampling steps 20 Time taken: 1m19s Conclusions So, are you having fun running your own generative AI models on your new $300 mini PC? I hope you do. Open-source AI models running on personal devices is one of the most exciting areas for tinkers since none of us will have the massive GPU pool to actually train a foundational model. This will enable a new generation of apps that are both super smart while still preserving our data privacy. What next? Run it on even smaller embedded devices: e.g., Raspberry Pi Run it on your smartphones (llama.cpp does support iOS and Android) And happy tinkering with AI, open source, and on-device!

How to Build a $300 AI Computer for the GPU-Poor

GPU-দরিদ্রদের জন্য $300 এআই কম্পিউটার কীভাবে তৈরি করবেন

So bauen Sie einen 300-Dollar-KI-Computer für GPU-arme Menschen

Cómo construir una computadora con inteligencia artificial de $ 300 para los pobres en GPU

Comment construire un ordinateur IA à 300 $ pour les pauvres en GPU

जीपीयू-खराब के लिए $300 का एआई कंप्यूटर कैसे बनाएं

GPU が苦手な人向けに 300 ドルの AI コンピューターを構築する方法

GPU가 부족한 사람들을 위해 300달러짜리 AI 컴퓨터를 만드는 방법

Como construir um computador com IA de US$ 300 para os pobres em GPU

Как построить компьютер с искусственным интеллектом за 300 долларов для тех, у кого мало графических процессоров

GPU'su Yoksullar için 300 Dolarlık Yapay Zekalı Bilgisayar Nasıl Oluşturulur

Cách xây dựng một máy tính AI trị giá 300 USD cho người có GPU kém

如何为 GPU 匮乏的人打造一台 300 美元的人工智能计算机

How to Run Your Own Local LLM: Updated for 2024 - Version 1

কিভাবে আপনার নিজের স্থানীয় LLM চালাবেন (2024 এর জন্য আপডেট করা হয়েছে)

So führen Sie Ihren eigenen lokalen LLM durch (Aktualisiert für 2024)

Cómo ejecutar su propio LLM local (actualizado para 2024)

Comment gérer votre propre LLM local (mis à jour pour 2024)

अपना खुद का स्थानीय एलएलएम कैसे चलाएं (2024 के लिए अपडेट किया गया)

独自のローカル LLM を運営する方法 (2024 年更新)

자신만의 지역 LLM을 운영하는 방법(2024년 업데이트)

Como executar seu próprio LLM local (atualizado para 2024)

Как запустить собственный местный LLM (обновлено для 2024 г.)

Kendi Yerel LLM'nizi Nasıl Yönetirsiniz (2024 için Güncellendi)

Cách chạy LLM địa phương của riêng bạn (Cập nhật năm 2024)

如何开办自己的本地法学硕士 (2024 年更新)

The last year was all about the race of commercial LLMs. Though the LLaMa leakage and further Meta actions on the open-source front were significant, and Hugging Face kept blooming as a platform for open source, this year starts with a powerful player stepping in with a real open-source approach. We speak, of course, about : by the Allen Institute for Artificial Intelligence. Newsletters such as AlphaSignal, TheSequence, Data Machina, Smol Talk, and Interconnects (Nathan Lambert is one of the authors of the OLMo paper) explained pretty well what’s the difference between almost open-source and truly open-source models. OLMo Accelerating the Science of Language Model, released The gist of it is that truly open source means that not only the weights of the model and inference code are released but truly the whole package: the training data, training and evaluation code, and a comprehensive framework for studying language modeling. Who is behind OLMo? What I found interesting is who stands behind the release. While EleutherAI’s Pythia and Big Science’s BLOOM previously set a precedent for releasing fully open-source models, the distinction with OLMo is its release by a true nonprofit organization — the Allen Institute for AI (AI2). AI2 was founded in 2014 by philanthropist and Microsoft co-founder Paul G. Allen, who committed to conducting high-impact research and engineering in artificial intelligence. He was also very interested in teaching machines “common sense.” And he funded this cause well. Once, I had a conversation with one of the top executives of AI2; the person said that thanks to Paul Allen’s financing structure, AI2 is well-funded, has no influence from large companies, and has no pressure to make money. AI2 is famous for conducting cutting-edge research in AI and aiming to influence the broader AI research community by releasing open-source software, datasets, and research findings. Projects like the academic search engine democratize access to information and accelerate scientific breakthroughs. Semantic Scholar Why OLMo is special The OLMo framework includes multiple training checkpoints, logs, exact datasets used, and a permissive license, establishing a new standard for openness in the field. They also don’t mind this model being used for commercial purposes. Unlike others, the researchers readily embrace openness, believing it outweighs the low misuse risk, as their models, not designed as chatbots, contribute to science rather than commercial products. Furthermore, they released ‘ ’ According to , ‘the name of the pretraining corpus, “Dolma,” stands for Data to feed OLMo’s Appetite.’ Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. Luca Soldani What also surprised me was that the authors highlighted the environmental impact of training large LMs, providing estimates of power consumption and carbon emissions. They advocate for transparency in reporting these impacts and emphasize the potential for open models like OLMo to mitigate future emissions by minimizing redundant model training. Great start to the year of open-source! News from The Usual Suspects © Google In their blog post, Google MobileDiffusion, a new approach for sub-second text-to-image generation on mobile devices, addressing the efficiency challenges of current large models. This model combines a pre-trained diffusion model with a GAN to enable one-step sampling during inference, resulting in a compact model size of 520M parameters capable of generating high-quality images rapidly on iOS and Android devices. The research includes an in-depth exploration of model architecture, optimizing for mobile deployment while maintaining image quality. announced Meta In , Mark Zuckerberg explained how Meta benefits from open source: Meta’s Q4 2023 earnings call yesterday “There are several strategic benefits. First, open source software is typically safer and more secure, as well as more compute efficient to operate due to all the ongoing feedback, scrutiny, and development from the community. This is a big deal because safety is one of the most important issues in AI. Efficiency improvements and lowering the compute costs also benefit everyone including us. Second, open source software often becomes an industry standard, and when companies standardize on building with our stack, that then becomes easier to integrate new innovations into our products. That’s subtle, but the ability to learn and improve quickly is a huge advantage and being an industry standard enables that. Third, open source is hugely popular with developers and researchers. We know that people want to work on open systems that will be widely adopted, so this helps us recruit the best people at Meta, which is a very big deal for leading in any new technology area. And again, we typically have unique data and build unique product integrations anyway, so providing infrastructure like Llama as open source doesn’t reduce our main advantages. This is why our long-standing strategy has been to open source general infrastructure and why I expect it to continue to be the right approach for us going forward.” — Mark Zuckerberg The freshest research papers, categorized for your convenience Language Modeling and Efficiency : Examines LLMs in mathematical reasoning, highlighting capabilities and challenges. Large Language Models for Mathematical Reasoning: Progresses and Challenges read the paper : Introduces an ∞-gram model for extensive text analysis. Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens read the paper : Proposes a sustainable and efficient LLM pre-training method. Scavenging Hyena: Distilling Transformers into Long Convolution Models read the paper : Presents a strategy for efficient LLM pre-training using rephrased web documents. Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling read the paper Advanced Reasoning and Contextual Understanding : Enhances multi-step reasoning in LLMs for improved accuracy and efficiency. Efficient Tool Use with Chain-of-Abstraction Reasoning read the paper : Improves performance on complex reasoning tasks through hierarchical text summarization. RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval read the paper : Investigates LLMs’ contextual understanding and the impact of compression techniques. Can Large Language Models Understand Context? read the paper Enhancements in AI Frameworks and Methodologies : Merges generative models with solvers for advanced concept learning. SymbolicAI: A framework for logic-based approaches combining generative models and solvers read the paper : Demonstrates the benefits of efficient exploration in LLM training. Efficient Exploration for LLMs read the paper : Discusses methods for improving LLM alignment with human preferences. Transforming and Combining Rewards for Aligning Large Language Models read the paper : Introduces a MoE approach for efficient multi-modal learning in LVLMs. MoE-LLaVA: Mixture of Experts for Large Vision-Language Models read the paper Novel Applications and Security Insights : Develops an autonomous agent for mobile devices using visual perception. Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception read the paper : Explores text diffusion as a competitive method for LLMs. Transfer Learning for Text Diffusion Models read the paper : Identifies a security issue in LLMs and proposes initial defense strategies. Weak-to-Strong Jailbreaking on Large Language Models read the paper In other newsletters A very interesting read about a new trend: VCs backing companies where the majority of revenues come from government contracts by Newcomer The research report gives interesting details about the types of security assessments Chinese AI labs are implementing to comply with generative AI regulations by ChinAI celebrates a large milestone towards democratizing AI inference with their Vulkan Kompute project that was adopted as one of the backends for the LLama.cpp and GPT4ALL frameworks. ML Engineer Newsletter Also published . here

FOD 39: Truly Open – We Explore Who Stands Behind OLMo's Release

If you just want to see the code - see . github What are we going to build and why? LLMs are a new hot piece of technology that everyone is experimenting with. Managed services like OpenAI are the cheapest and most convenient way to use them. You can use their LLMs only for a small fee and for approval to train new versions of your data (in some cases). In certain applications, it is required to run an on our own. You may want to process sensitive data (medical records or legal documents) or get great quality output in a language different than English. Sometimes, you have a specialized task that doesn’t require expensive big models from OpenAI. LLM Open-source LLMs are matching the quality of big players like OpenAI but still require a lot of expensive computing resources. There are smaller models available that can run on weaker hardware and perform well enough. As an experiment, let’s deploy a smaller open-source LLM on AWS lambda. Our goal is to learn more about LLMs and docker-based lambdas. We will also evaluate performance and cost to determine if any real-world applications are feasible. For this project, we are going to use the model, a 2.7 billion parameter LLM that matches the quality of outputs from 13B parameters or more open-source models. It was trained on a large dataset and is a viable model for many applications. From my experience, it hallucinates a lot but otherwise provides useful outputs. Its size is perfect for the AWS Lambda environment. Microsoft Phi-2 We will download the model from Huggingface and run it via package (bindings to the popular llama.cpp, heavily optimized model CPU runtime). We will use a smaller, quantized version, but even the full one should fit in Lambda memory. llama-cpp-python We will create an HTTP REST endpoint via the lambda URL mechanism for a given call: PROMPT="Create five questions for a job interview for a senior python software engineer position."

curl $LAMBDA_URL -d "{ \"prompt\": \"$PROMPT\" }" \
    | jq -r '.choices[0].text, .usage' (jq command used to decode returned JSON) Provides LLM output alongside execution details: Instruct: Create five questions for a job interview for a senior python software engineer position.
Output: Questions: 

1. What experience do you have in developing web applications?
2. What is your familiarity with different Python programming languages?
3. How would you approach debugging a complex Python program?
4. Can you explain how object-oriented programming principles can be applied to software development? 
5. In a recent project, you were responsible for managing the codebase of a team of developers. Can you discuss your experience with this process?

{
  "prompt_tokens": 21,
  "completion_tokens": 95,
  "total_tokens": 116
} For this tutorial, you need to know the basics of programming, docker, AWS, and Python. I am using an M-series Mac, so as a default, we’re going to aim at deployment to ARM AWS Lambda, as it simply costs less for more performance. Instructions on how to deploy the x86 version are included. Environment setup (AWS, Docker and Python) I won’t dive into the basics here. If you don’t have the tools installed, please follow these resources. You need an AWS account and AWS CLI installed and configured. Good tutorial from Google search: . tutorial Official docs, including if you’re using SSO: AWS official tutorial. You also need docker: docker docs. As an IDE, I am using Visual Studio Code. Set up lambda function with docker locally. Let’s start with getting the local environment running. If you need more information than I included here, check out the official AWS documentation. We will store every file in a single project directory without subfolders. First, we will need a basic Python lambda function handler. In your project folder, create a file called . lambda_function.py import sys

def handler(event, context):
    return "Hello from AWS Lambda using Python" + sys.version + "!" We will also create a file, in which we specify our dependencies. Let’s start with the AWS library for interacting with their services, just as an example. requirements.txt boto3 Then, we need to specify our docker image composition in file. Comments document what each line does. Dockerfile FROM public.ecr.aws/lambda/python:3.12

# Copy requirements.txt
COPY requirements.txt ${LAMBDA_TASK_ROOT}

# Install the specified packages
RUN pip install -r requirements.txt

# Copy function code
COPY lambda_function.py ${LAMBDA_TASK_ROOT}

# Set the CMD to your handler
CMD [ "lambda_function.handler" ] Finally, we will create a file to make our life easier when running and building the container. docker-compose.yml version: '3'

services:
  llm-lambda:
		image: llm-lambda
    build: .
    ports:
      - 9000:8080 OK, done! Make sure your docker engine is running, and type in the terminal: docker-compose up The container should be built and started. Our lambda is available under a long URL below (quirks of the official Amazon image). To test if the lambda is running, open another terminal window/tab and type in: curl "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}' Now you can come back to the tab you ran docker-compose in and type ctrl-c to stop the container. You can put this test code in a bash script, as seen in the project repo. Also, it will be useful to create a .gitignore file if you’re going to version control this project. Run an LLM inside a container. Now that we have a working lambda let’s add some AI magic. To run an LLM, we need to add to our . llama-cpp-python requirements.txt boto3
llama-cpp-python To build it, we need to introduce a . This is because the default Amazon docker image doesn’t include the build tools required for llama-cpp. We are doing a pip install with flags in order to use multi-threaded optimizations. docker build stage CMAKE_ARGS="-DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=OpenBLAS" The code below also includes downloading the model. Community hero, the Bloke, is sharing compressed (quantized, less computationally intensive but lower quality) versions of models, ready to download . To make use of them, we can install the huggingface CLI and run the appropriate commands. You can switch the repository ( ) and the model ( ) to whatever you like if you want to deploy a different model. here TheBloke/phi-2-GGUF phi-2.Q4_K_M.gguf RUN pip install huggingface-hub && \
    mkdir model && \
    huggingface-cli download TheBloke/phi-2-GGUF phi-2.Q4_K_M.gguf --local-dir ./model --local-dir-use-symlinks False # Stage 1: Build environment using a Python base image
FROM python:3.12 as builder

# Install build tools
RUN apt-get update && apt-get install -y gcc g++ cmake zip

# Copy requirements.txt and install packages with appropriate CMAKE_ARGS
COPY requirements.txt .
RUN CMAKE_ARGS="-DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=OpenBLAS" pip install --upgrade pip && pip install -r requirements.txt

# Stage 2: Final image using AWS Lambda Python image
FROM public.ecr.aws/lambda/python:3.12

# Install huggingface-cli and download the model
RUN pip install huggingface-hub && \
    mkdir model && \
    huggingface-cli download TheBloke/phi-2-GGUF phi-2.Q4_K_M.gguf --local-dir ./model --local-dir-use-symlinks False

# Copy installed packages from builder stage
COPY --from=builder /usr/local/lib/python3.12/site-packages/ /var/lang/lib/python3.12/site-packages/

# Copy lambda function code
COPY lambda_function.py ${LAMBDA_TASK_ROOT}

CMD [ "lambda_function.handler" ] Let’s modify our lambda code to run LLM inference. Some more work is required to get the prompt out of the request body in the production environment. import base64
import json
from llama_cpp import Llama

# Load the LLM, outside the handler so it persists between runs
llm = Llama(
    model_path="./model/phi-2.Q4_K_M.gguf", # change if different model
    n_ctx=2048, # context length
    n_threads=6,  # maximum in AWS Lambda
)

def handler(event, context):
    print("Event is:", event)
    print("Context is:", context)

		# Locally the body is not encoded, via lambda URL it is
    try:
        if event.get('isBase64Encoded', False):
            body = base64.b64decode(event['body']).decode('utf-8')
        else:
            body = event['body']

        body_json = json.loads(body)
        prompt = body_json["prompt"]
    except (KeyError, json.JSONDecodeError) as e:
        return {"statusCode": 400, "body": f"Error processing request: {str(e)}"}

    output = llm(
        f"Instruct: {prompt}\nOutput:",
        max_tokens=512, 
        echo=True,
    )

    return {
        "statusCode": 200,
        "body": json.dumps(output)
    } OK, time to test! My docker setup has 8GB of RAM assigned, but it should run well under 4GB. Let’s rebuild the container and start it again. It will take more time as the model needs to download (~1.8GB). docker-compose up --build This time, let’s test with a real prompt. The first run will take more time as the model needs to load. curl "http://localhost:9000/2015-03-31/functions/function/invocations" \
    -d '{ "body": "{ \"prompt\": \"Generate a good name for a bakery.\" }" }' Example output: Instruct: Generate a good name for a bakery.
Output: Sugar Rush Bakery.

{
  "prompt_tokens": 15,
  "completion_tokens": 6,
  "total_tokens": 21
} Deploy to AWS Lambda With everything working locally, let’s finally deploy our LLM to AWS! To execute the deployment successfully, the following steps are required: : Determine necessary information such as AWS region, ECR repository name, Docker platform, and IAM policy file, and disable AWS CLI pager. Initial Setup : Optionally confirm that the AWS CLI is correctly configured. Verify AWS Configuration : ECR (Elastic Container Registry) Repository Management Determine if the specified ECR repository already exists. If absent, create a new ECR repository. : IAM Role Handling Check for the existence of a specific IAM role for Lambda. If not found, establish the IAM role and apply the policy. AWSLambdaBasicExecutionRole : Securely log Docker into the ECR registry using the retrieved login credentials. Docker-ECR Authentication : Utilize Docker Compose to build the Docker image, specifying the desired platform. Docker Image Construction : Label the Docker image appropriately for ECR upload. ECR Image Tagging : Transfer the tagged Docker image to the ECR. ECR Image Upload : Fetch the ARN linked to the specified IAM role. Acquire IAM Role ARN : Assess whether the Lambda function exists. Lambda Function Verification : Set parameters like timeout, memory allocation, and image URI for Lambda. Lambda Function Configuration : Lambda Function Deployment/Update For a new Lambda function, create it with defined settings and establish a public Function URL. For an existing Lambda function, update its code using the new Docker image URI. : Obtain and display the Function URL of the Lambda function. Function URL Retrieval It is a simple but lengthy process to do manually, so I’ve created a script using AWS CLI. Feel free to modify the variables and run it yourself. The script is quite complex; you can do all the steps manually in the GUI if you want, apart from pushing the container. If you are not doing this on an M-series Mac, be sure to change Linux/arm64 to Linux/amd64. #!/bin/bash

# Variables
AWS_REGION="eu-central-1"
ECR_REPO_NAME="llm-lambda"
IMAGE_TAG="latest"
LAMBDA_FUNCTION_NAME="llm-lambda"
LAMBDA_ROLE_NAME="llm-lambda-role" # Role name to create, not ARN
DOCKER_PLATFORM="linux/arm64" # Change as needed, e.g., linux/amd64
IAM_POLICY_FILE="trust-policy.json"
PAGER= # Disable pager for AWS CLI

# Authenticate with AWS
aws configure list # Optional, just to verify AWS CLI is configured

# Check if the ECR repository exists
REPO_EXISTS=$(aws ecr describe-repositories --repository-names $ECR_REPO_NAME --region $AWS_REGION 2>&1)

if [ $? -ne 0 ]; then
    echo "Repository does not exist. Creating repository: $ECR_REPO_NAME"
    # Create ECR repository
    aws ecr create-repository --repository-name $ECR_REPO_NAME --region $AWS_REGION
else
    echo "Repository $ECR_REPO_NAME already exists. Skipping creation."
fi

# Check if the Lambda IAM role exists
ROLE_EXISTS=$(aws iam get-role --role-name $LAMBDA_ROLE_NAME 2>&1)

if [ $? -ne 0 ]; then
    echo "IAM role does not exist. Creating role: $LAMBDA_ROLE_NAME"
    # Create IAM role for Lambda
    aws iam create-role --role-name $LAMBDA_ROLE_NAME --assume-role-policy-document file://$IAM_POLICY_FILE
    aws iam attach-role-policy --role-name $LAMBDA_ROLE_NAME --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
else
    echo "IAM role $LAMBDA_ROLE_NAME already exists. Skipping creation."
fi

# Get login command from ECR and execute it to authenticate Docker to the registry
aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $(aws sts get-caller-identity --query Account --output text).dkr.ecr.$AWS_REGION.amazonaws.com

# Build the Docker image using Docker Compose with specific platform
DOCKER_BUILDKIT=1 docker-compose build --build-arg BUILDPLATFORM=$DOCKER_PLATFORM

# Tag the Docker image for ECR
docker tag llm-lambda:latest $(aws sts get-caller-identity --query Account --output text).dkr.ecr.$AWS_REGION.amazonaws.com/$ECR_REPO_NAME:$IMAGE_TAG

# Push the Docker image to ECR
docker push $(aws sts get-caller-identity --query Account --output text).dkr.ecr.$AWS_REGION.amazonaws.com/$ECR_REPO_NAME:$IMAGE_TAG

# Get the IAM role ARN
LAMBDA_ROLE_ARN=$(aws iam get-role --role-name $LAMBDA_ROLE_NAME --query 'Role.Arn' --output text)

# Check if Lambda function exists
FUNCTION_EXISTS=$(aws lambda get-function --function-name $LAMBDA_FUNCTION_NAME --region $AWS_REGION 2>&1)

# Parameters for Lambda function
LAMBDA_TIMEOUT=300 # 5 minutes in seconds
LAMBDA_MEMORY_SIZE=10240 # Maximum memory size in MB
LAMBDA_IMAGE_URI=$(aws sts get-caller-identity --query Account --output text).dkr.ecr.$AWS_REGION.amazonaws.com/$ECR_REPO_NAME:$IMAGE_TAG

# Deploy or update the Lambda function
if echo $FUNCTION_EXISTS | grep -q "ResourceNotFoundException"; then
    echo "Creating new Lambda function: $LAMBDA_FUNCTION_NAME"
    aws lambda create-function --function-name $LAMBDA_FUNCTION_NAME \
        --region $AWS_REGION \
        --role $LAMBDA_ROLE_ARN \
        --timeout $LAMBDA_TIMEOUT \
        --memory-size $LAMBDA_MEMORY_SIZE \
        --package-type Image \
        --architectures arm64 \
        --code ImageUri=$LAMBDA_IMAGE_URI

    aws lambda create-function-url-config --function-name $LAMBDA_FUNCTION_NAME \
        --auth-type "NONE" --region $AWS_REGION

    # Add permission to allow public access to the Function URL
    aws lambda add-permission --function-name $LAMBDA_FUNCTION_NAME \
        --region $AWS_REGION \
        --statement-id "FunctionURLAllowPublicAccess" \
        --action "lambda:InvokeFunctionUrl" \
        --principal "*" \
        --function-url-auth-type "NONE"
else
    echo "Updating existing Lambda function: $LAMBDA_FUNCTION_NAME"
    aws lambda update-function-code --function-name $LAMBDA_FUNCTION_NAME \
        --region $AWS_REGION \
        --image-uri $LAMBDA_IMAGE_URI
fi

# Retrieve and print the Function URL
FUNCTION_URL=$(aws lambda get-function-url-config --region $AWS_REGION --function-name $LAMBDA_FUNCTION_NAME --query 'FunctionUrl' --output text)
echo "Lambda Function URL: $FUNCTION_URL" Now, let’s add proper permissions and run the script. chmod +x deploy.sh
./deploy.sh
# AWS_PROFILE=your_profile ./deploy.sh 
# if you want to use a different profile than default As a result of the script, you should se the lambda function URL in your terminal. Let’s use it for a test. PROMPT="Create five questions for a job interview for a senior python software engineer position."

curl $LAMBDA_URL -d "{ \"prompt\": \"$PROMPT\" }" \
    | jq -r '.choices[0].text, .usage' Like before, the first run will take longer. chmod +x test_remote.sh
LAMBDA_URL=your_url ./test_remote.sh You should receive back an output similar to the one at the top of the article. If you want to see the full project code, see . github Performance Cost and speed You may have noticed that I chose to run the model under the configuration: 6 core multi-threading 10GB of RAM 5-minute timeout How was the performance, and how much RAM was used? Lambda never reserves compute for you and sometimes loads your program “cold.” When you make a request, anything you need to load (like a model) needs to happen at this moment. We call this a cold start. For the prompt about five interview questions, the execution time after the cold start was 17 seconds, which is not bad at all. After that, it consistently took 9 seconds to generate an output, tested in 4 subsequent runs. The average output length was 83 tokens. So, when running hot. we achieved ~9.2 tokens/second As a comparison, community-reported speeds for OpenAI models are GPT-4 10t/s, GPT-4-turbo 48t/s, and GPT-3-turbo 50-100t/s. So we’re matching a decent but not industry-leading speed with little effort. You could run this prompt 4444 times in a free tier. Above that, . The same 1000 runs with a GPT-3.5 turbo would cost $0.2. This makes this endeavor not really cost-effective, so low cost shouldn’t be your aim if you’re going to implement this in production. it would cost $1.2 per 1000 runs Optimization Could we run the model cheaper? The model uses under 2600MB of memory consistently. We could lower the allocated resources, with a caveat that 6 core CPU is assigned only to Lambdas above ~8846MB. I ran the first test at 3500MB (3 CPU cores/threads) to see how it compares. I didn’t change the code to run only 3 threads, which had a downgrading effect. In this case, the lambda always timed out (ran over 5 minutes, which meant the request was canceled). After changing the number of threads to the proper 3, the execution almost timed out at a cold start, returning results after 4 minutes and 33 seconds; the subsequent 3 runs took 26-40 seconds, averaging 3.15 tokens/second. 1000 paid runs would cost So we got slower and paid more, not a good optimisation! With this in mind, it makes sense to pay for more memory because AWS assigns CPU resources proportionally to the reserved RAM amount. $1.45. Next steps As you can see, running an open-source language model is possible even in a regular environment like AWS Lambda. The cost and quality of output do not match industry-leading services but can serve as a use-case-specific solution. If anything, this was a great opportunity to learn about basic LLM ecosystem libraries and tools, as well as strengthen your AWS knowledge. Some possible next steps if you want to keep exploring and bringing this project closer to production-ready: Try to run a different (bigger) model or a less compressed version of Microsoft Phi-2. Create a CDK config to automatically deploy the model and create resources. Create a GitHub Action to validate the project and run ./deploy.sh. Try a different serverless platform to run the model. Subscribe to my newsletter and follow me on social media. I may write about some of these topics, let me know in the comment if you’re interested in any of them! If you bump into any issues let me know as well. What do you think about this way of deploying LLMs? Subscribe to my profile by filling in your email address on the left, and be up-to-date with my articles! Don't forget to follow me on Twitter and subscribe to my blog’s for more tips and insights! @ horosin_ newsletter If you don't have Twitter, you can also follow me on . LinkedIn Also published . here

Deploying Open-Source Language Models on AWS Lambda

Introduction As LLM technologies gain more mainstream adoption and the ecosystem begins to mature, organizations are beginning to recognize the limits and costs of using LLM technologies. Many enterprises, originally enthusiastic about the application of , have abandoned centralized initiatives, pursuing instead a strategy of encouraging decentralized efforts to incorporate services like ChatGPT and Claude into their workflows. LLM technologies There are several reasons for this phenomenon. Lack of LLM expertise, MLOps requirements, and reliance on specialized GPU infrastructure are all barriers to implementing large-scale AI initiatives. Of these, however, the thorniest issue is the reliance on GPUs. In this article, we will discuss the specific difficulties posed by GPU reliance, explore a potential solution, and look at an exciting example from one of the trailblazing companies working in this area. GPU Availability as Limitations to LLMs Most publicly available and highly performant models, such as GPT-4, Llama 2, and Claude, all rely on highly specialized GPU infrastructure. GPT-4, one of the largest models commercially available, famously runs on a cluster of 8 A100 GPUs. Llama 2’s 70B model, which is much smaller, still requires at least an A40 GPU to run at a reasonable speed. This level of GPU requirement practically forecloses the possibility of running these models locally - a A100 GPU, assuming you can find a seller, costs close to $25,000. Once you obtain the GPUs, you need specialized skills to set up and maintain the servers. Very few organizations would be willing to incur such an outlay to experiment with LLM technologies. To solve this problem, several startups and cloud providers have developed extensive PaaS offerings. Some services like Replicate, which I have used in past articles and projects, allow users to rent GPU servers and pay for compute time used. Other providers, like OpenAI and Anthropic, offer their models as a per-token API, further abstracting away infrastructural complexities. However, these services require data to be sent to an external network, which makes using these services a non-starter for privacy-conscious organizations. Additionally, many of these services suffer from shortages during demand spikes as GPU usage overtakes availability, making them unreliable options for production-critical workloads. Additionally, GPU time, no matter how they are charged for, is expensive for large compute tasks - the companies that own and operate these GPUs need a return on their investment, after all. While these costs are nearly negligible for experimental use cases, commercial use cases often require embedding large contexts, fine-tuning, or multi-shot examples. These costs represent a significant barrier to adoption, especially for organizations with large datasets or those that lack the financial resources of large U.S. firms. In a previous article, we explored parameter compression as one strategy to reduce dependence on GPUs. In today’s article, we will explore another exciting technique called quantization. Before we dive into the exploration, however, we might want to learn a little bit about quantization first. Quantization (Optional Reading) In this section, we will briefly go over the basics of quantization. However, if you’re simply looking for a way to run powerful LLMs locally on your computer, you can feel free to skip this section for now and come back later. LLMWare, the company whose technology we will be using today, has built some amazing tools that let you get started with quantized models without having to get into the nitty-gritty behind it all. C/C++ implementations What is Quantization? Quantization is a technique that seeks to reduce the computational and memory requirements of running an LLM by using lower-precision numerical types. Many popular open-source models, such as Llama, Falcon, and Alpaca, use PyTorch as the underlying framework. By default, PyTorch models use 32-bit floating points, meaning a single parameter takes up 32 “bits” in the GPU memory. Quantization aims to replace these parameters with 16-bit floating points, 8-bit integers, or even 4-bit integers. Successful quantization leads to dramatic improvements in computational speed and reductions in memory usage, meaning large models become runnable on lower-end GPUs, embedded graphic chips, or even CPUs. This idea has been around for a while - PyTorch itself has added support for 16-bit floating points and model compilations as the technology matured, but progress has been slow due to early design decisions in the PyTorch framework. Does Quantization Degrade Performance? At this point, it’s natural to wonder, wouldn’t this severely degrade the accuracy of the model? The short answer is yes, but only if you do it carelessly. Every optimization comes with inherent trade-offs, but with some specialized techniques, researchers have been able to squeeze incredibly stable performance out of highly quantized models. While we won’t go into extreme technical detail, let’s go over the broad strokes of the most common strategies being used right now. If you want to learn more, you can find out more about it in a from HuggingFace. guide Calibrated Quantization During the quantization process, a calibration dataset is run through the model. The value of each parameter is recorded, and the range is used to determine how the parameters get quantized. Assuming the calibration dataset is representative of the inputs that the model will encounter, this results in improved accuracy of the resulting model. Quantization-Aware Whereas Calibrated Quantization happens after training, Quantization-Aware Training tries to optimize the model during training. While the model is training, the activations are put through a “fake quantization,” simulating errors that will likely be introduced by the quantization process. The model is then able to adapt to the errors, resulting in a more robust model that can specifically adapt to the potential distortions. Llama.cpp and GGUF While PyTorch quantization and optimizations have been long blocked by framework design, two recent open-source technologies broke through these barriers and made quantization technologies much more accessible to the general public. Let’s briefly cover them below. Llama.cpp Llama.cpp was a project by Georgi Gerganov to port the Llama model into C/C++. This got rid of the complexity introduced by PyTorch, and the native implementation allowed quantization to be implemented directly. Therefore, the resulting model could run with up to 4-bit integer quantization, allowing high-parameter-count Llama models to be run without a specialized GPU. The project has since been extended by the community to include a roster of open-source models, including popular ones like Falcon and Mistral. GGUF GGUF is Llama.cpp’s file format for storing and transferring model information. Quantized models are stored in this format so that they can be loaded and run by the end-user. GGUF is the successor format to GGML and aims to improve on GGML by providing more extensibility, backward compatibility, and stability while still allowing for rapid development. The development of a universal file format opened the door for the open-source community to extend the Llama.cpp to optimize other models, and innovators like TheBloke and LLMWare have been working over the last several months to miniaturize popular open-source models. LLMWare’s Quantized Dragon Model In today’s example, we will be using open-source libraries and quantized models provided by LLMWare, which provides convenient tools for quickly building specialized RAG workflows. Who is LLMWare? LLMWare, a generative AI company specializing in the legal and financial industries, has been actively involved in the quantization community. As I’ve written before, their focus on privacy-conscious sectors makes them a natural candidate for experimenting with and innovating in miniaturization technologies. Previously, I wrote about their RAG-optimized BLING models that squeeze incredible performance out of 1 to 3 billion parameter models for specialized tasks like contract review and financial analysis. While most open-source models with such parameter counts tend to be only useful for toy problems, LLMWare is able to generate production-ready performance out of these models by training them for narrowly targeted tasks. These miniaturized models are then able to run without an external GPU, allowing for increased privacy and scalability. What is Dragon? Dragon is a collection of LLMs that can be thought of as more powerful versions of their BLING cousins. The original intent of Dragon was to train a higher parameter model using the same instruction fine-tuning techniques, providing an option to users that need more performance and have access to lower-end GPUs. The added parameter count resulted in more powerful models that could leverage larger context windows and generate more complex outputs, but required the user to have more specialized hardware, such as a GPU-embedded laptop or a cloud compute container with a GPU attached. However, they still represented an improvement over the extremely large models, which would require waiting for access to scarce A40 or A100 GPUs. Quantized Dragon, the Best of Both Worlds Given the above, it is easy to see why quantization gave a significant boost to LLMWare’s suite of AI tools. With quantization, a user could run Dragon-tier models on the same environment as BLING models, allowing for much more powerful analysis on commodity computers. Over the course of the last month, LLMWare has published quantized versions of several Dragon models. Today, we will evaluate LLMWare’s Dragon model built on top of Llama with a legal analysis RAG problem and compare it with a similar BLING model. Interested reachers can also explore other models - a Mistral-based model and a Yi-Based model are available from LLMWare at the time of this writing. Additionally, has made running inferences on Llama.cpp models a breeze with their tight integrations with the ctransformers library, which allows gguf models to be swapped seamlessly with PyTorch based models. LLMWare We will use a Macbook Air with M1 Chip for this experiment, meaning we will only be using widely available hardware for this exercise. Testing Out Quantized Dragon Remember that in my previous article, we built a RAG application focused on legislation search. We used vector search to quickly search through several large legislations, found sections relevant to our question about Qualified Opportunity Zone Partnership Interest, and ran the question through a BLING model. In today’s article, we will run the same question through LLMWare’s quantized Dragon model and determine if it performs better than BLING models. In order to focus on model comparison and to reduce the amount of prior knowledge required, we will do a lot of the PDF parsing and vector search manually. This has the added benefit of making the problem artificially harder for the model - LLMWare’s default embedding search chunks the source material to about 1000 tokens, but handling the parsing manually allows us to bump the context up to around 3000 tokens. This will help us clearly demonstrate the difference between the Dragon and BLING models. However, you should be able to easily integrate with the rest of the LLMWare’s ecosystem if you want to leverage their tools by following the setup steps from my last article on LLMWare. In fact, if you simply replace the name of the BLING models with the quantized Dragon model from this article, everything should run seamlessly. Without further ado, let’s get started! First, let’s import the required dependencies: import sklearn
import sklearn.metrics # for cosine similarity
from llmware.prompts import Prompt
import time
import os
from openai import OpenAI
from PyPDF2 import PdfReader 

client = OpenAI() # the library now loads the key automatically as an environment variable. We can now load the PDF. In the previous example, we loaded several large legislations, but for today, we will only focus on the PDF version of the Tax Cuts and Jobs Act of 2017. reader = PdfReader([path to PDF of tax cuts and jobs act]) Now we can generate the embeddings for each page: embeddings = []
for pg in reader.pages:
    text = pg.extract_text() 
    embeddings.append(client.embeddings.create(
        input=text,
        model="text-embedding-ada-002"
    ).data[0].embedding) Let’s also generate the embeddings for the question we’re going to ask: question = 'What is a qualified opportunity zone partnership interest?'
q_embed = client.embeddings.create(
        input=question,
        model="text-embedding-ada-002"
    ).data[0].embedding With the embedding in hand, we can perform a vector search. Because our search space is small, we can just do this manually. cos_sim = [(idx, sklearn.metrics.pairwise.cosine_similarity([e], [q_embed])[0][0]) for idx, e in enumerate(embeddings)] Now we can take the most relevant page (which is index 132 or page 133 if you want to verify the results): most_relevant = sorted(cos_sim, key=lambda x: x[1], reverse=True)[0][0] And with that, we have come to the most crucial step. We will instantiate an LLMWare Prompter object with the quantized Llama Dragon model. The Prompter class is key here because it handles the prompt engineering for us and makes sure that our prompt is consistent with the structure of Dragon’s training data. The prompt class also automatically handles the llamacpp binding, so you can use the quantized Dragon model exactly like other models. model_name = "llmware/dragon-llama-7b-gguf"
prompter = Prompt().load_model(model_name)

response = prompter.prompt_main(question, context='\n\n'.join([reader.pages[132].extract_text()]), 
                                prompt_name="default_with_context", temperature=0.3) Wait a little while, and you should see the function call return. Now print the results: print(response['llm_response']) And you should see something like the following: • A capital or profits interest acquired by the qualified opportunity fund after December 31, 2017, from the partnership solely in exchange for cash;
•As of the time such interest was acquired, the partnership was a qualified opportunity zone business (or, in the case of a new partnership, it was being organized for purposes of being a qualified opportunity zone business);
•During substantially all of the qualified opportunity fund's holding period for such interest, the partnership qualified as a qualified opportunity zone business. This is quite a good answer! For comparison, let’s see how a BLING model would perform on the same problem. One of the issues we can expect is that the large context size can “overwhelm” a lower-parameter model and lead to a less informative answer. In my previous experiments, the sheared llama 2.7b was one of the best performers for this problem, so I decided to use that as the representative of the BLING models. model_name_2 = "llmware/bling-sheared-llama-2.7b-0.1"
prompter2 = Prompt().load_model(model_name_2)

response = prompter2.prompt_main(question, context='\n\n'.join([reader.pages[132].extract_text()]), 
                                prompt_name="default_with_context", temperature=0.3) After some processing, you should see something like this. A qualified opportunity zone partnership interest is a capital or profits interest in a domestic partnership if such interest is acquired by the qualified opportunity fund after December 31, 2017, from the partnership solely in exchange for cash. The response is still good but misses some of the details captured by the Dragon model. Specifically, the answer misses the holding period requirement and the new business case. This conforms to our expectations about the lower-parameter models’ difficulty with processing larger contexts. Interested readers can extend this experiment by using even lower parameter models or increasing the size of the given context. You should see the effect become increasingly more pronounced, after which the model will give a short, garbled answer. From this experiment, it should be clear that quantized Dragon models are able to outperform lower-parameter models for their intended use cases without noticeably compromising the model’s accuracy. And with that, we have used a quantized model to solve a real-world use case and learned about its performance characteristics in the process! Conclusion Today, we explored the exciting field of LLM quantization and looked at how companies like LLMWare are taking advantage of these developments to enhance their specialized language models. As I’ve argued before, miniaturization represents one of the most promising paths to the widespread adoption of AI technologies. By combining specialization, fine-tuning, and quantization, innovators in the AI space can create scalable and performant models that solve real-world problems. You can find LLMWare’s RAG framework on and their DRAGON and BLING models on LLMWare’s . Github Hugging Face repository By the way, I am working on an exciting project that seeks to use language AI and miniaturization to revolutionize education in the developing world. We’re working with incredible activists and educators worldwide, and we’re working to bridge the global digital divide. If you’d like to learn more about my project or simply want to talk about exciting developments in the LLM space, please do not hesitate to reach out to me on either or . Github LinkedIn

Run Llama Without a GPU! Quantized LLM with LLMWare and Quantized Dragon

একটি GPU ছাড়া লামা চালান! এলএলএমওয়্যার এবং কোয়ান্টাইজড ড্রাগনের সাথে কোয়ান্টাইজড এলএলএম

Führen Sie Llama ohne GPU aus! Quantisiertes LLM mit LLMWare und Quantized Dragon

¡Ejecute Llama sin GPU! LLM cuantificado con LLMWare y Quantized Dragon

Exécutez Llama sans GPU ! LLM quantifié avec LLMWare et Quantized Dragon

बिना जीपीयू के लामा चलाएं! एलएलएमवेयर और क्वांटाइज्ड ड्रैगन के साथ क्वांटाइज्ड एलएलएम

GPU なしで Llama を実行しましょう! LLMWare と Quantized Dragon を使用した Quantized LLM

GPU 없이 Llama를 실행해보세요! LLMWare 및 Quantized Dragon을 사용한 양자화 LLM

Execute o Llama sem GPU! LLM Quantizado com LLMWare e Dragão Quantizado

Запустите Llama без графического процессора! Квантованный LLM с LLMWare и Quantized Dragon

Lama'yı GPU olmadan çalıştırın! LLMWare ve Quantized Dragon ile Quantized LLM

Chạy Llama mà không cần GPU! LLM lượng tử hóa với LLMWare và Quantized Dragon

无需 GPU 即可运行 Llama！使用 LLMWare 和 Quantized Dragon 进行量化 LLM

Stories, News & Mentions about LLAMACORP

Llamacorp’s Stories on HackerNoon